Datadog CI/CD Optimization targets slow pipelines, flaky tests and CI costs

Finding where CI time actually goes

According to Datadog, AI-assisted development raises the rate at which teams produce code, which means more pull requests, builds, tests and pipeline executions. Slow jobs leave developers and coding agents waiting for feedback, flaky failures eat time in reruns and investigations, and unnecessary test execution raises runner demand.

CI/CD Optimization tracks pipeline execution time, queue time, failure rates and reliability trends across CI environments. Platform teams can use these signals to identify the pipelines and jobs with the greatest effect on delivery. From a single execution, they can look at the stages and jobs that add to total duration and link performance changes to commits, errors, logs, runners and infrastructure metrics.

Datadog stresses that this distinction guides the fix. A job that waits on an undersized runner fleet may benefit from more capacity. A job that has itself become slower points to a code, dependency, test or infrastructure change, so paying to run the same slow work on more runners would not help. CI/CD and test monitors can also alert teams when pipelines or jobs fail or degrade.

The company cites two customer results: Betterment reduced average build duration from almost 40 minutes to under 10 minutes, and The Browser Company reduced CI pipeline time by 50%.

Tools for handling flaky tests

Faster pipelines only help if developers trust the outcome. A flaky test can turn a valid build red, trigger a rerun and force an engineer to work out whether the failure relates to the change. Flaky Management in CI/CD Optimization lists flaky tests next to measures such as pipeline failures, CI time wasted and failure rate, so teams can prioritize the failures that cost the most developer time and compute.

• Auto Test Retries retries failing tests automatically up to a configurable limit, stopping when a test passes or the attempts run out. This keeps an intermittent failure from failing a whole build.
• Early Flake Detection and PR Gates are meant to keep newly introduced flakes from reaching the default branch while fixes are in progress.
• Flaky Management Explorer tracks failures caused by flaky tests, CI time lost to them, pipelines saved through automatic retries and time saved through test-execution improvements.

Datadog describes the controls as covering different points in the flaky-test life cycle: retries reduce immediate disruption, policies stop known failures from blocking unrelated changes, and detection helps stop the backlog from growing.

Running less work, and running it more efficiently

Adding runners raises capacity but does not remove unnecessary work. If every commit runs an oversized test suite, a bigger fleet just runs irrelevant tests faster at higher infrastructure cost.

Test Impact Analysis addresses this by mapping tests to the code they cover and using that to determine which tests a change affects. Tests not relevant to the change can be skipped. Datadog gives the example of a PR that only updates a README file, which may not need to run application tests.

Test Parallelization then speeds up the tests that still need to run. It uses expected test duration rather than splitting tests by count, which can leave one runner working through long tests after others finish. According to Datadog, it estimates the optimal number of CI workers and splits the tests between them to finish validation as soon and as cheaply as possible.

One view for pipelines and tests

Datadog says pipeline health and test health are often two views of the same delivery problem, and that separating them makes engineers spend time correlating data. CI/CD Optimization combines both in the same Datadog experience, so teams can move between pipeline executions, jobs and tests while investigating problems.

For platform teams, the pitch is that the same runner fleet can process more changes as development volume grows, without runner and compute spend rising at the same rate. For developers and coding agents, it is a faster verdict on each change. These outcomes are Datadog's own claims about the product.