Change failure rate is the one DORA metric you cannot game
Of the four DORA keys, one resists gaming, encodes your definition of failure, and starts the only argument worth having. A field guide to using it well.
Beyond Delivery Partners2 min readMetrics
The four DORA keys travel as a set, but they are not equally sturdy under organizational pressure. Deploy frequency and lead time can be flattered by ceremony, as we wrote in March. Time-to-restore depends on when you start the clock, which is a politics question wearing a stopwatch. Change failure rate, the share of changes that degrade service and need remediation, is the one that keeps telling the truth, for a structural reason: gaming it generally requires either shipping less (visible) or not calling failures failures (auditable).
Its honesty comes from the definition work it forces. To measure it at all, an organization must write down what counts as a failed change: a rollback, a hotfix, a feature flag kill, a sev-2 within some window of a deploy? Every one of those choices is a real decision about what the organization means by quality, and the argument the definition provokes is, in our experience, the most productive meeting a delivery diagnostic generates. We have watched teams discover mid-argument that half of them believed flag kills were routine and half believed they were incidents. Both had been reporting "stability" for a year.
Two usage notes from the field. Measure it over windows long enough to hold real counts: a team deploying daily has meaningful monthly rates, a team deploying weekly needs quarters, and small denominators make every dashboard twitch a false story. And resist the urge to drive it to zero, because a zero change failure rate over a long window usually means either heroic pre-release ceremony (paid for in lead time) or a definition that stopped counting. The DORA research consistently finds elite performers failing changes at nonzero rates and recovering fast; the design goal is cheap failure, not no failure.
Where it points, follow. A rising rate after a tooling change, a team whose rate doubles under deadline pressure, the correlation with batch size that the AI paradox made famous: each of those is the metric doing its one job, telling you where the system bleeds. The condition that reverses the advice is an organization that would use the number on individuals, in which case do not collect it, because a metric that can hurt a person will be starved of the failures it needs to mean anything.