Measuring Developer Productivity With AI: What METR, DORA and SPACE Each Miss

Experienced open-source developers took 19% longer to finish real tasks when they were allowed to use AI tools, in METR's 2025 randomised trial. Afterwards they believed they had been 20% faster. That gap is the central problem with measuring developer productivity with AI: the cheapest instrument available, asking the people doing the work, pointed the wrong way.

Short answer

To measure developer productivity with AI, combine three instruments. Use DORA's five delivery metrics for system throughput and stability, a SPACE-style survey for satisfaction and perceived flow, and a small controlled comparison on fixed tasks for causal speed. Never let self-reported speed stand alone, because METR showed it can point the wrong way.

What the METR study measured, and what it did not

METR is a non-profit research group that evaluates AI capabilities. In early 2025 it recruited 16 experienced open-source developers and asked them to submit real issues from their own repositories. Those repositories averaged more than 22,000 stars and more than 1 million lines of code, so this was mature, familiar work, not a toy exercise.

The design was a randomised controlled trial, which means each issue was randomly assigned to an AI-allowed or AI-disallowed condition before work began. The 246 issues averaged about 2 hours each. When AI was allowed, developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the frontier models of the period. METR paid participants $150 an hour.

The result was that tasks took 19% longer with AI allowed. The confidence interval ran from 2% to 39% slower, so the slowdown was statistically distinguishable from zero, but its size was uncertain.

The perception gap is the finding that survives

Before starting, the developers expected AI to speed them up by 24%. After finishing, having just experienced the slowdown, they still believed AI had sped them up by 20%. By our arithmetic on METR's figures, that is a gap of about 39 percentage points between what people felt and what the clock recorded.

This is the part of the study every engineering leader should keep. It does not say AI makes developers slower. It says that a developer's own estimate of their speed with AI is not a reliable measuring instrument, even for experienced people looking back on work they did last week.

That matters because most AI tooling business cases rest on exactly that instrument. The same pattern of high use and low confidence shows up in survey data, which is examined in the analysis of developer trust falling while usage rose.

METR's own correction in February 2026

METR ran a larger follow-up in late 2025 with 57 developers, 143 repositories and more than 800 tasks, at a lower rate of $50 an hour. In February 2026 it published the results and marked its 2025 finding as out of date.

For the 10 developers who returned from the original study, the new estimate was an 18% speedup, with a confidence interval from 38% faster to 9% slower. For the 47 newly recruited developers it was a 4% speedup, with an interval from 15% faster to 9% slower. Both intervals include zero.

METR then called its own data an unreliable signal of the current productivity effect. The reason was selection. Between 30% and 50% of surveyed developers said they had held back some tasks because they did not want to do them without AI. That means the hardest-to-replace AI work was quietly leaving the sample.

My view is that the 19% figure should stop being quoted as a fact about AI and start being quoted as a fact about measurement. The researchers who produced it have said as much, and they deserve credit for publishing the correction rather than defending the headline.

Why controlled trials of AI coding speed disagree

METR is not the only randomised trial of AI coding assistance, and the others point the opposite way. Putting them on one axis shows why no single trial result belongs on your scorecard.

A 2023 experiment by researchers from GitHub, Microsoft Research and MIT asked recruited developers to implement an HTTP server in JavaScript as quickly as possible. The group with GitHub Copilot finished 55.8% faster, with a confidence interval from 21% to 89% faster.

A 2024 trial inside Google gave 96 full-time engineers an enterprise task of building a logging service from a pre-existing change list. The authors' best estimate was that AI cut time on task by about 21%, though they stated the confidence interval was large.

Task shape decides the sign

The Copilot trial used a fresh, well-specified task with no existing codebase. The Google trial used a bounded enterprise task. METR used issues the developers had chosen from large, mature repositories they already knew intimately. METR itself says the developers' high familiarity with their repositories and the size and maturity of those repositories likely contributed to the slowdown.

So the trials are not contradicting each other. They are measuring different jobs. Your team's work is some mixture of all three shapes, which is why a trial result borrowed from someone else's population tells you very little about your own. The difference between benchmark tasks and real repositories is covered in more depth in the breakdown of coding agents on real repositories.

What DORA metrics measure about AI adoption

DORA is Google Cloud's long-running research programme on software delivery. Its 2025 report, published on 24 September 2025, surveyed nearly 5,000 technology professionals and added more than 100 hours of qualitative interviews. It is the largest current data set on how AI adoption relates to delivery outcomes.

DORA does not measure individual speed at all. It measures how a team's delivery system behaves, using five software delivery metrics grouped into throughput and instability. Throughput is change lead time, deployment frequency and failed deployment recovery time. Instability is change fail rate and deployment rework rate, the share of deployments that are unplanned fixes for a production incident.

Throughput up, stability down

The 2025 report found, for the first time, a positive relationship between AI adoption and software delivery throughput, and between AI adoption and product performance. It also found that AI adoption continues to have a negative relationship with software delivery stability.

That pairing is the most useful single result for a head of delivery. A team can ship more and break more in the same quarter, and a dashboard that tracks only lead time will report that as a win. The instability metrics exist to catch exactly this, and the downstream cost of fast, fragile code is measured in the analysis of AI code and technical debt metrics.

The amplifier finding is a warning about baselines

DORA's headline framing is that AI is an amplifier. In the report's words, AI does not fix a team, it amplifies what is already there. The research grouped teams into seven profiles, from "foundational challenges" to "harmonious high achievers", and argued that the same tool produces different outcomes depending on which profile adopts it.

The practical consequence is about baselines. If you do not know your delivery metrics before rollout, you cannot tell amplification from noise afterwards. DORA's companion AI Capabilities Model names seven capabilities that shape the return, including a clear and communicated AI stance, a healthy data ecosystem and a strong user-centric focus.

DORA is also survey data. Its 90% adoption figure and its finding that more than 80% believe AI raised their productivity are self-reports, and METR has just shown how far self-reported speed can drift. The delivery metrics are sturdier than the belief questions, and you should weight them that way.

What the SPACE framework adds, and where it stops

SPACE was published in ACM Queue in February 2021 by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck and Jenna Butler. Its central claim is that developer productivity cannot be measured by a single metric or dimension.

The name is the five dimensions: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Performance is defined as the outcome of a system or process. Activity is a count of actions or outputs, such as commits, pull requests or reviews. The authors' guidance, as summarised by GitHub, is that teams should use at least three of the dimensions and should never draw conclusions from one measure alone.

Activity is the dimension AI breaks first

SPACE predates widespread AI coding tools, and one of its five dimensions has aged badly. Activity counts are now partly a count of what the tool generated. Lines of code, commit volume and pull request counts all rise when a model writes more of the diff, whether or not more value reached a customer.

The framework already warns against reading activity alone, and that warning is now load-bearing. Any AI productivity claim built on "code written" or "suggestions accepted" is an activity metric presented as a performance metric. SPACE gives you the vocabulary to reject it in a review meeting.

Where SPACE stops is that it does not tell you which metrics to pick. It is a rule for assembling a scorecard, not a scorecard. Two teams can both claim to follow SPACE and track almost nothing in common, which is why the recommended set later in this post names specific measures.

METR, DORA and SPACE compared as measurement methods

The three are often cited as if they were competing answers to the same question. They are not. They operate at different levels and answer different questions, and each has a blind spot the others cover.

The last row is the reason to combine them. Each method wins on one question that the other two cannot answer at all.

If you only adopt one, DORA is the better instrument for a head of delivery. It measures the outcome your organisation pays for, and it already includes a quality counterweight. It is still not enough on its own, because it cannot tell you whether AI caused the change you see.

Where this argument is weakest

This post argues for a combined scorecard. Here is the case against it, stated as fairly as the evidence allows.

A scorecard is still a proxy

Every metric recommended below is a proxy for value delivered, and proxies can be optimised directly once people know they are being watched. A team rewarded on deployment frequency can split work into smaller deploys without delivering more. Combining metrics makes this harder, not impossible.

There is also a reasonable view that productivity measurement of knowledge work does more harm than good, because the act of measuring changes behaviour. I do not hold that view, but it has serious defenders, and the safest response is to use these numbers for diagnosing systems rather than ranking individuals. The line between those two uses is drawn in the piece on performance review criteria in the AI era.

Small teams cannot run the causal check properly

METR could not get a reliable signal from 57 paid developers and more than 800 tasks. A 12-person team running its own comparison will get far noisier results. The controlled check recommended below is directional at that scale, and you should label it that way on the dashboard.

The selection problem METR found will also hit you harder. Once developers depend on AI, asking them to work without it on randomly chosen tasks is both unpopular and less representative each quarter. A fixed benchmark task set avoids some of this, at the cost of measuring work that is less like your real backlog.

Finally, the evidence base itself is thin. Four trials and one large survey programme do not settle the question for every stack, language and team size. Anyone claiming a precise AI productivity multiplier for your organisation from this literature is extrapolating well past the data.

A developer productivity scorecard for AI you can defend

The set below takes DORA's delivery metrics as the backbone and adds the SPACE dimensions DORA does not cover. It also keeps one small causal check in the METR style. Every perception measure is paired with a behavioural one, so a gap between them shows up as a finding rather than hiding inside an average.

The review-wait metric is there because generation speed has outrun review capacity in many teams. If AI raises output and review time climbs with it, the gain is sitting in a queue. That pattern, and how teams are rebuilding around it, is measured in the analysis of code review as the new bottleneck.

The fixed-task comparison is the smallest version of METR's method that still means something. Pick 6 to 10 representative tasks from last quarter's backlog and have developers rotate through them in both conditions. Score the output by a reviewer who does not know which condition produced it. Report the result with its sample size on the dashboard, every time.

What to leave off the scorecard

I would not put AI suggestion acceptance rate on any productivity scorecard. It measures how often a developer pressed accept, which is a usage metric, and it rises when developers trust the tool more whether or not the code is better.

The same goes for the share of code written by AI, lines of code per developer and raw pull request counts. All four are SPACE activity metrics, all four are inflated by the tool being evaluated, and all four make a rollout look successful regardless of what reached production. They belong on an adoption report, clearly labelled, not on a productivity one.

Leave off individual rankings too. The numbers above describe a delivery system. Using them to rank people invites the gaming described earlier and destroys the honesty of the survey half of the scorecard. When leadership asks for a single ROI figure instead, the measurement trade-offs in the shift away from productivity as the top AI ROI metric are worth reading first.

Frequently asked questions

Where to start this week

Pull your last 90 days of deployment data and calculate all five DORA metrics, not just lead time. If you cannot compute change fail rate or rework rate from what you log today, fixing that instrumentation is the first piece of work. It matters more than any AI tool decision you make this quarter.

Then write one survey question that asks developers how much faster AI makes them, and put it on the same page as the delivery numbers. The first time those two disagree, you will have learned more about your team than a vendor benchmark can tell you. For how survey design itself distorts AI return figures, see the methodology gap in AI ROI surveys.

Related reading