Devin, Factory and the Gap Between Agent Demos and Real Repositories

Five coding agents opened 7,156 reviewed pull requests on public GitHub projects. 61.6% of Devin's were accepted. 77.9% of OpenAI Codex's were. On Scale AI's private commercial codebases the best system resolves 51.5% of tasks, against 61.5% on public ones. Demo repositories flatter autonomous agents. Yours will not.

The short answer

Autonomous coding agents complete a large minority of well-scoped tasks on code they have effectively seen before, and a smaller minority on private code they have not. Published evidence puts the drop at roughly 11 percentage points. Agent choice matters less than task type, and review burden costs more than the licence.

That is the finding a buyer needs. The rest of this post shows the working, names the sources, and sets out a protocol you can run on your own repository inside a week.

One framing note first. This post is written for a head of engineering or head of delivery deciding whether to roll an agent past a pilot. That persona was chosen because the deciding numbers here are throughput and review load, not model quality, and those land on delivery leadership rather than on individual practitioners.

What counts as a real repository

The phrase does a lot of quiet work in vendor marketing. Every agent demo runs on a repository. The question is which one.

Demo repositories are pre-solved problems

A demo repo is small, conventionally structured, and documented. Its dependencies resolve. Its tests pass before the agent starts. Most importantly, a public repository with a permissive licence and a popular framework is well represented in model training data.

Legacy code inverts every one of those properties. It carries local conventions nobody wrote down, a build that only works in one order, and tests that were already failing on Tuesday. An agent has to infer the conventions from the code, and inference is where the failures cluster.

Contamination is the quiet variable

Scale AI built SWE-Bench Pro specifically around this problem. Its public split draws on repositories under strong copyleft licences, on the reasoning that the viral licence terms make inclusion in training corpora less likely. Its commercial split uses code that has never been published at all.

That design choice is the most useful thing in the benchmark. It separates the question "can the model do this" from the question "has the model already read the answer". Those are different capabilities, and buyers keep paying for the first while measuring the second.

Completion rates fall as the repository gets more real

Three data points, in ascending order of realism, tell a consistent story.

From SWE-bench Verified to SWE-Bench Pro

When Scale published SWE-Bench Pro on 19 September 2025, top models were already scoring above 70% on SWE-bench Verified. On the new public set, GPT-5 managed 23.3% and Claude Opus 4.1 managed 23.1%. The benchmark is 1,865 instances across 41 repositories, split into 731 public, 858 held out and 276 commercial.

A 47 point drop between two benchmarks that both claim to measure real GitHub issues should end the practice of quoting a single headline score. It does not, because the higher number is easier to put on a slide.

From public copyleft to private commercial

The more interesting comparison sits inside SWE-Bench Pro, because the task construction is held constant and only the code changes. Scale reported Claude Opus 4.1 falling from 22.7% to 17.8% on the commercial split, and GPT-5 falling from 23.1% to 14.9%.

Models have improved a great deal since. The gap has not closed. Reading both Scale Labs leaderboards on 21 August 2026, the five highest-scoring public systems average a 10.9 point decline when the same task set is run against private commercial code.

Merge rate is the number that survives contact with a codebase

Benchmarks award a point when hidden tests pass. Nobody ships on that basis. The production question is whether a human merged the change.

The 7,156 pull request study

Giovanni Pinna, Jingzhi Gong, David Williams and Federica Sarro published a task-stratified analysis of agent pull request acceptance in May 2026. They took 7,156 closed pull requests from the AIDev dataset, restricted to repositories with more than 100 stars and permissive licences, and kept only those that received a review or comment from someone other than the author.

This is observational data, not an experiment. It is also the largest public accounting of what happens to agent-written code after it meets a reviewer, and it draws on a dataset spanning more than 456,000 agent pull requests across 61,000 repositories.

Read the last column before the second. Devin was observed for 32 weeks, nearly three times the window for Codex, so its figure blends older agent versions with newer ones. Claude Code's 139 pull requests are too few to separate it from Cursor with confidence.

Task type matters more than agent choice

The paper's central result is not the ranking. It is that the ranking is smaller than the task effect. Documentation changes were accepted 82.1% of the time against 66.1% for features, a 16 point differential. Chore work reached 84.0% and performance work fell to 55.4%, a 29 point spread.

No agent won everywhere. Codex led on fixes at 83.0%, Claude Code led on documentation at 92.3%, and Cursor led on tests at 77.8%. My reading is that agent selection is a second-order decision and work routing is the first-order one. Most teams have this backwards, and run a long bake-off between vendors before deciding which tasks to send.

Review burden is the cost nobody prices

Every agent pricing page quotes a subscription. None of them quotes the reviewer hours that a merged change consumes. That omission is where most pilot business cases go wrong.

The Pinna data gives a usable range. Copilot pull requests attracted 4.94 reviews each. Codex pull requests attracted 1.39. The authors are careful to say the causal direction is unclear, and they are right to be. It could be that harder tasks attract both more review and more rejection.

The direction of causation does not change your staffing though. If you route 100 tasks a month to an agent, you are buying somewhere between 139 and 494 review passes. That is the line item, and it is denominated in senior engineering attention rather than in dollars.

The 2025 DORA report, drawing on nearly 5,000 technology professionals, found 90% using AI at work and more than 80% reporting a productivity gain, while 30% reported little or no trust in AI-generated code. DORA also found AI adoption keeping its negative relationship with software delivery stability. Low trust and high volume is precisely the combination that produces 4.94 reviews per pull request.

Older evidence points the same way. METR's randomised trial of 16 experienced open-source developers on 246 tasks in repositories averaging over a million lines found them 19% slower with AI tools while believing they had been 20% faster. METR now labels that result historical and says it does not necessarily describe current tools. The perception gap it measured is the part I would not assume has gone away.

What Devin and Factory actually cost per completed task

Two vendors, two pricing models, and neither one prices the thing you care about.

Cognition repriced Devin in April 2025, dropping the entry point from $500 a month to $20 with pay-as-you-go billing after that. Usage is metered in agent compute units at $2.25 each on the entry plan and $2.00 on the team plan. One unit is roughly 15 minutes of active agent work, so an hour of Devin runs about $9.00.

Factory sells seats instead. Its published tiers are $20 for Pro, $100 for Plus at around five times the usage, and $200 for Max at around ten times. Factory does not publish an absolute usage number for any tier, and it has not published a SWE-bench Verified score. It has published a Terminal-Bench result: Droid scored 58.75% across 80 containerised tasks in September 2025, ahead of Claude Code at 43.2% and Codex CLI at 42.8%.

Cost per merged change is the only honest denominator

Take the published rates and do the arithmetic. It is arithmetic, not measurement, and the assumptions are visible so you can replace them with your own.

Every row above is arithmetic on published rates, not a measured result. The 2 units per task and the 15 minutes per review pass are assumptions, and they are the two numbers your own trial should replace first. The point of the table is the ratio: reviewer time runs 11 to 41 times the agent bill.

Where this analysis is weakest

Three problems, and the first one is serious enough that I would rather state it in a heading than bury it.

This post does not carry first-party trial data

The honest version of "we ran a consistent task set across agents" is that we did not. Zan Digital has no proprietary benchmark, and inventing one would be worse than admitting the gap. What is original here is the paired public-versus-private calculation, which is arithmetic on two public leaderboards read on one date, and the cost table, which is arithmetic on published prices.

That means every completion figure in this post is somebody else's measurement. Treat the ranking as directional. Treat the direction of the gap, which four independent sources agree on, as the finding.

Benchmarks and merge rates measure different things

SWE-Bench Pro scores a hidden test suite. The Pinna study counts human decisions on public repositories. A maintainer can reject correct code for style, scope or timing, and can merge code that passes review and breaks later.

The two measurements are also drawn from different code. SWE-Bench Pro's commercial split is genuinely private. The pull request study is entirely public and permissively licensed, which is the training-data-rich case. If anything, the acceptance rates above are an optimistic reading of what happens inside a closed enterprise repository.

Every number here goes stale in weeks

Scale's September 2025 figures of 23.3% and 23.1% were state of the art for a few months. The public leaderboard now leads at 61.5%. Anything quoted from a leaderboard is a photograph of a moving object, and the specific model names in this post will look dated quickly.

The structural finding is more durable than the scores. A gap between seen and unseen code has persisted through a near tripling of absolute performance, and there is no published evidence that it is closing.

A one-week protocol for testing agents on your own repository

Vendor trials are designed to be passed. Design one that can be failed instead. This is the shape I would use, and it takes a week of calendar time rather than a quarter.

Pull 20 closed issues from your own backlog, from the last six months, weighted the way your real work is weighted. Do not select for suitability. If a quarter of your work is performance tuning, a quarter of the sample is performance tuning, even though that is the category agents handle worst.

Run the same 20 through each candidate agent with identical prompts and identical repository access. Route the output into your normal review process with the author anonymised, so reviewers are not softening or hardening because they know a machine wrote it.

Then record three numbers per agent, and only three. Merge rate, which is merged pull requests divided by attempts. Review passes per merged change, counted from your own tooling. Wall-clock time from task assignment to merge. Vendor demos optimise the first number. Your delivery schedule lives on the third.

A useful pass threshold is a merge rate above your team's own baseline for equivalent work, with review passes no more than one higher. If you have never measured your human baseline, that is the first week's real finding, and it is the same discipline argued for in the piece on building versus buying coding agents.

Frequently asked questions

Where to start

If you are three weeks into a vendor bake-off, stop and re-cut the sample. The Pinna data says task type moves acceptance nearly twice as much as agent choice does, so a bake-off run on unrepresentative tasks is measuring the wrong variable at some expense.

If you have not started, begin with the number you already own. Pull your merge rate and review passes per change for the last quarter of human work. Without that baseline, any agent result you produce is uninterpretable, and you will end up arguing about a vendor's benchmark instead of your own delivery.

And when a vendor quotes a resolution rate, ask one question: on whose code. If the answer is a public benchmark, the number describes performance on code the model has probably read.

Related reading