From Sanskriti Khandelwal | Product & Market Analysis
TheAgentCompany Benchmark: Agents Finish 30% of Office Tasks, 8% in Finance
On this page
The best agent in the TheAgentCompany paper finished 30.3% of 175 simulated office tasks. That number hides a split. On the TheAgentCompany benchmark, the same agent completed 37.7% of engineering tasks and 8.3% of finance tasks. The departments people assume are easy to automate are the ones agents fail most often.
Key takeaways
- The paper's best agent finished 30.3% of 175 office tasks end to end. Gemini 2.5 Pro running in OpenHands passed every checkpoint on 53 tasks, at an average cost of $4.20 per attempt.
- Admin, data science and finance trail engineering by a wide margin. The same agent completed 26 of 69 engineering tasks but only 5 of the 41 admin, data science and finance tasks combined.
- The score metric is half partial credit, so it reads higher than completion. A score of 39.3% implies agents passed about 48% of checkpoints on average, while finishing under a third of tasks.
- The newest leaderboard entries changed who plays the colleagues. The top two submissions, at 42.86% and 41.14%, ran with a different simulated-coworker model than the paper, so their gains are not like-for-like.
The short answer. TheAgentCompany is a Carnegie Mellon benchmark that drops AI agents into a simulated software company and grades them on 175 work tasks. The best agent in the paper fully completed 30.3% of them. Engineering, project management and HR tasks finished most often. Admin, data science and finance tasks finished least.
I would read this post as an operations lead deciding where to pilot agents, because that is who it is written for. The question here is which functions are ready for end-to-end delegation, and which are not.
What TheAgentCompany benchmark tests, and why it differs
Most agent benchmarks test one skill in one environment. A coding benchmark gives an agent a repository. A browsing benchmark gives it a website. TheAgentCompany gives it an office, with an intranet, files, a chat server and coworkers.
The authors built a fictional software firm out of four self-hosted, open-source tools. GitLab holds code and a wiki. ownCloud holds files and office documents. Plane tracks issues and sprints. RocketChat carries messages between staff. The agent works through a bash shell, a Python notebook and a Chromium browser.
The task list started from labour data rather than from what is easy to grade. The authors began with release 29.1 of the O*NET occupational database. They looked for job categories with many workers and high median pay, then excluded jobs that need physical labour. That process is how the benchmark ended up set inside a software company.
Twenty computer science students, software engineers and project managers wrote the tasks over two months. The paper puts the effort at about 3,000 person-hours. Two senior HR and admin professionals collaborated on the admin, HR and finance tasks.
The colleagues are played by a model
Many tasks require the agent to ask someone for information, chase an approval or notify an assignee. Those coworkers are simulated on the Sotopia platform. By default every simulated colleague is backed by Claude 3.5 Sonnet, version 20241022. That detail matters later, because the newest leaderboard entries changed it.
This is the design choice that makes TheAgentCompany worth paying attention to. Real office work is rarely a solo puzzle. It is a chain of small dependencies on other people, and the benchmark tests whether an agent can work that chain. If you have read our breakdown of which computer use tasks OSWorld says are ready, this is the next layer up. The question is not "can it click the button", but "can it get the job done when the job runs through three colleagues".
How partial credit works in TheAgentCompany scoring
Every task is broken into checkpoints. Each checkpoint is worth points, and a checkpoint can award some of its points. The paper reports two numbers for every model, and they mean very different things.
Success and score are separate metrics
The success rate is binary per task. An agent gets 1 only if it passes every checkpoint, and 0 otherwise. Score blends that result with partial credit. The paper defines it as 0.5 times the share of checkpoint points earned, plus 0.5 times the full-completion result.
The paper's worked example is a sprint-management task for the RisingWave project. The agent must move unfinished issues to the next sprint in Plane and notify assignees on RocketChat. It must then clone the repository from GitLab, run code coverage, upload a summary to ownCloud and act on feedback from a simulated project manager. In that example the agent earned 44 of 88 possible points and did not finish. Its score was 0.5 times 0.5, plus 0, which is 25%.
Most checkpoints are graded by deterministic Python functions. For deliverables that are unstructured, such as a written summary, the graders first try keyword matches and then fall back to an LLM. The paper says 51 tasks, or 29%, involve LLM evaluation, always as a supplement to deterministic checks. For how far an LLM grader can be trusted, see our review of the studies on LLM judge agreement and bias.
Reading score as checkpoint completion
Because score is defined exactly, you can invert it. Twice the score minus the success rate gives the average share of checkpoint points earned per task. This is my calculation from the paper's published numbers, not a figure the paper prints.
For Gemini 2.5 Pro, that is 2 times 39.3, minus 30.3, which equals 48.3% of checkpoint points. Finished tasks earn 100% by definition. So across the 122 tasks it did not finish, the agent earned about 25.8% of available points on average.
My position is that score is the wrong headline number for anyone buying agent software. A task that is a quarter done is usually a task a human must redo from the start, because finding where the agent stopped costs time too. Completion rate is the number that maps to labour saved.
AI agent task completion by department
This is the table the benchmark exists to produce. It comes from Table 5 of the paper, which splits results by the nature of the task. The counts in brackets are the number of tasks in each category. I have added the number of tasks completed, because percentages on 12 tasks hide how small the base is.
| Department (tasks) | Gemini 2.5 Pro success / score | Claude 3.7 Sonnet success / score | Claude 3.5 Sonnet success / score | Tasks finished by best model |
|---|---|---|---|---|
| Software engineering (69) | 37.68% / 45.05% | 30.43% / 36.78% | 30.43% / 38.02% | 26 of 69. |
| Project management (28) | 39.29% / 52.61% | 42.86% / 56.32% | 35.71% / 51.31% | 12 of 28. |
| HR (29) | 34.48% / 44.98% | 27.59% / 40.02% | 24.14% / 34.49% | 10 of 29. |
| Data science (14) | 14.29% / 20.06% | 14.29% / 22.80% | 14.29% / 21.70% | 2 of 14. |
| Admin (15) | 13.33% / 19.17% | 13.33% / 23.89% | 0.00% / 11.59% | 2 of 15. |
| Finance (12) | 8.33% / 21.56% | 0.00% / 18.89% | 8.33% / 25.17% | 1 of 12. |
| Other (8) | 12.50% / 20.16% | 12.50% / 24.06% | 12.50% / 22.40% | 1 of 8. |
Source: TheAgentCompany, Table 5, arXiv 2412.14161 v2 and v3. Gemini 2.5 Pro and Claude 3.7 Sonnet ran on OpenHands 0.28.1, Claude 3.5 Sonnet on 0.14.2. The last column is the highest success rate in the row multiplied by the task count, rounded to whole tasks.
Engineering, project management and HR lead
Software engineering tasks finished at 37.68% for the best model, which the paper notes is the reverse of what a human would expect. These are the tasks with the most public training data. Code, tickets and commit messages are everywhere on the open web. Claude 3.7 Sonnet did slightly better than Gemini 2.5 Pro on project management, completing 12 of 28 sprint and issue tasks.
HR is the result I find most underrated. Gemini 2.5 Pro completed 10 of 29 HR tasks, close to its engineering rate. Most commentary on this benchmark says agents fail at people work. The table says something narrower. Agents fail at people work that runs through files and spreadsheets, not at people work that is structured process.
Admin, data science and finance trail
The paper itself says these three categories score lowest, "with many LLMs completing none of the tasks successfully." Across the 41 tasks in the three categories combined, Gemini 2.5 Pro completed 5. That is 12.2%, against 35.8% on the remaining 134 tasks. Finance is the extreme. No model in the paper completed more than one of the 12 finance tasks.
Finance has a quieter signal worth noticing. Its score runs well above its success rate. For Gemini 2.5 Pro the implied checkpoint completion on finance is about 34.8%, against 8.33% success. Agents get partway into finance work, then stop. One example finance task in the paper asks the agent to complete Section B, the Alternative Simplified Credit, of IRS Form 6765. That is exactly the shape of work where partial output has no value.
Agent completion by tool: chat and files break agents
The paper also splits results by the tool a task requires. A task that needs both GitLab and RocketChat counts under both, so these counts do not sum to 175. This view explains the department table better than the department labels do.
| Model | GitLab (71 tasks) | Plane (17 tasks) | RocketChat (79 tasks) | ownCloud (70 tasks) |
|---|---|---|---|---|
| Gemini 2.5 Pro | 33.80% / 43.36% | 41.18% / 51.67% | 29.11% / 40.39% | 12.86% / 22.44% |
| Claude 3.7 Sonnet | 23.94% / 32.60% | 41.18% / 52.63% | 29.11% / 41.81% | 11.43% / 23.97% |
| Claude 3.5 Sonnet | 30.99% / 40.25% | 41.18% / 50.37% | 21.52% / 34.68% | 10.00% / 21.81% |
| GPT-4o | 11.27% / 19.46% | 23.53% / 33.68% | 5.06% / 16.08% | 1.43% / 7.76% |
Source: TheAgentCompany, Table 4. Rows selected from the 13 configurations the paper reports. Tasks can require more than one platform.
ownCloud is where agents collapse. The best model finished 12.86% of the 70 tasks that touch it, roughly 9 tasks. ownCloud is the file and office-document system. Spreadsheets, forms and shared folders live there, and so do most admin and finance deliverables. The department gap in the previous section is largely a document gap.
The paper names a specific failure. A closable welcome pop-up in ownCloud stopped the OpenHands agent, which reads pages as text, from making progress. It failed to click the "x". The visual-browsing agent, OWL RolePlay, handled pop-ups better but got lost in complex web interfaces and clicked the wrong elements more often. Neither approach solves office software yet.
RocketChat is the social layer, and the paper reads its low scores as a lack of communication skill. Its examples are worth quoting in substance. In one task an agent was told which colleague to ask next, then never followed up and declared the task done. In another, unable to find the right person in chat, the agent renamed a different user to the intended name. The paper calls this "deceiving oneself". I call it the failure mode to design your pilots around, and it maps closely onto the patterns in our piece on why agent pilots fail.
The TheAgentCompany leaderboard since the paper
The paper is a snapshot. The project also runs a public leaderboard, and its data file lists 17 entries. The newest is dated 10 November 2025. Here are the top six, with the field most coverage skips: the environment model that plays the colleagues and runs the LLM graders.
| Agent and model | Success | Score | Submitted | Environment model |
|---|---|---|---|---|
| TTE-MatrixAgent, DeepSeek-V3.2 | 42.86% | 52.40% | 10 Nov 2025 | Qwen Plus. |
| MUSE, Gemini 2.5 Flash | 41.14% | 51.78% | 13 Oct 2025 | GPT-4o. |
| OpenHands-Versa, Claude Sonnet 4 | 33.14% | 43.19% | 14 Jun 2025 | Claude 3.5 Sonnet. |
| OpenHands-Versa, Claude 3.7 Sonnet | 30.86% | 40.18% | 14 Jun 2025 | Claude 3.5 Sonnet. |
| OpenHands, Gemini 2.5 Pro | 30.29% | 39.28% | 10 May 2025 | Claude 3.5 Sonnet. |
| OpenHands, Claude 3.7 Sonnet | 26.29% | 36.41% | 10 May 2025 | Claude 3.5 Sonnet. |
Source: leaderboard.json in the project's website repository, retrieved 3 November 2026. The TTE-MatrixAgent submission README reports 75 of 175 perfect completions. The field name in the file is spelled "enviroment_model".
Different colleagues, different test
The top two entries did not run against the same simulated coworkers as everything below them. One used Qwen Plus, the other GPT-4o, where the paper used Claude 3.5 Sonnet. A coworker model that answers more directly, or an LLM grader that is more lenient, can move completion without the agent improving at all. I am not saying either submission is wrong. I am saying the jump from 33.14% to 41.14% cannot be read as progress on one fixed test.
The cleaner comparison is within the Claude 3.5 Sonnet environment. There, the best entry is OpenHands-Versa with Claude Sonnet 4 at 33.14%, up from 24.0% in December 2024. That is real movement over six months. It is also a long way from the "agents can run the office" framing that tends to travel with this benchmark. Our explainer on why LLM leaderboards disagree on methodology covers the general version of this problem.
Cost per finished task on TheAgentCompany
The paper reports an average cost per task attempt, calculated from token counts and list prices with no prompt caching. Dividing by the completion rate gives a cost per finished task. This is a Zan Digital calculation, and it is the figure that compares to a salary.
Gemini 2.5 Pro cost $4.20 per attempt and finished 30.3%, so about $13.86 per completed task. Claude 3.5 Sonnet cost $6.30 per attempt at 24.0%, or $26.25 per completion. Gemini 2.0 Flash was the cheapest per completion among the paper's models, at about $5.26, despite finishing only 11.4% of tasks.
The cheap entries carry a hidden cost. The paper found that Gemini 2.0 Flash took about 40 steps per task, often because it looped or explored aimlessly. A low token bill on a failed task still leaves a human to redo it, and that labour is not in the cost column.
The TTE-MatrixAgent entry lists $0.40 per task with DeepSeek-V3.2, which works out near $0.93 per completion. The leaderboard does not say how that cost was measured, so treat it as directional. If it holds, the cost of an attempt is no longer the constraint. Reliability is.
Where this evidence is weakest
I have leaned on this benchmark for a department-level verdict. Here is the case against doing that.
Twelve tasks cannot carry a verdict
Finance has 12 tasks, admin 15 and data science 14. One task is 8.33 points of finance success. The paper reports a single value per model and says nothing about repeated runs or variance. A different random seed could plausibly turn 1 of 12 into 3 of 12, and the headline would change. The department ranking is a strong hint. It is not a measurement with a confidence interval.
The authors also state that their tasks lean "more straightforward" because every task must be graded by a program. They exclude creative work such as brainstorming product ideas or designing system architectures. They did not measure human performance, citing cost, since one task can take a person anywhere from 10 minutes to several hours. So there is no human baseline to compare 30.3% against.
The setup may favour some models
One model family plays the colleagues and grades the 29% of tasks with LLM evaluation. In the paper, that is Claude 3.5 Sonnet. The paper does not test whether a coworker or grader from the same family as the agent changes outcomes. The Claude results here may carry a small home advantage, or none at all. Nobody has published the experiment that would settle it.
The model list is also dated. The newest leaderboard entry is from November 2025, and the paper's best model is Gemini 2.5 Pro. Models released in 2026 have not been submitted, so this benchmark says nothing direct about them. Read every figure here as a floor for current systems on this task mix, not as their measured level. If you want to run a comparable test on your own workflows, our guide to building an LLM eval suite is the practical starting point.
Which job functions agents can do end to end today
On this evidence, no department clears half. The strongest number in the paper is 42.86% on project management, which is 12 of 28 tasks. If your bar for end-to-end delegation is "finishes most of the time without a human", nothing in TheAgentCompany meets it.
A lower bar is more useful. Where an agent finishes a third of tasks and a human checks every output, you get real throughput. That bar is met in three places on this benchmark: structured engineering work in a code repository, sprint and issue management, and process-driven HR tasks. Those share a pattern. The inputs are structured, the tools are text-first, and success is easy to verify.
I would not hand an agent admin or finance work end to end on this evidence. The failures there are concentrated in document handling, and the outputs are the kind where a half-finished form is worse than none. If you pilot there, scope the agent to drafting and data gathering, with a human owning submission.
The consensus reading of this benchmark is "agents are bad at office work". I think that undersells it. The better reading is that agents are as good as the interface they are given. Clean APIs and text-based tools get engineering-grade results. Pop-ups, file browsers and ambiguous colleagues get admin-grade results. That is a design problem your team can partly fix before the models improve.
Frequently asked questions
What is TheAgentCompany benchmark?
TheAgentCompany is a benchmark from Carnegie Mellon University researchers that tests AI agents inside a simulated software company. It has 175 tasks across engineering, project management, HR, admin, data science and finance. Agents work through self-hosted GitLab, ownCloud, Plane and RocketChat, and talk to simulated colleagues backed by a language model. Each task is graded on checkpoints, so both full completion and partial progress are measured.
What is the best score on TheAgentCompany?
In the paper, the best agent was Gemini 2.5 Pro on OpenHands, finishing 30.3% of tasks with a score of 39.3%. On the project leaderboard, the top entry is TTE-MatrixAgent with DeepSeek-V3.2, at 42.86% success and 52.40% score, submitted in November 2025. That entry used a different model to play the colleagues, so it is not a direct comparison.
How does partial credit work in TheAgentCompany?
Each task is split into checkpoints worth points. Success is 1 only when every checkpoint passes. Score is half the share of checkpoint points earned plus half the success result. An agent that earns half the points but does not finish scores 25%. Most checkpoints are graded by Python code, with an LLM fallback for unstructured outputs on 51 of the 175 tasks.
Which office tasks can AI agents complete?
On TheAgentCompany, agents complete engineering, project management and HR tasks most often, at roughly 34% to 43% for the best models. Admin, data science and finance are far lower, between 0% and 14%. The weak categories rely heavily on documents and spreadsheets in ownCloud, where the best model finished only 12.86% of tasks.
Why do AI agents fail at finance and admin tasks?
The paper points to document-heavy interfaces and social steps. Agents got stuck on an ownCloud welcome pop-up, struggled with complex web interfaces, and failed to follow up with colleagues over chat. In finance, agents earned about a third of checkpoint points but completed only 1 of 12 tasks, which suggests they start the work and cannot finish multi-step deliverables.
Is TheAgentCompany a reliable LLM evaluation for agents?
It is one of the more realistic agent benchmarks, because tasks span several tools and require talking to colleagues. It has limits. Several categories have fewer than 15 tasks, results are reported from single runs, there is no human baseline, and the same model family plays colleagues and grades some tasks. Treat department rankings as strong hints rather than precise measurements.
Where to start this week
Take one workflow you are considering for an agent pilot and count its tools. Write down whether each step runs through a text interface, a document, or a person. If more than one step is a document or a colleague, expect admin-grade completion and budget the human review time accordingly.
Then ask your agent vendor for a completion rate, not a score, on a task set that looks like yours. If they quote a TheAgentCompany number, ask which environment model they ran it with. A vendor who knows the answer has measured something. A vendor who does not has read a leaderboard.
Related on agent readiness
For the single-tool view, read what OSWorld says about computer use tasks. For the money side, see how agent payback claims compare by function.
References
- Frank F. Xu, Yufan Song, Boxuan Li et al., Carnegie Mellon University, TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks, arXiv 2412.14161, v1 18 December 2024, v3 10 September 2025. Used for task counts, Tables 1, 4 and 5, scoring formula, failure modes and limitations.
- TheAgentCompany paper, HTML version 2, 19 May 2025. Used for the verbatim scoring definition, the RisingWave worked example and Appendix E on LLM evaluation.
- TheAgentCompany project, leaderboard data file, retrieved 3 November 2026. Used for leaderboard entries, dates and environment models.
- TheAgentCompany experiments repository, evaluation submissions, including the TTE-MatrixAgent with DeepSeek-V3.2 README reporting 75 of 175 completions. Retrieved 3 November 2026.
- TheAgentCompany, benchmark source repository. Used for the environment setup and the link to the public leaderboard.
The weakest part of this source base: every department figure rests on one run per model over 8 to 69 tasks, and the leaderboard entries are self-submitted. Cost per completed task and checkpoint share are Zan Digital calculations from published figures. Last verified 3 November 2026.
Related reading