Your AI Proof of Concept Was a Demo: 8 Tests Before You Commit

An AI proof of concept proves a system can work once. Production asks it to work every time, on traffic nobody curated, at a volume nobody rehearsed. Deloitte's own survey puts 38% of organisations piloting agentic AI and 11% running it in production. The distance between those two numbers is a testing problem before it is a technology problem.

What production changes that a proof of concept holds still

A proof of concept is a hypothesis test with most of the variables pinned in place. Somebody chose the prompts. Somebody cleaned the documents. The run happened during working hours, with an engineer watching the screen, ready to rerun anything that looked wrong.

Production removes all four controls in the same week. Input arrives in whatever shape the sender felt like using. Volume arrives in bursts that match your customers' calendar, not your load plan. Nobody is watching, and a rerun is a support ticket rather than a keystroke.

The result is that a system can be genuinely capable and still unfit to deploy. Capability is what the demo measured. Consistency, latency at the tail, cost per unit of real work and behaviour under hostile input are what production measures, and none of them appear on a demo agenda.

My position is blunt on this point. A pilot that ran once, on inputs the team selected, has produced an anecdote rather than evidence, and it should not be allowed to carry a production budget decision. The eight tests below exist to convert that anecdote into something you can argue about with numbers.

The production gap, in the numbers the firms actually disclosed

Two sources anchor the current picture, and both are worth reading directly rather than through the summaries.

Deloitte's Tech Trends 2026 reports a four-stage pipeline from its 2025 Emerging Technology Trends in the Enterprise Survey. 30% are exploring agentic options, 38% are piloting, 14% have something ready to deploy and 11% are running it in production. Deloitte does not publish a sample size for that survey in the article, which is a real limitation of the figure and one you should carry with it.

Gartner supplies the forward-looking half. In a June 2025 press release it predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Senior director analyst Anushree Verma described most current agentic projects as early stage experiments or proofs of concept driven by hype and often misapplied.

Read the order of Gartner's three reasons. Cost comes first, business value second, and risk controls third. Only the third is unambiguously a technology problem, which means most cancellations are decisions made by a finance function looking at a line item, not by an engineering team looking at an error rate.

The last row matters more than the first three. A number that gets attributed to a named research firm, but that the firm never published, is the most quotable and least checkable kind of statistic. Treat the pipeline stages as the citable object.

Tests 1 to 3: what production traffic does to your inputs

These three cost the least and catch the most. Run them before you write a business case, not after.

Test 1. Replay real traffic, unedited

Take a fixed window of actual requests from the system this AI is meant to change. A month of support tickets, a quarter of invoices, a week of search queries. Do not clean them, do not drop the malformed ones, and do not remove the duplicates.

Then run the whole window through the pilot and score it. The number you want is not the average quality score. It is the share of inputs the system handled without a human touching it, measured against the same window handled by your current process.

The failures here are boringly physical rather than intellectual. Scanned documents at the wrong orientation, a field that is empty in 4% of records, text in a second language, an attachment format nobody mentioned. A curated demo set has these edited out precisely because they are uninteresting, which is exactly why they survive into production untested.

Test 2. Score pass^8, not pass@1

The τ-bench benchmark introduced a metric worth stealing. pass@k asks whether at least one of k attempts succeeded. pass^k asks whether all k attempts succeeded, which is the question a production system actually faces when eight different customers arrive with the same request.

The published result is stark. The τ-bench paper reports that state of the art function calling agents succeed on under 50% of tasks, and that pass^8 falls below 25% in the retail domain. A capability that looks like a coin flip on one attempt looks considerably worse when the same task recurs.

Running this yourself is not difficult. Pick 20 representative tasks, run each 8 times with identical inputs, and record the share where all 8 runs produced an acceptable result. If that number is far below your single-run score, you have a variance problem, and variance is what generates support tickets rather than headlines.

Test 3. Force the multi-turn path

Pilots are usually driven by people who know what they want and say it in one go. Real users do not. They start vague, add a constraint, correct themselves, and change the goal halfway through.

Researchers at Microsoft Research and Salesforce measured what that does. In LLMs Get Lost In Multi-Turn Conversation, published May 2025, the authors ran more than 200,000 simulated conversations. They found an average 39% performance drop across six generation tasks when the same instruction was delivered over several turns instead of one. Their decomposition is the useful part: the loss is mostly increased unreliability rather than reduced aptitude.

Their finding on recovery is the one I would put in front of a steering committee. When a model takes a wrong turn early, it tends to commit to that assumption and does not recover. So the test is not just whether your system handles a second turn. It is whether it can be corrected after it has been wrong, which most demo scripts never require.

Tests 4 to 6: what volume does to speed, context and cost

The first three tests fail loudly. These three fail quietly, and usually after the contract is signed.

Test 4. Measure p99 under concurrency, not the average

Demo latency is measured on an idle system with one request in flight. Production latency is measured on a busy system with a queue, a cold cache and an external tool that is having a bad afternoon.

Record time to first token and total completion time at your realistic peak concurrency, and report p50, p95 and p99 rather than the mean. The average hides the shape of the distribution, and the tail is what users experience as the product being broken. A system with a fine median and a ten second p99 is unusable for anything interactive, and no average will tell you that.

Set the pass threshold before you run the test. Writing down "p99 under 6 seconds at 40 concurrent sessions" in advance is the difference between a measurement and a negotiation.

Test 5. Grow the context until accuracy bends

Long context windows are quoted as capacity. They behave more like a budget with declining returns. Chroma's July 2025 research tested 18 frontier models and found that performance varies significantly as input length changes, even on simple retrieval and text replication tasks. In their conversational question answering work, focused prompts of roughly 300 tokens outperformed full prompts of roughly 113,000 tokens.

That is vendor research rather than peer reviewed work, and Chroma sells a retrieval product, so treat the direction as informative and the magnitude as unsettled. The direction alone is enough to justify the test. Run your accuracy suite at four context lengths spanning your realistic range, and find the point where the curve bends rather than the point where the model refuses.

This matters most for agents that accumulate context as they work. A pilot task that finishes in three steps and a production task that finishes in thirty are not the same test, even when the underlying request is identical.

Test 6. Price one month at production volume

Pilot economics are almost always flattering, because the pilot ran on a small sample and often on promotional credits. Take the replayed traffic window from Test 1, count the tokens consumed including retries and failed runs, and extrapolate to a month at your expected volume.

Include three costs that pilots routinely omit. Retries after a failure, the evaluation runs you will keep running in production, and the human review time for anything the system escalates. The last one is frequently the largest, and it is the one that turns a cost saving into a cost transfer.

Then compare that figure against the cost of the process today. Unit costs behave counterintuitively as usage grows, which is covered in the piece on why falling token prices keep producing rising AI bills. The margin arithmetic sits in the breakdown of what AI cost of goods does to real gross margin.

Tests 7 and 8: what happens when something is hostile or absent

These two are the ones most pilots skip entirely, and they are the two that produce incidents rather than complaints.

Test 7. Send adversarial input, including through your own documents

The OWASP Top 10 for LLM Applications 2025 places prompt injection at LLM01 for the second consecutive edition. The mechanism is structural rather than incidental. Instructions and data travel down the same channel, so a model cannot reliably distinguish content it should process from an instruction it should follow.

Direct injection is the easy case to test. Indirect injection is the one that matters for enterprise deployments, because the hostile text arrives inside a document, a web page or a ticket that your own system retrieved. Plant instructions in a handful of documents in your test corpus and see whether the agent obeys them.

Score two things separately: whether the injection changed the output, and whether it caused a tool call the agent had permission to make. The second is the real exposure, and it is why OWASP recommends least-privilege tooling and human approval for high-risk actions rather than filtering alone. Where the answer is uncomfortable, the contractual side of that exposure is worth reading alongside the analysis of liability caps and remedies in AI agent contracts.

Test 8. Remove a dependency mid-run

Every agent depends on something it does not control. A vector store, a CRM API, an authentication service, a model endpoint. Production will take one of them away at some point, usually briefly and usually at a bad moment.

The test is simple to run and rarely run. Kill each dependency in turn while a task is in flight, and record what the system does. You are grading between three outcomes. A clean failure with a clear message, a silent degradation where the answer gets worse without anyone noticing, or a partial write that leaves data in a state nobody designed.

The middle outcome is the dangerous one. A system that keeps answering after its retrieval layer has gone quiet will produce confident output from no evidence, and there is nothing in the response that signals it. If your pilot has never been run with a broken dependency, you do not know which of the three you have.

The checklist, in one table

Copy this into whatever tracks your work. The third column is the one people skip, and skipping it is what turns a test into a discussion.

Effort estimates are for a team that already has the pilot running and access to real traffic. They are planning figures based on the work each test requires, not measured averages across deployments, and they will be wrong for you in at least one direction.

Two sequencing notes. Run tests 1, 2 and 3 first, because they are cheap and they invalidate the most business cases. Run test 7 before any deployment that grants tool permissions, regardless of what the other results said, because it is the only one on the list where a failure is somebody else's decision to exploit.

The structure of the list mirrors the Measure and Manage functions of the NIST AI Risk Management Framework, whose Generative AI Profile recommends both internal evaluation and external red-teaming before and after deployment. If you need the checklist to survive a governance review, mapping each row to that framework is the shortest path.

Where this checklist is weakest

Three things it does not do, stated plainly, because a checklist that oversells itself gets abandoned at the first surprise.

Passing all 8 does not predict adoption

Every test here measures the system. None of them measure whether anyone will use it. MIT's Project NANDA work on the generative AI divide located the failure in the learning and workflow gap rather than in model quality. A technically sound tool that never enters the workflow it was bought to change still returns nothing. That argument, and where measurable return has actually appeared, is covered in the analysis of who is actually making money from generative AI.

So run these tests, and run a separate adoption test. Put the system in front of ten real users doing real work for two weeks, and count how many are still using it in week three without being asked.

The thresholds are yours, and that is the soft part

I have deliberately left every pass condition as a variable. There is no published, credible, cross-industry threshold for acceptable p99 latency or acceptable pass^8, and any post that hands you one has made it up. The value of the column is that writing a number down in advance stops the number from moving after the result arrives.

This is also where the checklist can be gamed. A team that sets its thresholds after seeing the pilot results has run a ceremony rather than a test. If you own this process, the single control that matters is that thresholds are signed off before the first run, and that a failure has a named consequence.

The counter-case deserves a hearing too. Eight tests is a real cost, and for a low-stakes internal tool with no tool permissions and no customer exposure, running all eight is over-engineering. In that situation I would run tests 1, 2 and 6 and ship. The full list earns its cost when the system writes to a system of record, talks to a customer, or spends money.

Frequently asked questions

Where to start this week

Pick the AI system closest to a production decision in your organisation, and find out whether anyone has ever run it on a full, unedited window of real traffic. In most cases the answer is no, and establishing that takes one conversation.

Then run test 2 tomorrow, because it costs a day and it is the most likely to change the decision. Choose 20 tasks, run each 8 times with identical inputs, and record the share where all 8 runs were acceptable. Take that single number to whoever is being asked to sign, alongside the single-run score they have already seen.

If those two numbers are close, you have a system worth putting through the remaining six tests. If they are far apart, you have saved everyone a procurement cycle, and you have found the thing to fix first.

Related reading