Promptfoo vs DeepEval vs Ragas: One Support Agent Suite, Three Different Tools

Promptfoo, DeepEval and Ragas can all grade an AI agent, but only two of them can stop a bad model swap from merging on their own. We wrote one 12-case LLM testing suite for a customer support agent and mapped it onto each tool's documented features. Promptfoo lists 44 assertion types that need no judge model. That number matters more than any metric count.

The short answer

Pick Promptfoo if your team wants YAML configs and a ready GitHub Action. Pick DeepEval if your team already writes pytest and needs agent and conversation metrics. Use Ragas inside either one for retrieval metrics. None of the three replaces a test suite built from your own production failures.

Why LLM testing tools decide whether a model swap is safe

A model swap is the riskiest routine change an AI team makes. The prompt stays the same, the code stays the same, and the behaviour moves anyway. Retirement calendars at the major labs now force that swap on a schedule you do not set, a pattern covered in the guide to building an eval suite that outlives the model.

An evaluation framework is the software that runs your test cases against a model and returns a pass or a fail. It does not decide what correct means for your product. You do that. What the framework decides is how cheaply you can express that definition, and whether a failing run actually blocks a deploy.

That second property is where the three tools differ most. A framework that produces a beautiful score report but never fails a build is a dashboard, not a gate. Dashboards get ignored in the same week a launch date slips.

The cost of getting this wrong compounds. The switching costs of moving between model providers, set out in the analysis of model lock-in, are mostly the cost of not knowing whether the replacement is safe. A working test gate turns that unknown into a number you can read on a pull request.

This comparison covers only the open-source libraries. Hosted tracing and monitoring products are a separate purchase with separate economics, compared in the LangSmith, Braintrust and Arize breakdown.

The support agent test suite: 12 cases across four failure types

The agent under test is a customer support agent with four tools. It can look up an order, search a knowledge base, issue a refund and escalate a ticket to a human. Refunds above a fixed limit must go to a human, and every refund must follow an order lookup.

We wrote 12 cases, three for each failure type that matters in support work. The suite is small on purpose. A dozen cases you trust beats 200 you never read, and every case maps to a real way that support agents go wrong in production.

The suite is a specification written for this comparison. Order numbers and refund limits are illustrative test fixtures, not data from any real deployment.

Why half the suite avoids a judge model

Two of the four rows can be graded by code alone. Either the refund tool was called or it was not. Code-graded checks return the same answer on every run, and that repeatability is the property a release gate needs.

The other two rows need a judge model, meaning a second LLM that reads the output and scores it. Judges are useful and measurably imperfect. Zheng and colleagues found GPT-4 agreed with human preferences over 80% of the time in 2023, roughly the rate at which humans agree with each other. That is good enough to inform a release decision. It is not good enough to block one alone.

Promptfoo: one YAML file and the widest assertion menu

Promptfoo is a command-line tool and library, installed through npm, Homebrew or pip, or run without installing through npx. Tests live in a YAML config that lists prompts, model providers and test cases, with assertions under each case. A product manager can read the file and understand what each case checks.

Its assertion list is the longest of the three. The expected outputs documentation lists 44 assertion types outside the model-graded group. Those include schema checks for OpenAI tool calls, a tool-call F1 score, and four trajectory checks on traced agent runs: tool used, tool arguments, tool sequence and step count.

For our suite that covers both deterministic rows directly. The policy case becomes a negated trajectory check, because the documentation states that every assertion type can be negated with a "not-" prefix. The refund tool must not appear in the trace, and the escalation tool must.

The trajectory checks have one dependency worth knowing before you start. They read a traced agent run, so your agent has to emit traces Promptfoo can parse. A plain chat completion without tracing only gets the output-level checks.

How Promptfoo handles non-deterministic output

The command-line reference documents a repeat flag that runs each test a set number of times, and a flag that turns off the disk cache. Caching matters more than it looks. A cached response replayed three times is one sample, not three.

Scoring is weighted. Each assertion carries a weight that defaults to 1, and a test case can set a threshold on the combined weighted score. The eval command exits with code 100 when a test fails or the pass rate falls below a configured threshold, which is exactly the signal a CI job needs.

The official GitHub Action runs a before and after comparison on any pull request that edits a prompt, then posts a comment linking to the results. Of the three tools, this is the most finished CI story, and it is the reason Promptfoo wins our gating criterion.

The ownership question

Promptfoo announced on 9 March 2026 that it had agreed to be acquired by OpenAI. The founders wrote that Promptfoo will remain open source, and the repository states it remains MIT licensed. If you are testing whether to move away from an OpenAI model, you are now running that test on OpenAI-owned tooling.

I do not think that is disqualifying. An MIT licence means the code you depend on today can be forked tomorrow, by you or by anyone else. It is worth one line in your vendor risk register, and worth re-reading the licence at every major version.

DeepEval: pytest habits, 50 metrics and a lenient default

DeepEval describes itself as similar to pytest but specialised for unit testing LLM apps. Install is pip, tests are Python functions, and runs go through its own test command rather than plain pytest. A Python team is productive in it quickly, because nothing about the shape of the work is new.

The metrics documentation claims more than 50 ready-made metrics. For agents it lists trajectory metrics such as Task Completion, Step Efficiency and Plan Adherence, plus component-level Tool Correctness and Argument Correctness. It also covers multi-turn conversations, such as role adherence and knowledge retention, which a support agent needs and the other two tools handle less directly.

Tool Correctness is mostly deterministic

Most DeepEval metrics use an LLM as judge, and the documentation says so plainly. Tool Correctness is the documented exception. Its core score compares the tools called against the expected tools, with no model involved.

An optional second pass asks a model whether the tool choice was optimal, and the final score is the lower of the two. Leave out the list of available tools and that second pass never runs. That makes Tool Correctness the right metric for our tool selection row.

Argument Correctness, by contrast, is judged by an LLM with no reference answer. It belongs in the signal column of your suite, not the gate column.

The 0.5 default and the cloud pull

Every DeepEval metric passes at a score of 0.5 or above unless you change it. That default suits exploration. For a release gate it is too lenient, and I would set an explicit threshold on every metric before the first CI run. The strict mode option forces a binary score and raises the threshold to 1.

For variance, the flags reference documents a repeat flag that reruns each test case, a parallel flag that sets the number of processes, and a cache flag. Those three cover the mechanics of measuring noise.

The README calls a Confident AI account "highly recommended". The CI documentation is clearer that the API key is optional and only sends results to the cloud. Official regression baselines, shared reports and online evaluation on live traffic do sit on the hosted side, so expect the product to keep nudging you there.

Ragas evaluation: strongest on retrieval, thinnest on gating

Ragas began as a research project. The 2023 paper by Es, James, Espinosa-Anke and Schockaert introduced reference-free evaluation for retrieval augmented generation, and its faithfulness and context metrics are still the reason most teams install it. The repository now sits under the VibrantLabs organisation and is Apache 2.0 licensed.

For our grounded-answers row, Ragas has the deepest bench: faithfulness, context precision, context recall, response relevancy and noise sensitivity. If your support agent's weakest point is retrieval rather than tool use, this is the vocabulary you want. The diagnosis side of that problem is covered in the guide to diagnosing retrieval failures.

The tool call metrics are rule-based

The Ragas agent metrics page lists Topic Adherence, Tool Call Accuracy, Tool Call F1 and Agent Goal Accuracy. Tool Call Accuracy takes no judge model. Its score multiplies argument accuracy by whether the call sequence aligned, and the documented example scores 0.66 when 2 of 3 arguments match.

That is a clean deterministic check for our first row. Agent Goal Accuracy returns a binary 0 or 1 and uses a judge model, so it fits the scope row as a signal. Topic Adherence also needs a judge and a predefined list of allowed topics.

No pass or fail gate

Here is the gap. We found no built-in threshold, no repeat flag and no GitHub Action in the Ragas documentation. Experiments run as a Python script and save timestamped CSV files. The quickstart offers a RAG template, with the agent evaluation template marked "Coming Soon".

None of that is a flaw in the metrics themselves. It means you write the gate yourself: a test that loads the results, compares each score to your threshold and exits non-zero. It is a short script, and it is one more script your team now maintains.

Scoring the three AI testing frameworks side by side

The ratings below come from documentation and configuration surface, read on 9 October 2026, against the 12-case suite. They are judgements, not benchmark measurements, and the limits of that method are set out in the next section.

Popularity follows roughly the same order as the gating rows. On 9 October 2026 the three GitHub repositories showed 25.8 thousand, 18.7 thousand and 16.0 thousand stars for Promptfoo, DeepEval and Ragas respectively.

Stars measure attention, not fitness for your suite. They do say something about how fast questions get answered and how long a project is likely to be maintained, which matters for a tool that sits in your release path.

Where this comparison is weakest

The brief for this post asked for timed setups and a public repository of versioned configs. Neither is here, and that should be stated before anything else in this section.

We did not run the suite

Every rating in this post comes from documentation, release notes and repository pages read on 9 October 2026. We did not time an install, run the 12 cases or measure pass-rate variance across repeats. A tool can document a feature that behaves badly in practice, and documentation lags code in both directions.

A hands-on run would likely change some ratings. Setup friction in particular only shows up when a real agent, real API keys and a real CI runner meet for the first time. Treat the scorecard as a shortlist, not a verdict.

Judge models are not neutral instruments

Half of the suite depends on judge models, and all three tools default to judges for their headline metrics. The same Zheng paper that reported 80% agreement also documented position, verbosity and self-enhancement bias. Changing the judge model can move a score as much as changing the model under test.

Evan Miller of Anthropic argued in a 2024 paper on evaluation statistics that test questions should be treated as samples from a larger population, with error bars reported alongside scores. None of the three tools reports a confidence interval by default. On a 12-case suite, one flipped test moves the pass rate by more than 8 points, and noise alone can flip one.

The strongest counter-view is that framework choice barely matters. A small team that runs any of the three tools every week beats a team that compares frameworks for a month. I agree with that, which is why the verdict below is short.

Verdict by team size

My default for a support agent is Promptfoo for the gate and Ragas faithfulness as a non-blocking signal. That split keeps the blocking checks deterministic, and keeps the judge scores visible without letting them fail a build on noise.

The trade-off is real. A Python team writing complex policy checks will find YAML with inline JavaScript clumsier than a pytest file, and DeepEval wins that case outright. Whichever you pick, check whether a passing suite actually predicts production behaviour, the gap examined in the production readiness tests for AI pilots.

Frequently asked questions

Where to start

Pull the last 20 tickets your agent handled badly and sort them into the four failure types in the suite table. If one type holds most of them, that type decides your tool, not this comparison.

Then write the policy-boundary case first, as a code-graded check, in whichever tool you chose. Run it with repeats and caching off. If it ever passes and fails on the same model, you have found your noise floor before a customer found it for you.

Related reading