OpenAI Evals Shuts Down 30 November: 4 Destinations and What Each One Imports

OpenAI's hosted Evals dashboard and API shut down on 30 November 2026, and existing evals have been read-only since 31 October. That leaves 25 days from today to move every dataset and grader you still hold there. OpenAI points to Promptfoo, but its own migration guide says the move is a manual rebuild, not an export. The OpenAI Evals deprecation is a migration project, not a settings change.

What the OpenAI Evals shutdown actually removes

OpenAI Evals was the hosted evaluation product inside the OpenAI platform. You uploaded a dataset, attached graders, ran the eval against a model, and read the results in a dashboard or through the API. All of that is going away on one date.

The OpenAI deprecations page lists the Evals platform under a 3 June 2026 entry. It says existing evals "become read-only" on 31 October and that "the Evals dashboard and API are scheduled to shut down" on 30 November. It also notes that graders documented for eval workflows "are part of this transition," while fine-tuning timelines sit in a separate section.

The timeline in three dates

Count the days and the window looks generous. From announcement to shutdown is 180 days. The trouble is that most teams treat a deprecation notice as a reminder rather than a deadline. The read-only phase removes the option to fix things inside OpenAI before you leave.

The same 3 June entry list also schedules Agent Builder and the v1/prompts reusable prompts API to shut on 30 November. If your eval prompts were stored as reusable prompt objects, you have two migrations due on the same day. OpenAI's advice for the second is to move prompt content into your application code.

OpenAI's notice does not define what read-only permits. It does not say whether you can still trigger a run on an existing eval, or whether only edits are blocked. I would assume the narrow reading: you can view and download, and nothing else. Plan the export as if every eval is frozen, because the date has already passed.

Export checklist for OpenAI Evals data before 30 November

The OpenAI cookbook guide, Moving from OpenAI Evals to Promptfoo, does not describe an export. Its words are blunt: "This guide covers manually recreating an evaluation in Promptfoo. It does not require an OpenAI Evals export feature". So the checklist below is about preserving what you will need to rebuild, whichever destination you choose.

Eval definitions and grader configs

The Evals API reference documents a "List evals" endpoint at GET /evals with pagination parameters. Page through it and save every eval object as JSON. Each object carries the data source config and the testing criteria, which is where your graders live.

Save the grader blocks verbatim, including model names, prompts, pass thresholds and label sets. A score model grader is only as portable as its prompt text. If you lose the prompt, you lose the definition of "good" that your team argued over for months.

Datasets and run history

Download every dataset file referenced by an eval's data source. JSONL is the format the four destinations below share, so keep the raw JSONL rather than a spreadsheet copy. Record which eval each file belonged to, because filenames rarely say.

Run history is harder. In the portion of the API reference we could read, run creation and cancellation were documented. But we could not confirm the endpoints for retrieving runs and per-item outputs. Test retrieval against one eval today. If it fails, screenshot the summary scores of your last good run, because that number is your regression baseline in the new tool.

How OpenAI grader types map to each destination

The Evals API reference lists 5 grader types in an eval's testing criteria. They are string check, text similarity, score model, label model and Python. String check supports four operations: eq, ne, like and ilike. Every one of these needs an equivalent in the new tool, and none of the four destinations publishes a grader-by-grader mapping.

The table below is our mapping, built from each tool's official documentation on 5 November 2026. It is a starting point for the rebuild, not a guarantee that scores will match.

Mapping by Zan Digital from vendor documentation, 5 November 2026. The label model row is the weakest: no vendor documents a direct equivalent, so each entry is the closest mechanism, not a like-for-like port.

Why model-graded scores will drift

The cookbook itself warns that "similarity-based scoring may not produce identical numerical results across systems". It adds that any recreated grader, "especially an LLM-as-a-judge grader, should be validated before you rely on it for regression decisions". I agree, and I would go further: assume every model-graded threshold is wrong until you have rerun your last good dataset and compared. The reasons judges disagree with each other are covered in the review of LLM judge agreement and bias studies.

Promptfoo, the destination OpenAI points to

Promptfoo is an open-source command-line tool and library for testing prompts, models and agents. Its GitHub README says it "is now part of OpenAI" and "remains open source and MIT licensed". OpenAI announced the acquisition on 9 March 2026, three months before the Evals deprecation notice.

That sequence matters for your risk view. The company shutting your eval tool owns the tool it recommends. That is convenient, because the migration guide and the destination come from the same place. It also means your evaluation layer is still tied to one model vendor's roadmap, which is the dependency many teams wanted to escape. The general cost of that kind of tie is worked through in the analysis of model switching costs and lock-in.

What the cookbook actually asks of you

The guide's instruction is to "recreate your evaluation's prompts, provider, test cases, and assertions in a promptfooconfig.yaml file". You then run promptfoo eval -c promptfooconfig.yaml --no-cache and open results with promptfoo view. It also says the new run "is separate from any previously completed OpenAI Evals runs," with no step for bringing old results across.

The data side is easy. Promptfoo's test case documentation accepts CSV, XLSX, JSON, JSONL and YAML, with each JSONL line holding vars and assert. Two of its model-graded assertions, factuality and model-graded-closedqa, are documented as using "OpenAI's public evals prompts," which helps teams whose graders started from those templates. Python assertions return a boolean, a float or a grading result object, so Python graders port with the least rewriting. A fuller head-to-head of Promptfoo as a test runner sits in the Promptfoo, DeepEval and Ragas comparison on one agent suite.

The paid side is opaque. Promptfoo's pricing page lists a free Community tier with red teaming capped at 10,000 probes a month, and Enterprise and On-Premise tiers at custom prices. If you need shared dashboards, SSO and permissions, you are buying a quote, not a list price.

Braintrust, LangSmith and DeepEval as OpenAI Evals alternatives

The other three destinations solve different problems. Braintrust and LangSmith are hosted platforms with dataset management and a review UI. DeepEval is a Python testing framework with a hosted layer, Confident AI, sold separately.

Braintrust: hosted experiments, flat team pricing

Braintrust stores dataset records with four top-level fields: input, expected, metadata and tags. Its docs show a UI upload for CSV and JSON with column mapping. There is also a CLI command, bt datasets create my-dataset --file records.jsonl, that accepts JSONL. Mapping an OpenAI row onto input and expected is a short script.

Its open-source scorer library, autoevals, says many of its evaluations "are adapted from OpenAI's" evals project. We counted 24 scorers in its scorer reference, including Factuality, ClosedQA, Levenshtein, ExactMatch and EmbeddingSimilarity. For label graders, the LLMClassifier with choice scores is the nearest mechanism.

LangSmith: the natural home if you already trace there

LangSmith's dataset documentation describes importing "an existing dataset from CSV or JSONL file" from the New Dataset screen. Datasets store inputs, outputs and optional reference outputs as arbitrary JSON, so OpenAI's JSONL rows load without reshaping. Evaluators come from the open-source openevals package, whose create_llm_as_judge takes a prompt and a model. It returns either a continuous score or one of a fixed set of choices.

That choices option is the cleanest documented fit for an OpenAI label model grader among the four tools. If your production traces already sit in LangSmith, keeping evals beside them saves a second vendor review. How its trace-based billing compares to Braintrust and Arize is covered in the LLM observability pricing comparison, so this post does not repeat it.

DeepEval: pytest-style evals for Python teams

DeepEval's dataset documentation supports loading JSON, JSONL, CSV and Hugging Face datasets into an evaluation dataset. A JSONL line carries input, expected_output and optional context. Its metrics page claims "50+" ready-made metrics, most of them LLM-as-a-judge, plus G-Eval and DAG for custom criteria.

One default needs attention. DeepEval passes a metric when the score is equal to or above a threshold that defaults to 0.5. An OpenAI score model grader with a pass threshold of 0.8 will silently loosen if you forget to set it. Write the old threshold into every metric explicitly.

LLM evaluation platform pricing at team scale

All four products have a free path. So the price question is really about the hosted layer: shared results, review queues, access control. On list prices retrieved 5 November 2026, the hosted tiers split cleanly into per-seat and flat-fee models.

Sources: Braintrust pricing, LangSmith pricing, Confident AI and Promptfoo pricing pages, all retrieved 5 November 2026. LangSmith lists pay-as-you-go overage without a per-trace rate on the page.

The seat crossover

Here is the original figure. LangSmith Plus at $39 per seat costs $234 a month for 6 seats and $273 for 7. Braintrust Pro is $249 flat with unlimited users. So the platform fee favours LangSmith up to 6 seats and Braintrust from the 7th. Against Confident AI's $200 Starter, LangSmith is cheaper up to 5 seats and dearer from 6.

Seat count is the variable most teams underestimate. Evals stop being an engineering artefact once product managers and domain reviewers start labelling outputs. A team that budgets for 4 engineers often ends up with 10 people who need to see results. On a per-seat plan that is $390 a month before any usage.

Verdict by team type for the OpenAI Evals migration

There is no single best destination. Any post that names one without asking about your team is guessing. The right choice depends on who reads eval results and where your traces already live.

My own pick for a team with fewer than 5 eval users and no existing tracing vendor is Promptfoo. The migration guide maps to it, the config lives in your repository, and you can move again later without asking a vendor for an export. That last point is the lesson of this deprecation, and it applies to every tool on this list. The steps for designing a suite you own end to end are in the guide to building an LLM eval suite.

Where this comparison is weakest

We did not migrate a real OpenAI eval into all four tools and compare scores. The grader mapping is built from documentation, not from runs. A mapping that looks clean on paper can still produce different pass rates, especially for label and score model graders. We cannot tell you by how much.

The API evidence is incomplete. We confirmed the list evals endpoint, but we could not confirm run and output-item retrieval in the reference we read. Some third-party write-ups describe an export menu that produces Promptfoo configs and results files. The current cookbook text describes only manual recreation, so we could not verify that path and did not build advice on it.

Pricing is the least stable part of this post. Three of the four vendors publish list prices, and all of them have changed tiers during 2026. Usage overage, which often decides the real bill, is excluded from the seat crossover. A team that runs large datasets daily could pay more in Braintrust score overage than in LangSmith seats, which reverses the verdict above.

Finally, the strongest counter-case to leaving OpenAI's orbit is simple. Promptfoo is OpenAI's own recommendation and is now OpenAI-owned, so it is the least likely destination to be surprised by future model or API changes. If your stack is OpenAI end to end, that alignment is worth something, and my lock-in concern may be overstated for you.

Frequently asked questions

Where to start before 30 November

Today, page through the list evals endpoint and commit every eval object and dataset file to a repository. This takes an afternoon, it is the only step with a hard deadline, and it keeps every destination open.

Then pick your single most important eval and rebuild it in your chosen tool first. Rerun it on the saved dataset and compare against your last OpenAI score. If the pass rate moves by more than a few points, fix the grader before you port anything else. Every later eval will inherit the same error.

Related reading