LLM Observability: LangSmith, Braintrust and Arize Bill Three Different Units

LangSmith bills you for a trace. Arize bills you for a span. Braintrust bills you for gigabytes of processed data and for every score you write. A single agent run that calls a model, a tool and the model again is one billable unit in the first tool and four in the second. LLM observability vendors have converged on features and diverged on units, which makes the feature grid the least useful page in any of their docs.

What LLM observability actually covers in 2026

The category name covers three jobs that used to belong to three different teams. Buying decisions go wrong when a team compares vendors on one job and then gets billed on another.

Tracing is the record, evaluation is the verdict

Tracing captures what your system did. Each model call, retrieval step and tool invocation is recorded with its inputs, outputs, latency and token counts. That record is what you open at 3am when a customer forwards a bad answer.

Evaluation is a separate job. It scores an output against a rubric, a reference answer or a judge model, and it produces a number you can track across releases. Tracing tells you what happened. Evaluation tells you whether what happened was acceptable.

Monitoring is the third job. It watches those scores over time and raises an alarm when behaviour moves. All three vendors here do all three jobs, and they have done so for at least a year. They disagree about which job you start from, and that disagreement is the real product difference.

One thing none of them do is tell you whether the product is worth building. They tell you whether it behaves the way it did last week. Teams that expect an observability purchase to settle a return question are asking the wrong system, and the right test is a production readiness check run before the pilot ships.

LLM observability pricing: three tools, three billing units

Here is the comparison that vendors do not print, because printing it would make their prices hard to rank against each other.

Figures are the vendors' own published pricing pages as of 28 August 2026. Enterprise tiers on all three are custom and are not represented here. Free tiers exist on all three and are left out so the paid comparison stays like for like.

LangSmith bills traces and seats

LangChain's documentation defines a run as a single unit of work, such as a model call or a retrieval step. A trace is a collection of runs for a single operation. The docs say plainly that if you already know OpenTelemetry, a run is a span.

That makes the LangSmith unit the coarsest of the three. A deep agent loop with 30 tool calls still bills as a single trace. Seats then carry the rest of the cost, at $39 each per month on the Plus plan, so your bill scales with the size of your team as much as with your traffic.

Per-trace overage is the part worth checking against your own invoice. The LangChain usage and billing docs state a base charge of .05 cents per trace, with extended retention traces at 10 times that. Read as dollars, that is $0.50 per 1,000 base traces and $5.00 per 1,000 extended traces.

Braintrust bills bytes and scores

Braintrust meters processed data in gigabytes and meters scores separately. Starter includes 1 GB and 10k scores, then charges $4 per GB and $2.50 per 1,000 scores. Pro is $249 per month with 5 GB and 50k scores, then $3 per GB and $1.50 per 1,000 scores.

Seats are free on every tier, which inverts the LangSmith shape completely. A team of 20 pays the same $249 as a team of 3, while a verbose retrieval pipeline pays far more than a terse classifier at identical request volume.

The second meter is the one teams underestimate. Every automated judge you run writes a score, so eval coverage becomes a direct cost driver rather than a free byproduct of tracing. That is a defensible way to price a quality tool, and it does mean the cheapest month is the month before you take evaluation seriously.

Arize bills spans and storage together

Arize AX Free includes 25k spans, 1 GB and 15-day retention. AX Pro is $50 per month for 50k spans, 10 GB and 30 days. Enterprise adds SaaS or self-hosted deployment on custom terms.

Spans are the finest unit in this comparison, which is honest instrumentation and unforgiving arithmetic. The same workload that reads as 50k traces elsewhere can read as several hundred thousand spans here, and neither number is wrong.

One gap is worth naming. The Arize pricing page publishes the included volumes but not the overage rates, so the only way to model a bill past the caps is to ask sales. Third-party roundups quote per-span overage figures that disagree with each other by orders of magnitude, which is why none of them appear in this post.

Retention is the setting that decides your drift budget

Every vendor in this category sells drift detection. None of them sell it at the default retention window, and this is the most expensive small detail on the whole pricing page.

Drift is a comparison between two periods. If your retention is 14 days, the longest comparison available to you is a fortnight, and slow degradation across a quarter is invisible by construction. What actually moves underneath a deployed system, and on what clock, is covered in the piece on agent drift and production monitoring.

LangSmith's answer is extended retention at 400 days for 10 times the base trace price. Braintrust extends beyond the included period at $0.50 per GB per month. Arize sets retention by tier and takes anything longer to a contract. Three different answers, one shared assumption, which is that long memory is a premium product.

My position is that you should price the long window on day one rather than after the first incident. Teams almost always buy the entry tier, then discover during a post-mortem that the evidence they needed expired 11 days ago. The upgrade is cheap in advance and worthless in arrears, because retention is not retroactive on any of these platforms.

How each tool shapes the way your team debugs

Feature lists converge because vendors read each other's changelogs. Workflows diverge because each product decides what the first screen is, and the first screen decides what a new engineer learns to look at.

LangSmith starts from the graph

If your application is built on LangChain or LangGraph, the trace view mirrors the code you already wrote. You open a run, expand the graph, and read the prompt at the node that misbehaved. Setup cost is close to zero, because the instrumentation ships with the framework you are already importing.

The cost of that fit is gravity. A team that debugs through the framework's own lens tends to keep building inside the framework, which is comfortable until the day you want to move one component out of it.

Braintrust starts from the score

Braintrust treats evaluation as the product and tracing as the supporting evidence. The daily loop is a dataset, a scoring function, and an experiment comparison between two prompt versions. That shape suits teams that gate releases on a regression suite rather than on an engineer's judgement.

It also fits the way the pricing works. Unlimited seats mean product managers and domain reviewers can sit inside the same tool as the engineers, which is where human review belongs in the architecture rather than in a spreadsheet beside it.

Arize and Phoenix start from the span

Arize came out of the machine learning monitoring world and its instinct is infrastructure. Phoenix, the source-available sibling, runs on a laptop or in your own cluster, is built on OpenTelemetry, and uses the OpenInference conventions that Arize maintains.

That is the right starting point for a team that already runs an observability stack and wants LLM spans to land beside everything else it monitors. It is the wrong starting point for a product team that wanted a scoring workflow by Tuesday.

Portability and the OpenTelemetry escape hatch

Every vendor in this category now says the word OpenTelemetry, and the claim is broadly fair. Instrumentation you write against the standard can, in principle, be pointed somewhere else later.

The detail underneath is less settled than the marketing suggests. The GenAI semantic conventions moved out of the main OpenTelemetry semantic-conventions repository during 2026 into a dedicated repository whose README still carries a TODO where the schema URL belongs.

Read that as a signal about maturity rather than as a reason to avoid the standard. The direction of travel is clearly toward a shared vocabulary for model calls, agent steps and tool invocations. What does not exist yet is a versioned contract you can pin a build against, which is the same pattern visible in the way agent interoperability standards get adopted ahead of their own stabilisation.

The practical consequence is narrow and specific. Portability protects your instrumentation code. It does not protect your history, your datasets or your scoring definitions, and those three are the assets that make switching painful 18 months in.

So put the export in the contract rather than in the evaluation notes. Ask each vendor for a bulk export of traces, datasets and scores in a documented format, run it once during the trial, and check that scoring definitions come out alongside their results. A vague answer here is a price increase you have not been quoted yet, and it arrives at the exact moment your negotiating position is weakest.

What the evaluation layer cannot tell you

All three tools will run a model as a judge over your outputs. That capability is what turns tracing into a quality system, and it is also where the strongest claims in this category get overstated.

Judges agree with humans mostly on questions they can already answer

Research on automated grading is direct about the limit. In No Free Labels, Krumdick and colleagues built a benchmark of 160 financial questions and 1,200 model responses graded by financial experts.

Judges came out more reliable than the other automated grading methods tested. They also showed high agreement with the experts mainly on questions the judge model could answer correctly itself, and supplying judges with expert-written reference answers largely closed that gap.

The operational reading is straightforward. A judge without human grounding measures the judge as much as it measures your system, and the fix is a small set of expert-written references rather than a longer prompt. That is the discipline behind building an eval suite that survives contact with production, and it is work no vendor performs on your behalf.

Where this comparison is weakest

Three things in this post could mislead you, so here they are before you act on any of it.

First, list prices are not the prices serious teams pay. Every vendor here routes real volume to a custom Enterprise contract, and published rates are best read as an anchor rather than as a forecast. If your workload is large enough for this decision to matter financially, the numbers above are the wrong numbers.

Second, the sourcing on trace pricing is genuinely messy. Several third-party pricing roundups state that LangSmith base traces cost $2.50 per 1,000, while LangChain's own documentation states .05 cents per trace with extended retention at 10 times that. Those claims cannot both hold, because the roundup figures imply a 2 times multiple where the vendor states 10. This post uses the vendor documentation and flags the disagreement rather than averaging it. The same failure explains why credit and token billing units get restated wrongly across the software press.

Third, workflow fit is a judgement, not a measurement. I can show you that Braintrust starts from a score and that LangSmith starts from a graph. I cannot tell you which one your team will open at 3am, and a two-week trial on your own traffic answers that better than any comparison post, including this one.

A selection sequence you can run this week

Run these in order. The first step disqualifies more vendors than the other four combined.

Teams skip step 1. It is the only step that converts three incomparable price lists into one comparable number.

Frequently asked questions

Where to start

Do the arithmetic before the trial, because a trial without it measures how nice the interface is.

Instrument one representative user journey and count what it emits. Record the number of spans, the number of traces and the payload size in kilobytes for a single request. That one measurement turns three incompatible price lists into three comparable monthly numbers, and it takes an afternoon.

Then decide who owns the alerts before you decide who hosts the data. An observability tool with no named owner produces dashboards nobody opens. That is a familiar pattern for anyone who has watched the operations role behind production agents get created after the first outage rather than before it.

Related reading