From Sanskriti Khandelwal | Product & Market Analysis
Retrieval Augmented Generation Did Not Stop Fake Citations in Court
On this page
A public database of court decisions now lists 2,202 cases where a judge found that a filing relied on AI-hallucinated material. When we searched it for the three big legal research brands, 28 decisions named a Westlaw, CoCounsel or Lexis tool. Those products are built on retrieval augmented generation, the architecture sold to lawyers as the cure for invented case law. RAG hallucination is no longer a lab result. It is a docket entry.
Key takeaways
- Grounded legal tools appear in the sanctions record by name. 28 of the 2,202 decisions in the AI Hallucination Cases database name a Westlaw, CoCounsel or Lexis product, and 17 of those 28 were decided in 2026.
- A real citation and a supported claim are different promises. Retrieval makes it likely that a cited case exists. It does not make the case say what the brief claims it says, which is where many of these filings failed.
- The failure is often misgrounding, not invention. Stanford researchers found leading legal RAG tools hallucinated on 17% to 33% of benchmark queries, frequently by citing a real source for a proposition it does not support.
- Courts sanction the signer, not the software. Penalties in the 28 cases range from warnings to $31,100 and removal from a case, and no judge treated the tool's marketing as a defence.
The short answer. Retrieval augmented generation reduces fabricated citations but does not eliminate them. Court records show lawyers sanctioned after using tools from Thomson Reuters and LexisNexis, and the most common failure is a real authority cited for something it does not say. Grounding moves the error from invention to misattribution. It does not remove the need to read the source.
What the AI hallucination court database actually counts
The AI Hallucination Cases database is maintained by Damien Charlotin, a legal researcher who has tracked these decisions since 2023. It is the most complete public record of AI hallucination court cases, and it is the source most press coverage quietly draws on.
Its inclusion rule is narrow, which is what makes it useful. A case enters only when a court or tribunal has found, or clearly implied, that a party relied on hallucinated content. Mere allegations are excluded. A small number of decisions where AI use was alleged but not confirmed are included and labelled as such.
That rule means the database undercounts the problem by design. It records the fabricated citations a judge caught and wrote about, not every fabricated citation filed. The real volume of hallucinated material in court filings is larger, and nobody can measure it.
Who the 2,202 decisions involve
The filters on the database page give a breakdown by party. Self-represented litigants account for 1,267 decisions, and lawyers for 872. Judges appear in 32, experts in 17. The United States accounts for 1,501 of the total, Canada for 234 and Australia for 112.
The nature of the error is also tagged. Fabricated material appears in 1,806 decisions, misrepresented material in 933 and false quotations in 587. A single decision often carries several tags, which is why those numbers sum to more than the total.
28 decisions name a grounded legal research tool
We searched the database on 20 October 2026 for three strings: Westlaw, CoCounsel and Lexis. The searches returned 14, 8 and 14 rows. After removing duplicates and one false match on a party named Alexis, 28 distinct decisions remain.
Of those 28, 26 involve lawyers and 2 involve self-represented litigants. In 21 the AI tool field names a legal research product alone. In the other 7 it names a legal product alongside a general chatbot such as Gemini, Claude or ChatGPT, or alongside tools the court could not identify.
That second group matters. A brief built with several tools cannot be cleanly blamed on any one of them. We report the full 28, and we report the split, so you can discount the mixed cases if you prefer.
One of the 28 decisions is from 2024, 10 are from 2025 and 17 are from 2026, with the year not yet over. If retrieval had solved the problem for professional users, this line would be flat or falling as the tools matured. It is doing the opposite.
The table below selects eight of the 28. They were chosen to show the range of failure and outcome, not the worst cases. Every row is drawn from the database entry for that decision.
| Case and court | Tool named | What the court found | Outcome |
|---|---|---|---|
| Lacey v. State Farm, C.D. Cal., May 2025 | CoCounsel, Westlaw Precision, Google Gemini | About 9 of 27 citations wrong, at least 2 cases nonexistent | Briefs struck, $31,100 against two firms |
| Gonzalez v. Texas Taxpayers, W.D. Tex., Jan 2025 | Lexis Nexis's AI | 4 fabricated authorities, AI use first denied | Response struck, $3,961 in fees and costs |
| Flowz Digital v. Dalal, C.D. Cal., May 2025 | Lexis+ AI | 1 fabricated, 2 misrepresented authorities | Monetary sanction, bar referral |
| Pelishek v. City of Sheboygan, E.D. Wis., Sep 2025 | Westlaw Quick Check and AI case search | 9 misrepresented authorities, 3 false quotes | $4,500 sanction |
| United States v. Farris, 6th Cir., Apr 2026 | Westlaw CoCounsel | Fabricated doctrine and 2 false quotations | Counsel removed without pay, bar referral |
| Thompson v. County of San Diego, Cal. Ct. App., Jul 2026 | Lexis+ with Protégé | 6 false quotes, 4 misrepresented, no invented case | Counsel withdrew the material |
| In re Rosslyn2016, Bankr. S.D. Tex., Jul 2026 | Westlaw Precision | 3 fabricated, 2 misrepresented authorities | Civil contempt, $29,877 |
| Mercante v. Tarzia, E.D.N.Y., Oct 2026 | Westlaw AI | A cited decision proved real but miscited | Rule 11 violation, no sanction |
Tool attributions are as recorded in the database, which usually takes them from counsel's own declaration. Several of these entries carry the database's "Vendor Disputed" flag. A named tool is an admission by the lawyer, not a finding of product fault.
What retrieval augmented generation promised lawyers
Retrieval augmented generation is a design where the model first searches a defined body of documents, then writes its answer from what it retrieved. In legal research that body is a vendor's database of cases, statutes and commentary. The pitch is simple: the model cannot invent a case if it can only cite what it found.
The vendors said so plainly. The Stanford paper collected the claims. LexisNexis said Lexis+ AI delivered "100% hallucination-free linked legal citations". Casetext said CoCounsel "does not make up facts, or 'hallucinate'". A Thomson Reuters executive said retrieval "dramatically reduces hallucinations to nearly zero."
Read the LexisNexis wording again. The guarantee covers linked citations, meaning the link resolves to a real document. It says nothing about whether that document supports the sentence it is attached to. In the Flowz Digital case, counsel quoted that very marketing claim back to the court after filing a fabricated authority.
The benchmark that tested the promise
In 2024, researchers at Stanford's RegLab and Institute for Human-Centered AI built a preregistered set of over 200 legal queries and ran them through the leading tools. They defined a hallucination as a response that is incorrect, or that falsely asserts a source supports a proposition.
The resulting paper found that Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated between 17% and 33% of the time. Lexis+ AI answered 65% of queries accurately, and Westlaw's tool 42%. Both were better than a general chatbot. Neither was close to the marketing.
We covered how legal-specific tools compare with general assistants on accuracy in the Harvey, Legora and general AI scorecard. The court record adds something a benchmark cannot: what happens when those error rates meet a real deadline and a tired associate.
Where the RAG retrieval chain broke in court
A grounded answer passes through several stages before it reaches a judge. Each stage has its own failure, and the court record shows all of them. Mapping them matters because each needs a different control.
Retrieval returned something close, not something right
The Stanford authors argue that legal queries often do not have a single clear answer, and that retrieval in law is unusually prone to textually similar but irrelevant documents. A case about the same statute in a different posture looks relevant to a search system. It is not relevant to a judge.
Their benchmark also showed one tool failing to correct a user's mistaken premise. If the question assumes a rule that does not exist, a system built to be helpful will often find something that looks like support for it.
Generation attached the wrong words to a real source
This is the failure that grounding was supposed to prevent, and it is the most common one among the 28. In Thompson v. County of San Diego, the database records six false quotations and four misrepresented authorities, and no invented case at all. In Mercante v. Tarzia, a decision the court first believed fictitious turned out to exist but had been miscited.
Coverage of the Farris appeal reported that the brief described one real decision as reversed when it had been upheld. That is not a fabrication a link checker can catch. The link works. The sentence is false.
The handoff stripped out the warning
Lacey v. State Farm shows the organisational failure. According to the ABA Journal's account of the order, one lawyer used CoCounsel, Westlaw Precision and Google Gemini to build an outline. Colleagues at a second firm turned it into a brief without knowing AI had been involved.
Special Master Michael Wilner flagged two bad citations. The firm refiled with those two fixed, and other AI-generated problems remained. Wilner wrote that he was persuaded by the authorities, looked them up, and found they did not exist. "That's scary," he wrote.
What courts sanctioned in the RAG hallucination cases, and what they did not
Across the 28 decisions, a monetary amount is recorded for 11. Those amounts run from $1,001 to 55,597, with a median of $4,000. The rest ended in warnings, admonishments, bar referrals, struck briefs or withdrawal of the material.
The money is rarely the real penalty. In Farris, appointed counsel lost his compensation for the appeal and was removed from the case. In Heimkes, the lawyer was reprimanded, referred to the bar and ordered to file the decision in later cases. A bar referral follows a lawyer for years. A $2,500 fine does not.
No court accepted the tool as a defence
We found no decision among the 28 where naming a reputable legal research product reduced the lawyer's responsibility. In Hill v. Workday, counsel's declaration blamed a Westlaw citation copying feature. The court called the explanation improbable and said the citation bore the hallmarks of an AI hallucination.
The rule courts apply is the old one. The person who signs the filing certifies that the authorities exist and say what is claimed. Which software produced the draft is irrelevant to that certification.
A written AI policy did not save the lawyers either
The clearest test of process came in Johnson v. Dunn, a 2025 case in the Northern District of Alabama involving ChatGPT rather than a legal tool. The firm had adopted an AI policy in June 2023 requiring written approval and independent verification, according to EDRM's analysis of the order.
The policy protected the firm, which was not sanctioned. It did not protect the three lawyers, who were reprimanded, disqualified from the case and referred to the state bar. The court found a fine and public reprimand insufficient. A policy that nobody follows under deadline is a liability shield for the institution, not a control.
Why RAG hallucination looks different from chatbot hallucination
General chatbot errors in court tend to be total inventions: a case name, a reporter citation and a holding that never existed. Those are embarrassing but easy to catch. Any citation checker flags a case that is not in the database.
Grounded tool errors are harder. The case exists. The citation resolves. The quotation is close to something in the opinion, or the holding is a plausible reading of a different part of it. Our audit of deep research tools found the same pattern outside law: dead links are the visible failure, and unsupported claims attached to live links are the expensive one.
That shift changes what a reviewer has to do. Checking that a citation exists takes seconds. Checking that a case supports a proposition means reading the case. The tools reduced the first job and left the second untouched, while their marketing implied both had been handled.
Across all 2,202 decisions, misrepresented authorities appear in 933 and false quotations in 587. Those are the categories where a real source was used badly. They are not the majority, but they are a large minority, and they are the categories retrieval is least able to fix.
Where this argument is weakest
We think the record is clear enough to act on. It is also weaker than the headline number suggests, in four specific ways.
The tool attribution is self-reported. Most of the 28 tool names come from a lawyer's declaration after being caught. A lawyer has an incentive to name a reputable product rather than admit pasting from a free chatbot. Several entries carry the database's Vendor Disputed flag, and the database does not resolve those disputes.
There is no denominator. We do not know how many briefs were drafted with these tools and contained no errors. 28 failures across millions of research sessions would be an excellent safety record. The court data cannot tell you the rate, and anyone who claims it does is guessing.
The benchmark is ageing. The Stanford tests ran on 2024 versions of these products, and the vendors disputed parts of the method. Thomson Reuters argued that one of the tools tested was not designed for that kind of research. The products have shipped new versions since.
Most rows name no tool at all. On the database's first page, nearly every entry records the tool as implied or unidentified. The 28 is a floor, but it is also possible the grounded tools are under-represented because careful users of them catch more errors before filing. We cannot separate those effects.
What survives all four caveats is narrow but important. Grounded tools can produce filings that get lawyers sanctioned, and the errors they produce are the kind that pass a quick check.
What the court record means for your enterprise RAG deployment
Law is not your industry, probably. It is still the best public test of retrieval augmented generation under consequences, because courts publish their findings and name the tools. Your internal RAG system faces the same failure points with less scrutiny.
The fix is not a better retriever alone. Our piece on diagnosing RAG retrieval failures covers the search side, and the hybrid search and reranker benchmarks show what retrieval upgrades buy. Neither touches the generation and handoff stages, which is where the court cases failed.
| Stage | What failed in the court record | Control for your deployment |
|---|---|---|
| Query | False premises answered rather than corrected | Include false-premise questions in your evaluation set |
| Retrieve | Similar but inapplicable authorities returned | Measure retrieval relevance separately from answer quality |
| Generate | Real sources attached to wrong quotes or holdings | Exact-match every quotation against the source text automatically |
| Hand off | Downstream reviewers unaware AI drafted the material | Tag AI-drafted passages so the tag travels with the document |
| Sign off | No one read the cited source before it went out | Require a named human to open each load-bearing source |
Test claims, not citations
Most RAG evaluation checks whether the answer cites a retrieved document. That is the LexisNexis guarantee, and the court record shows it is not enough. Your evaluation suite needs a claim-support check: does the cited passage actually state what the answer says it states? Our guide to building an LLM eval suite covers how to score that.
Quotations are the cheapest place to start. A quote either appears verbatim in the source or it does not, and a script can check it. Six of the Thompson errors were false quotations, so a single string comparison would have caught them all.
Put the human at the right stage
A human reviewer at the end of the chain who only skims for tone adds nothing. The review has to be the source-reading kind, and it has to be assigned to a named person. The design questions are covered in our piece on human-in-the-loop architecture.
Our view is blunt. If your team cannot afford the time to read the sources behind the outputs that matter, it cannot afford to ship those outputs. Grounding lowers the error rate. It does not change who answers for the errors.
Frequently asked questions
Does retrieval augmented generation prevent hallucinations?
No. Retrieval augmented generation reduces hallucinations but does not prevent them. Stanford researchers found leading legal RAG tools hallucinated on 17% to 33% of benchmark queries, compared with higher rates for general chatbots. Court records also show lawyers sanctioned after using Westlaw, CoCounsel and Lexis tools. The common failure is a real source cited for a claim it does not support.
How many AI hallucination court cases are there?
The AI Hallucination Cases database maintained by Damien Charlotin listed 2,202 decisions worldwide as of 19 October 2026. It counts only decisions where a court found or implied reliance on hallucinated material. The United States accounts for 1,501. Self-represented litigants appear in 1,267 decisions and lawyers in 872. The true number of affected filings is higher.
Have lawyers been sanctioned for using Westlaw or Lexis AI?
Yes. A search of the Charlotin database on 20 October 2026 found 28 decisions naming a Westlaw, CoCounsel or Lexis tool. Examples include Lacey v. State Farm, with $31,100 in sanctions against two firms, and United States v. Farris, where appointed counsel lost his fee and was removed. Tool names usually come from the lawyer's own declaration.
What is the difference between a fabricated citation and a misgrounded one?
A fabricated citation points to a case or source that does not exist. A misgrounded citation points to a real source but attributes a quote or holding to it that the source does not contain. Retrieval largely solves the first problem and barely touches the second. Misgrounded errors are harder to catch because the citation resolves when checked.
What are the main RAG limitations for enterprise use?
The main RAG limitations are retrieval of plausible but wrong documents, generation that misstates what a real source says, failure to correct false premises in a question, and loss of context when AI-drafted material passes between people. Better retrieval fixes only the first. The others need claim-level evaluation, automatic quote checks and named human review.
Can a company blame its AI vendor if grounded AI gives a wrong answer?
In court, no decision among the 28 we reviewed reduced a lawyer's responsibility because a reputable legal research tool produced the error. The person who signs the filing certifies its accuracy. In commercial settings your remedy depends on the contract, and most vendor terms disclaim accuracy. Plan on owning the output.
Where to start this week
Pull 20 recent answers from your own RAG system that cite a source. For each one, open the source and check whether it actually says what the answer claims. Count the misses. That number is your real hallucination rate, and it is almost certainly higher than the one on your vendor's slide.
Then add one automated check before anything leaves the building: every quotation in an AI-drafted document must match its source text exactly. It takes an afternoon to build, and it would have caught a meaningful share of the errors that ended up in court.
Related on evaluation
The tooling for claim-level checks is compared in Promptfoo, DeepEval and Ragas side by side.
References
- Damien Charlotin, AI Hallucination Cases database, last updated 19 October 2026. Used for the 2,202 total, party, jurisdiction and category counts, and every case-level detail in the selected decisions table.
- Zan Digital analysis of the same database, 20 October 2026. Text searches for Westlaw, CoCounsel and Lexis, de-duplicated by hand. Used for the 28 decisions, the year split, the tool split and the penalty median.
- Magesh, Surani, Dahl, Suzgun, Manning and Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Stanford RegLab and HAI, first posted 30 May 2024, since updated. Used for hallucination and accuracy rates and the quoted vendor claims.
- ABA Journal, Judge imposes sanctions for "collective debacle" involving AI hallucinations and 2 law firms, 14 May 2025. Used for the Lacey tools, citation counts and the Wilner quotes.
- EDRM, When AI policies fail: the AI sanctions in Johnson v. Dunn, 31 July 2025. Used for the firm policy and the sanctions imposed.
The weakest part of this source base is tool attribution. It rests on lawyers' own statements after being caught, several entries are disputed by vendors, and there is no public denominator of error-free filings. The Farris detail on a reversed decision comes from secondary coverage, not the opinion itself.
Related reading