AI Testing Before Go-Live: What SR 11-7 Model Validation Looks Like for an LLM

For 15 years, every material model inside a large US bank went through the same validation routine before and after it went live. That routine, Federal Reserve guidance SR 11-7, was retired on 17 April 2026, and its replacement explicitly leaves generative AI out. The method is still the most complete public template for AI testing before go-live, and it maps onto an LLM deployment almost line for line.

The short answer

To apply SR 11-7 to an LLM, treat the whole deployment as the model. Write down why the design is sound, test it on inputs it has not seen, and compare it with a challenger. Check its outputs against expert judgement, have someone outside the build team challenge it, and keep monitoring after launch. Record all of it in one file.

SR 11-7 is retired, and it is still the best AI testing template available

The Federal Reserve and the Office of the Comptroller of the Currency issued SR 11-7 in April 2011. The FDIC adopted it in 2017.

On 17 April 2026 the three agencies replaced it. The Fed's SR 26-2 supersedes SR 11-7, and the OCC's Bulletin 2026-13 rescinds the 2011 bulletin along with the Model Risk Management booklet of its examiner handbook. The new text is principles-based, and the OCC states that non-compliance "will not result in supervisory criticism".

The OCC is explicit: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." What that exclusion means for data governance is covered in the five bank data controls worth copying for agents. This post deals with the other half of the old guidance: validation.

Three reasons to borrow it anyway. First, SR 11-7 is the most heavily tested validation standard in public. Banks built entire departments around it.

Second, the regulators did not abandon the method. The 2026 guidance still lists "model validation and monitoring, including validating conceptual soundness and outcomes analyses" among its core principles.

Third, other regulators are moving the opposite way on AI. Canada's OSFI Guideline E-23, finalised in September 2025, defines a model as including "AI/ML methods" and asks institutions to handle "the elevated potential for model drift". I would build to the stricter reading now.

How an LLM deployment fits the SR 11-7 definition of a model

The FDIC's 2017 adoption letter quotes the 2011 definition. It defines a model as "a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates". It split every model into three components: an information input component, a processing component and a reporting component.

An LLM produces text rather than quantitative estimates, which is one reason the 2026 definition excludes it. The three-part split still describes the system well, and it decides what your validation has to cover.

The 2011 guidance gave model risk two sources. The first is a model with fundamental errors that produces inaccurate outputs. In an LLM deployment that means a fabricated citation, a retrieval step that pulls the wrong policy, or a prompt that silently drops a constraint.

Your test plan should name which of these it is looking for. "The model is accurate" is not a testable claim. "The assistant cites a document that exists and supports the sentence" is.

The second source is a sound model used incorrectly or inappropriately. This is the more common LLM failure in my experience. A drafting assistant validated for internal summaries ends up answering customers, and nobody re-runs the tests.

The defence is a written scope with prohibited uses. It costs an hour to write, and no automated eval knows what the system was not meant to do.

The SR 11-7 to LLM evaluation mapping, requirement by requirement

The table below takes each element of the 2011 guidance and states what it means for a deployed LLM system. The last column points to the validation file section holding the evidence.

Middle column paraphrases the 2011 supervisory guidance as summarised by the FDIC and the agencies. The translation column is the author's mapping, not regulatory text.

Conceptual soundness for an LLM: justify the design before running a test

Conceptual soundness is the check most LLM teams skip, because it does not produce a score. A design that cannot be justified on paper will not be rescued by a good benchmark run.

The question is whether, given the task, the data and the risk, this is the right architecture. A pre deployment AI testing framework that starts with evals has already assumed the answer.

Write one paragraph for each decision a reviewer could reasonably question. Which model, and which alternatives were tried. Whether the system retrieves documents or relies on the model's own knowledge. What the system prompt instructs, and what it forbids. Which tools the model can call, and with what permissions.

Each paragraph should name its evidence. "We chose retrieval because the policy corpus changes monthly" is a rationale. "We chose retrieval because it is best practice" is not, and an independent reviewer should send it back.

Developmental evidence when you did not build the model

SR 11-7 assumed the bank could see how its models were built. With a hosted LLM you cannot. The 2011 guidance anticipated a version of this problem. The FDIC's adoption letter says it covers "the incorporation of vendor products into the institution's model risk management framework following the same principles as in-house models".

In practice that means the vendor's model card and published evaluations are inputs to your file, not substitutes for it. A general benchmark score says nothing about whether the model reads your refund policy correctly. Why public benchmarks cannot act as the release gate is set out in the guide to building an LLM evaluation suite.

Ongoing monitoring and outcomes analysis: two LLM evaluation stages teams merge by mistake

SR 11-7 separated three validation elements: evaluation of conceptual soundness, ongoing monitoring, and outcomes analysis. Teams building LLM systems tend to collapse the last two into one dashboard. They answer different questions, and conflating them hides failures.

Ongoing monitoring asks whether the system in production is the one you validated, and whether conditions have shifted. Outcomes analysis asks whether its outputs were right. A system can pass the first and fail the second for months.

Process verification: is production the system you tested

Process verification in the 2011 guidance meant confirming that a model was implemented as designed and fed the data it expected. For an LLM, the equivalent is configuration pinning. Record the exact model identifier, a hash of the system prompt, the retrieval index snapshot and the tool list, then check production against that record on every deploy.

Benchmarking against a challenger

For an LLM, the cheapest challenger is a second model run on the same test set. The most informative is a human baseline: what an experienced employee would have written for the same 50 inputs.

I would always run the human baseline for any customer-facing system. It answers what an executive will actually ask: does the system beat the people it supplements?

Outcomes analysis without a ground truth

Back-testing under SR 11-7 compared forecasts with what actually happened, on data not used in development. Most LLM outputs have no single correct answer to compare against. The working substitute is expert grading on a hold-out set that the build team never used while tuning prompts.

After launch, outcomes analysis becomes sampling. Pull a fixed number of production conversations each week, have a domain expert grade them against a written rubric, and track the rate. Override rates and complaint tags help, but they measure what users noticed, not what was wrong. Turning them into alerts is covered in the guide to monitoring agent drift in production.

Effective challenge: who can stop an LLM launch

The FDIC's summary of the 2011 guidance gives a definition. Effective challenge is "critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes". The last clause is the part most AI governance programmes miss. A reviewer who can only comment is not a challenge.

The validator for a high-tier LLM system needs three things, in my view. It needs no stake in the launch date, enough technical depth to read the test results, and written authority to block or condition the release. If any one is missing, the review is a formality.

Large US banks describe this separation in their annual reports. JPMorgan Chase's 10-K for fiscal 2025 says the firm uses models "including those based upon machine learning or artificial intelligence techniques". It names a dedicated independent function, Model Risk Governance and Review, that "defines and governs the Firm's policies relating to the management of model risk". It also states that model risks "are owned by the users of the models" within the business lines.

That is the pattern to copy at any scale: the business owns the risk, and a separate function sets the bar. The filing does not describe LLM-specific validation, so do not read it as evidence of how the bank tests generative systems.

Risk.net reported in 2026 that most banks avoided formal model validation for Copilot and other generative AI applications by classifying them as productivity tools rather than models. Morgan Stanley went the other way for its advisor assistant. Its vendor's case study says the bank used an evaluation framework to test every AI use case before deployment, per OpenAI's published account. That source is vendor marketing, so treat it as a description of intent rather than verified practice.

That label stops working the moment the tool's output reaches a customer or a decision. How to place a human checkpoint once it does is laid out in the human-in-the-loop architecture guide.

The AI model validation file: eleven sections an auditor can read cold

SR 11-7 set a documentation standard that is worth stating as a test. Documentation should be complete enough that someone unfamiliar with the model can understand how it works, its limitations and its key assumptions. If a new hire cannot read your file and explain why the system was allowed to launch, the file has failed.

Each section of the template below has a pass condition, so the go-live decision becomes a check rather than a meeting.

Tiering decides how thick the file gets

SR 11-7 asked for validation rigour proportional to a model's materiality, and the 2026 guidance makes that tailoring more explicit. OSFI's E-23 goes further on limitations: a model "may be approved despite identified weaknesses or limitations provided that compensating mitigants are in place". That sentence is the reason section 9 exists.

A low-tier internal assistant needs a short file. A system that answers customers or moves money needs every section in full.

SR 26-2 moves away from default annual cycles toward materiality-based timing, according to practitioner readings of the new text. For LLM systems, the events that invalidate your evidence are a model version change, a prompt edit, a new retrieval source, a new tool, or a scope expansion.

List those triggers in section 11 and wire at least the first one to an alert. Adversarial tests need re-running after each change too, as the red-teaming programme guide notes.

Where applying SR 11-7 to LLMs breaks down

The mapping strains in three places.

Outcomes analysis becomes judgement

Back-testing worked for credit and market models because default rates and prices are observable. Expert grading is the substitute, and it is slower, more expensive and less consistent than comparing a forecast with a realised number. The honest description is that outcomes analysis for LLMs is a structured opinion, not a measurement.

The model changes underneath you

SR 11-7 assumed the bank controlled when a model changed. A hosted LLM can change behaviour on the provider's schedule, and retire on it too. A validation file for a vendor model is a dated certificate, and the vendor sets the expiry. That makes section 11 matter more than any single test.

The regulators chose not to do this

The strongest argument against this post is the 2026 guidance itself. The agencies that wrote SR 11-7 looked at generative AI and decided the framework did not fit yet. Banks that classified Copilot as a productivity tool were not obviously wrong either. For a low-exposure writing aid, an eleven-section file is overhead with no matching risk. I would still apply the full method to anything customer-facing, but the case for a lighter use-governance track for internal drafting tools is a fair one.

Frequently asked questions

How to start a validation file this week

Pick the one LLM system in your company that is closest to a customer. Write sections 1 to 3 for it today: scope with prohibited uses, a tier, and the exact configuration running in production. That takes an afternoon and usually surfaces at least one use nobody approved.

Then find your validator. Name one person outside the build team who can read the test results and is allowed to say no. If nobody in the organisation fits that description, you have found the gap that matters most, and no eval tool will close it.

Related reading