Legal AI has a compliance problem that better models will not solve, because it is not a model problem. It is an architecture problem.
Nearly every legal AI product on the market is built on retrieval: documents are cut into fragments, a similarity search selects a handful of fragments, and the model answers from what it was handed. When the answer is later challenged, by a client, a court, a regulator, or an insurer, the honest reconstruction of "what did the AI actually consider?" is impossible in three separate ways:
Regulation is now converging on exactly the capability this architecture cannot provide. The EU AI Act demands transparency, interpretability of outputs, and automatic record-keeping for high-risk systems. The SEC's posture on AI has shifted from prescriptive rulemaking to materiality-based disclosure and enforcement, which puts the burden on firms to show, not merely assert, how AI-derived work was produced. And the courts are already sanctioning lawyers over AI-assisted work at an accelerating rate, in the most significant professional-responsibility issue involving AI the profession has yet faced.
This paper argues that the compliant architecture for AI legal advice is retention, not retrieval: give the model the complete document, and retain the model's full working state, losslessly and tamper-evidently, as a routine artefact of every answer. Our controlled research demonstrates that this architecture is not a compliance tax, it is also materially more accurate: on the legal retrieval industry's own benchmark, whole-document reading roughly tripled the rate at which the model produced the exact governing clause, and improved it roughly tenfold on merger agreements. The gap holds on both of the benchmark's scoring metrics.
With retention, a firm no longer has to take the answer on trust. The record of which documents the model held and what it produced exists, and can be produced when the answer is challenged. That is what regulators, courts, and insurers are beginning to ask for, and retrieval architectures cannot supply it.
A retrieval-augmented generation (RAG) system splits each document into chunks of a few hundred characters, embeds them as vectors, and at question time selects the top handful by similarity to the question. Only those fragments are shown to the model. The design was a rational response to small context windows and high inference costs; both constraints are dissolving.
The compliance consequence is that the system's evidential basis is decided by an unsupervised similarity search, before the language model is ever involved. If the search misses the governing clause, the model answers, fluently and confidently, from whatever it was handed instead.
In our controlled study (Section 3), a retrieval-based arm was asked whether an NDA's obligations survive termination. It answered with genuine survival language, quoted verbatim, in confident legal register. It was the wrong clause, from the wrong part of the agreement; the retrieval step had fetched a lookalike passage and the model had no way to know. Only span-level scoring against lawyer-annotated ground truth revealed the error.
This is the signature failure mode of retrieval in legal work: the wrong answer is indistinguishable from the right one at the surface. No fluency check, confidence score, or output-level guardrail catches it, because the defect entered upstream, in what the model was allowed to see. A supervising lawyer reviewing the output sees plausible contract language and a citation; the malpractice exposure is invisible until it is litigated.
The industry's current answer is logging: store the prompt, store the retrieved chunks, store the output. This is necessary but not sufficient, for two reasons.
First, a transcript records what went in and what came out, not what the model did in between. For multi-step agentic workflows, where the model reads, extracts, cross-references, and concludes over several passes, the intermediate reasoning is the substance of the professional work product, and it is discarded.
Second, transcripts invite the reconstruction fallacy: the assumption that one can re-run the system later to see what happened. In production serving stacks, generation is not reproducible across model updates, serving-engine versions, batching conditions, and hardware. A re-run months later, under subpoena or regulatory examination, produces a different artefact and shows nothing about the original event.
The only record that fully answers "what did the AI consider, and what did it do?" is the model's own working state at the time, retained without loss.
The industry's own ROI literature, including guides published by the leading legal AI platforms, computes return as hours saved multiplied by billing rates: typical users saving tens of hours a month, worked examples reaching twenty-fold returns on licence cost from time savings alone. The arithmetic is sound as far as it goes, and every term in it assumes the output was right.
Nowhere in that calculus is a risk term: the probability that the governing clause was never in front of the model (Section 3 measures it), or the cost when the resulting error surfaces in litigation (Section 2.3 measures that too). Using the same illustrative economics those guides publish, a single sanctioned matter erases roughly a month of the modelled time savings for an entire team, and the record needed to defend the work was never retained.
The same omission shows up in how the market talks about security. Vendor security reviews cover tenant isolation, access controls, and data-privacy certifications: protections for the documents going in. None of it speaks to the defensibility of the answer coming out. A firm can hold every certification in the questionnaire and still be unable to show what its AI considered in a specific matter.
The return on legal AI is not time saved. It is time saved minus expected liability, and only one of those terms is currently being measured.
The EU AI Act (Regulation (EU) 2024/1689) sets the global template. Its high-risk obligations for stand-alone Annex III systems were due to apply from 2 August 2026; under the Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since 27 July 2026, that date has moved to 2 December 2027, and the Annex I product-embedded regime to 2 August 2028. This is a deferral, not a dismantling: the risk-based architecture and the record-keeping and transparency obligations are intact, and the longer runway is time to build the compliant architecture, not a reason to defer it. None of its provisions requires retention of a model's internal working state: the Act is drafted around inputs, outputs and events. What three of them do is set a standard of interpretability that a discarded-state architecture makes harder to meet.
Annex III(8) expressly designates AI systems intended for use in the administration of justice as high-risk. Legal-advice tools sold to firms and in-house teams sit adjacent to that category today and squarely within the Act's transparency philosophy; vendors betting that legal AI will remain forever outside high-risk classification are betting their product architecture on a regulatory perimeter that has only ever expanded. The 2026 Omnibus deferred the dates; it left the classification untouched.
The practical point is architectural: Articles 12 and 13 describe capabilities that must be designed in, not bolted on. Retention is one way to design them in, and it goes further than the Act requires.
The SEC's trajectory is instructive precisely because it is not a single prescriptive rule. The Gensler-era predictive data analytics proposal was formally withdrawn in 2025. What replaced it is arguably harder to satisfy:
For regulated financial institutions consuming legal AI (and for their counsel), the operative question in an examination is no longer "do you have an AI policy?" but "can you show what the system did in this specific matter?" A materiality regime without a prescriptive checklist rewards the firm that can produce a record and punishes the firm that can only produce a policy.
Regulation is prospective; sanctions are current. Public trackers documented, as of April 2026, over 1,300 court proceedings involving AI-fabricated content, nearly 500 of them involving licensed attorneys, with the monthly case rate accelerating through late 2025 and into 2026. Individual sanctions have escalated from roughly $5,000 in 2023 to over $55,000 in a single matter, and in March 2026 a federal appellate panel ordered two attorneys to pay $15,000 each, plus opposing fees, over briefs containing more than two dozen fabricated citations. Courts have moved from warnings to fee awards, disqualification, and bar referrals, noting explicitly that admonition has not solved the problem.
Fabricated citations are the visible tip. The quieter, larger exposure is the failure mode of Section 1.2: real citations to the wrong governing language, produced by a retrieval step nobody can inspect. Professional-liability insurers are beginning to ask how AI-assisted work is supervised; "the associate read the output" is a thin answer when the process that selected the AI's evidence is unlogged and its reasoning unretained. The defensible position, for the lawyer and for the vendor behind the tool, is an inspectable record of what the system saw and did, produced as a by-product of normal operation.
A compliance argument is stronger when the compliant architecture also wins on the merits. In July 2026 we ran a controlled comparison on LegalBench-RAG, the benchmark built by a retrieval company on four lawyer-annotated legal corpora (privacy policies, NDAs, commercial contracts, and public-company merger agreements; 776 questions across 72 documents). The benchmark's authors only ever scored the retrieval step; we instrumented the end-to-end layer, from question to final answer, that the benchmark's authors never ran.
We compared, with the same model, same deterministic decoding, and the authors' own best published retrieval configuration:
The structured read implements the multi-step chain with text-level handoff only: each step is an independent inference call, with prior step outputs interpolated into the next prompt as strings. No working state is retained or reused between steps; prefix sharing within each call is handled by the serving engine's built-in prefix cache. This arm isolates the contribution of whole-document reading and structured prompting from any infrastructure optimisation.
The study scores answers two ways. Its primary metric is token overlap against the annotating lawyer's clause (SQuAD-style F1). Its strict secondary metric, tabulated below, asks whether the answer contained that clause verbatim. We report both; the architecture gap holds on each.
| Corpus | RAG (8 chunks) | RAG (64 chunks) | Whole document | Structured read |
|---|---|---|---|---|
| Privacy policies | 17% | 20% | 34% | 36% |
| NDAs | 13% | 14% | 71% | 73% |
| Commercial contracts | 28% | 30% | 74% | 78% |
| Merger agreements* | 4% | 6% | 41% | 40% |
| Overall | 16% | 17% | 55% | 57% |
*Merger agreements: full coverage, no exclusions. The 31 questions on the three longest agreements in the study (roughly 126K to 203K tokens) were run at the model's native 262K-token window. On those 31 questions, retrieval at standard settings produced the governing clause zero times; whole-document reading identified the correct clause at the same rate it achieves on shorter documents. The longest, highest-stakes agreements are precisely where retrieval fails absolutely.
Under the primary metric, across all 776 questions, the proportion of answers reaching a token F1 of 0.5 or better was 23.7% for 8-chunk retrieval, 23.6% at 64 chunks, 69.2% for whole-document reading and 65.3% for the structured read. At a stricter 0.8 threshold: 9.1%, 7.5%, 46.8% and 33.1%. The ranking of retrieval against whole-document reading is unchanged and the margin is wider than on the verbatim metric: 2.9× at the 0.5 threshold and 6× at 0.8. The choice of metric does not rescue retrieval. Where the two metrics do disagree is on the ordering of the two whole-document arms, which Finding 3 addresses.
Four findings bear directly on the compliance argument:
A note on a published counterexample. One legal AI vendor has reported that on its internal contract dataset, retrieval performed on par with full-document reading, and slightly ahead of it, hypothesising a “noise reduction” effect from supplying only the pertinent clauses to the model. Three observations. First, that result followed extensive retrieval engineering: embedding models from five vendors were benchmarked and the strongest performer on legal text selected. World-class tuning brought retrieval to parity with the whole-document read, not past it; the full document is the reference point the tuning is chasing. Second, the reported figure is an average over thousands of questions, with no per-length or per-difficulty breakdown published, and the comparison used a single-shot prompt with no structured reasoning chain and no state retention. A slight average edge is fully compatible with the catastrophic tail we document, where retrieval at standard settings found the governing clause in none of 31 questions on the longest agreements while the whole-document read held its corpus-average rate; averages are exactly where silent tail failures hide. Third, our miss decomposition locates the noise effect they describe on the retrieval side: at 64 chunks, coverage rises but precision falls, because the model is drowned in lookalike fragments (finding 1). A single coherent document did not behave like a pile of chunks in our study. We could not test their corpus, and the disagreement is worth resolving on shared, public data. But even at exact accuracy parity, retrieval reads fragments and discards state: there is nothing to retain, reload, or inspect afterwards. The retrieval failures we document (embedding misses, ranking failures, fragmentation) are properties of the architecture, not of the benchmark, and the architecture the compliance argument requires is the one that keeps the record.
Revyzor retains the complete internal working state of the model, losslessly, using a hardware-accelerated mechanism inside the standard GPU serving stack (vLLM; integration validated on NVIDIA NIM, July 2026). The mechanism is disclosed under NDA; it requires no custom silicon and no changes to the model. Three properties make this compliance infrastructure rather than a logging feature:
These are properties of the output record, not the data pipeline. They complement, rather than duplicate, the tenant isolation and certification controls that standard vendor security reviews already cover (Section 1.4): those protect what goes in; retention makes defensible what comes out.
| Obligation | What retrieval architectures offer | What retention offers |
|---|---|---|
| EU AI Act Art. 13: outputs interpretable by deployers | Output text plus a list of fragments a similarity search chose | The complete context and working state behind the output, reloadable on the same weights |
| EU AI Act Arts. 12/19: automatic recording, log retention | Request/response logs; the decision-relevant state is discarded | The decision-relevant state is the record, produced automatically by normal inference |
| Art. 86: meaningful explanation of the AI's role | Post-hoc narrative reconstruction | An explanation grounded in the retained state of the actual event, not a reconstruction of it |
| SEC materiality-based disclosure and examination | Policy documents and assertions | A per-matter record capable of supporting the governance claims made in disclosure |
| Malpractice supervision duties (Model Rules 1.1, 5.1, 5.3 analogues) | Lawyer reviews the output surface | Lawyer, opposing counsel, or the court can establish which documents the system held and what it produced |
| Insurer due diligence on AI-assisted work | Vendor questionnaire answers | An artefact per engagement, tamper-evident, retained on the client's terms |
Three claims we deliberately do not make. Retention does not make a model correct: it establishes what the model held and what it produced, which is the precondition for supervision, dispute resolution, and improvement. It is not a regulatory certification: no product makes a deployer compliant; architecture determines what compliance postures are achievable. And it is not conditional on any single regulation surviving in its current form: the direction across the EU, the US, and the courts is the same, toward show-your-work, and retention is the only architecture that has work to show.
An obvious objection to retention deserves a direct answer: does keeping the model's working state not create a new sensitive artefact? It does, and recent research makes the point sharper than the objection intends, because the sensitive artefact already exists in most serving stacks, ungoverned.
The model's working state during inference (the KV cache, in serving terminology) is not an opaque by-product. Peer-reviewed research has demonstrated that the source text can be reconstructed from it. Work presented at NDSS 2025 showed prompt reconstruction from shared caches in multi-tenant serving at up to 99% accuracy; follow-up work at NDSS 2026 demonstrated reconstruction attacks that operate across all layers of the model, approaching perfect recovery when the adversary has partial prior knowledge of the content. The same researchers observe that in today's serving stacks this state is "almost always processed, transmitted between compute nodes, and persisted in plaintext." In plain terms: the working state of a model that has read a merger agreement is that merger agreement, in a different coordinate system.
The regulatory consequence follows directly. Under the EDPB's Opinion 28/2024, data escapes personal-data status only if it cannot be extracted "by means reasonably likely to be used"; published reconstruction tooling is exactly such a means. A persisted working state derived from a personal-data document is therefore personal data in its own right: erasure obligations, retention limits, subject-access rights, and Article 32 security requirements attach to the cache, not only to the source document.
This cuts two ways, and both favour deliberate retention over the industry default:
There is also a reason the industry default is plaintext, and it is the same economics as Section 4.1: working state is gigabyte-scale, and encrypting it at serving latency has been treated as unaffordable. Lossless compression is what changes that arithmetic. Compression is not only what makes retention economical; it is what makes retention governable.
The immediate use case: M&A and transactional due diligence.
Merger agreements, credit facilities, and complex commercial contracts are where retrieval fails most severely (Section 3), where regulatory and malpractice exposure is highest, where several lawyers question the same document across weeks of sessions, and where a single retained read therefore amortises furthest. In-house M&A and compliance teams also buy on shorter cycles than large firms, and their buying criterion, an answer they can defend, is exactly what retention produces.
For legal AI vendors, retrieval is becoming a liability architecture: it caps accuracy on exactly the documents where the fees are (Section 3), and it cannot generate the records that Articles 12 and 13 style obligations, or a malpractice defence, will require. Retention slots underneath existing products as infrastructure; it does not require abandoning the application layer.
For law firms and in-house teams, the procurement question to put to every AI vendor is one sentence: "When this answer is challenged in two years, what exactly can you show me?" A vendor whose truthful answer is "the output text and a list of chunks" is selling you their liability.
For regulated institutions, the examination-ready posture is a record per matter, produced automatically, retained losslessly, inspectable on demand. That is a statement about serving architecture, not about policy documents.
The legal industry spent two years asking whether AI answers are good enough. Regulators, courts, and insurers are now asking a different question: can you show what the AI did? Retrieval architectures cannot answer it. Retention is the architecture that can, and, as it turns out, it finds the governing clause three to ten times more often while doing so.