Whitepaper
Inference Under the EU AI Act: Why the Rulebook Points Toward Local LLMs
Europe’s risk-based AI regime does not mandate where a model runs, but its demand for control, not merely residency, quietly reshapes the answer.
Author: Revyzor · Published: June 2026 · Reflects the law as of June 2026
The EU AI Act is technology- and deployment-neutral: it never says an LLM must run on-premise. Yet for high-risk and personal-data use cases it imposes obligations (control of the data, model and decisions; lifecycle logging; consistent, documented accuracy; human oversight; cybersecurity) that are far easier and far less risky to satisfy on infrastructure you control. Layered onto GDPR and the extraterritorial reach of laws like the US CLOUD Act, the result is a structural tilt: EU-region cloud delivers residency but not control, while on-premise, edge and sovereign inference deliver both.
A practical consequence follows: running local LLMs for high-risk AI means running them on constrained GPU hardware. Without KV cache compression, enterprises face a brutal choice: limit context length, limit concurrent users, or buy more GPUs. The dominant approach, lossy compression, changes the model’s output, which means the conformity assessment no longer describes the system actually running. Lossless KV cache compression delivers both: GPU headroom that pays for itself, without altering the system your conformity assessment describes.
01
Where the rules stand in mid-2026
The AI Act (Regulation (EU) 2024/1689) entered into force on 1 August 2024 and applies in staggered phases. Prohibited practices and AI-literacy duties have applied since February 2025; obligations on general-purpose AI (GPAI) models since August 2025. The most consequential layer, the regime for high-risk AI systems, was due to apply from 2 August 2026.
That date moved. Facing missing standards and heavier-than-expected compliance costs, the Commission tabled the Digital Omnibus on AI in November 2025; EU co-legislators reached political agreement on 7 May 2026, and the regulation entered into force on 27 July 2026 as Regulation (EU) 2026/1744. It resets the key dates:
| Date | What applies | Status |
|---|---|---|
| Feb 2025 | Prohibited practices & AI-literacy duties | In force |
| Aug 2025 | GPAI model obligations (docs, copyright policy, training summary) | In force |
| 2 Aug 2026 | Article 50 transparency & GPAI enforcement powers | On schedule |
| 2 Dec 2026 | Watermarking of AI-generated content; new Art. 5 bans | Re-dated |
| 2 Dec 2027 | High-risk obligations: stand-alone systems (Annex III: hiring, credit, education, justice…) | Deferred |
| 2 Aug 2028 | High-risk obligations: AI embedded in regulated products (Annex I: medical devices, machinery…) | Deferred |
One caution matters. This is a deferral, not a dismantling: the risk-based architecture, governance and core obligations are intact, and the Article 50 transparency and GPAI enforcement dates in August 2026 were not moved. The runway is longer; the destination is unchanged.
02
How the Act bites at inference time
The Act regulates uses, systems and models, not data centres. But once a high-risk system is running, the duties are squarely operational, and they govern the act of inference itself:
Data governance: the data fed to and produced by the system must be managed, lawful and controlled.
Record-keeping & logging: automatic, event-level logs across the system's lifecycle, retained and producible to regulators.
Human oversight: operators must be able to interpret, intervene in and override outputs.
Accuracy, robustness & cybersecurity: the system must perform consistently with its documented accuracy and resist manipulation.
Post-market monitoring: continuous evidence that the deployed system still behaves as assessed.
The common thread is not a location requirement; it is a control-and-evidence requirement. You must be able to govern the running system and hand an auditor a complete, truthful account of it.
03
Residency is not control: the cloud gap
Nowhere does the Act say “keep the data in the EU.” What it requires is that you control the data, the model and the decisions, and can produce an audit trail on demand. That distinction is where the inference architecture question turns.
The control test
Choosing AWS Frankfurt or Azure Germany gives you data residency: the bytes sit in Europe. It does not give you control: under the US CLOUD Act, a US-parented provider can be compelled to produce data about an EU data subject. The moment a foreign authority can reach your inference endpoint, control (the thing the Act and GDPR actually require) is gone.
This is why “sovereign by contract” is weaker than “sovereign by architecture.” Data-processing agreements and EU regions reduce exposure, but they cannot fully sever extraterritorial legal reach. Gartner expects 75% of European enterprises to “geopatriate” workloads by 2030; the AI Act is one of the forces behind that move.
04
Why this tilts toward local LLMs
Data never leaves the perimeter. On-prem, edge or sovereign inference collapses the transfer, CLOUD-Act and data-governance risk for personal and high-risk data.
You freeze and document the exact model. Hosted APIs silently update the model beneath you — breaking the documented baseline your conformity assessment depends on. Local deployment keeps the version stable, documented and auditable.
Logs and telemetry are yours. Article 12 logging and Article 72 monitoring are clean when no third party sits in the loop.
Open weights, lighter load. The open-weight models that dominate local deployment (Llama, Mistral, Qwen) carry lighter GPAI obligations — a partial exemption for free and open-source models, absent systemic risk.
Honest counterweight
The Act does not ban the cloud. An EU-parented sovereign cloud with strong safeguards can be compliant, and local deployment is not free: it shifts the logging, oversight, robustness and cybersecurity duties onto you. The point is not prohibition; it is that local inference lowers structural compliance risk and cost for exactly the workloads the Act scrutinizes most.
05
The lossless corollary: how you compress matters
There is a subtle compliance consequence in how local inference is optimized. Conformity, technical documentation and accuracy are assessed against a defined system configuration (Art. 11, 15); post-market monitoring (Art. 72) presumes the deployed system keeps behaving as assessed; and an auditor may reasonably expect you to show exactly what the system was working with when it produced a given output.
The dominant trick for fitting big models onto constrained hardware, lossyKV-cache quantization or token eviction, quietly changes the model’s outputs relative to the validated baseline. That muddies your accuracy claims, breaks the link between the record and what actually ran, and can amount to a change that must be re-documented and re-assessed.
The word quietly is doing real work there, and it has been measured. Across 25,282 questions on six public benchmark suites, moving an 8B model to an FP8 KV cache moved the headline accuracy score by 0.79 points, while 9.03% of the individual answers came back different. That is the worst possible shape for a conformity assessment: the aggregate metric in your technical documentation barely moves, so post-market monitoring shows nothing, while roughly one answer in eleven is no longer the answer the assessed system would have given. A headline score is an average over a population. A refused application or a flagged transaction is a single case, and the single case is what you are asked to account for.
Where Revyzor fits: headroom without changing a single output
Revyzor is exact, lossless KV-cache compression, byte-for-byte checked at capture: ~1.4× more memory headroom on the NVIDIA GPUs you already run, with the model state preserved exactly as in the configuration you certified. It drops into vLLM, SGLang and NVIDIA NIM. The effect under the Act: you gain the efficiency to run capable models locally (longer context, more concurrent users, edge deployment) without re-opening conformity or polluting the audit trail. The system a regulator inspects is exactly the system you validated.
06
What deployers should do now
Classify before you build. Map each AI use case to its risk tier; the high-risk ones (Annex III) drive the heaviest duties and the strongest case for control.
Default high-risk + personal-data inference to architectural control. Prefer on-prem, edge or genuinely sovereign deployment over residency-only cloud.
Freeze and document model versions, and keep inference logs inside your perimeter to satisfy Articles 12 and 72.
Prefer lossless optimizations. Keep the running system unchanged from the validated configuration; treat lossy quantization as a change-management event.
Use the 2027–2028 runway to build, not defer. The architecture takes time; the direction of travel is fixed.
Sources
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex
- European Parliament, “AI Act: deal on simplification measures,” 7 May 2026
- Gibson Dunn, “EU AI Act Omnibus Agreement,” May 2026
- Taylor Wessing, “The EU Digital Omnibus on AI,” 8 May 2026
- Pinsent Masons, “Rules on ‘high-risk’ AI to be delayed,” 7 May 2026
- AI in Europe, “Act Tracker,” 1 June 2026
- Particula Tech, “EU AI Act August 2026: Why Frankfurt Region Isn’t Enough,” April 2026
- A&O Shearman, “Digital Omnibus on AI: the trilogue deal is done,” June 2026
Disclaimer. This whitepaper is for general information and reflects the state of play as of June 2026; it is not legal advice. The Digital Omnibus amendments described rest on a provisional political agreement that, at the time of writing, had not yet been published in the Official Journal and may change before formal adoption. Article and obligation references summarize the regime and omit detail; organizations should seek qualified counsel for their specific circumstances.
Build the architecture now.
The runway is longer. The destination is unchanged. Revyzor gives you a cryptographically signed inference record that supports Article 12 logging today.
Request a Pilot