1. The question

Every published benchmark result carries an unstated assumption: that the score is a property of the model. Ask the same questions again and you should get the same number. If that assumption fails, then a single benchmark pass is not a measurement, it is one sample from a distribution, and comparisons built on single passes inherit the noise.

The assumption is testable, and it matters twice over. Once for anyone comparing models on a league table. And once, more seriously, for anyone in a regulated setting who needs to know what their AI system actually reasoned when it produced an answer, because if two identical runs can reason differently, then the reasoning behind any specific answer cannot be recreated afterwards. It either was recorded at the time, or it is gone.

If you run the identical evaluation twice, under every determinism control available, do you get the same answers, and the same reasoning?

2. The protocol

We used two tasks from LegalBench, the standard legal reasoning benchmark: insurance policy interpretation and supply-chain disclosure verification. 100 questions per task, selected with a fixed random seed so both runs see the identical questions in the identical order.

Each question is answered through a three-step chained reasoning plan: the model first extracts the relevant terms, then maps the facts against them, then commits to a final answer. Multi-step chaining is deliberate. It gives any divergence room to compound, and it produces intermediate reasoning we can compare step by step, not just final answers. Per run, that is 200 final answers and 600 intermediate reasoning steps.

The whole protocol was run twice on each platform, minutes apart, with nothing changed in between, and the determinism controls a normal deployment would use engaged:

• Local: Gemma 4 31B at full precision (bf16, official published weights), served by vLLM, the widely used open-source inference engine, on a single dedicated GPU, deterministic decoding, fixed seed. No compression or retention layer in the path: this measures the serving engine's own behaviour, and other engines will have their own baseline variance.
• Cloud: a widely used frontier model through its provider's public API, temperature zero, the provider's seed parameter set, and the backend configuration fingerprint the API reports recorded for every response.

We then diffed the two runs of each platform against each other: final answer by final answer, reasoning step by reasoning step.

3. The results

Between two identical runs	Local (Gemma 4 31B, own GPU)	Frontier model via API
Final answers that changed	2 / 200	11 / 200
Reasoning steps that changed	47 / 600	303 / 600
Questions with any reasoning change	36 / 200	173 / 200
Largest score movement on one task	2 points	7 points
Backend configurations serving one run	1	7
Reasoning steps that changed, with batch-invariant mode enabled	0 / 600	not available to the caller

Two runs per platform, identical questions, identical prompts, deterministic settings, minutes apart. "Backend configurations" counts the distinct system fingerprints the API reported within a single run of 200 questions. The final row is a separate pair of runs on the same model and hardware with the serving engine's batch-invariant mode switched on; see section 5.

The scores themselves

Task	Local, run 1 / run 2	API, run 1 / run 2
Supply-chain disclosure	70 / 68	70 / 70
Insurance interpretation	57 / 57	65 / 58

At first glance this table looks contradictory: each platform holds steady on one task and moves on the other. The answer-level diffs resolve it, and they show why scores alone mislead in both directions.

The API's 70 / 70 on supply-chain is not stability. Two answers changed between the runs. They changed in opposite directions, one gaining a point and one losing it, so the score came out identical by cancellation. An unchanged score does not mean unchanged answers, which is exactly why this study diffs every answer and every reasoning step rather than comparing totals.

The local 70 / 68 is real, and small. Two answers changed locally too, both costing a point. That is the serving engine's residual scheduling noise, present on any platform, and we report it rather than rounding it away.

The contrast between the platforms is therefore one of magnitude, not kind. Locally, the worst case across both tasks was 2 answers in 200 and a 2-point score movement. Through the API, it was 11 answers in 200 and a 7-point movement: the same 100 insurance questions scored 65, and then, minutes later, 58, with nothing changed except which run it was. A single benchmark pass through this API cannot tell you the model's insurance score to better than seven points.

The local model scored 57 both times on that task, inside the API model's own 58–65 run-to-run band, and level with it on the other task. We draw no ranking conclusion from two tasks; the numbers are the claim. The local scores were repeatable to within two points. The API's insurance score is a range seven points wide.

The reasoning moved even more than the answers

Final answers understate the effect, on both platforms. Locally, only 2 answers changed, but 47 reasoning steps did: the model often reasoned differently and landed in the same place. Through the API, half of all reasoning steps changed, across 173 of 200 questions. The chain of reasoning that produced any given answer was, for most questions, different the second time.

And within a single API run, the fingerprint field the API itself returns to identify its serving configuration took seven distinct values across the 200 questions. The field does not say what changed, whether hardware, engine build, or configuration; it says only that the serving environment was not constant. One evaluation, one model name, seven distinct self-reported serving configurations. Which of them answers your question is not something the caller controls, or can even hold constant across a run. The provider documents that its determinism controls apply only while this fingerprint stays the same, which is precisely the condition a caller cannot arrange.

4. Why this happens, and why it is nobody's defect

None of this is a flaw in any provider, and we name no provider because the mechanics are industry-wide. Modern inference engines batch many requests together for throughput, and the composition of a batch changes the order of floating-point operations. Floating-point arithmetic is not associative, so answers that sit on a near-tie between two tokens can land either way depending on what else happened to be in the batch. Provider documentation itself acknowledges that temperature-zero outputs can vary across backend updates and serving configurations, and that determinism controls are best-effort.

A local deployment on a dedicated GPU minimises the effect, because the batch contains only your own work, but the default configuration does not eliminate it: inference engines schedule work asynchronously and batch their own internal requests, so the exact interleaving of floating-point operations still varies from run to run even on a dedicated GPU with a fixed seed. Our 2-in-200 residual is that scheduling noise, and we have reported the same effect in our compression studies, where it appears with or without our layer in the path. It is removable: see section 5, where we switch the engine's batch-invariant kernels on and the residual goes to zero. A public API maximises the effect, because your requests are batched with the world's traffic, served by a heterogeneous fleet, and subject to backend changes you cannot see or pin.

The consequences, however, do not care whose defect it is:

1. For benchmarking: single-pass API scores on tasks of this size carry several points of serving noise. Multiple passes are not methodological caution, they are the only way to measure the score at all. And part of any variance measured through an API is the serving infrastructure, not the model.
2. For accountability: through an API, the reasoning behind a specific answer cannot be recreated by re-running the question, and the caller has no setting that changes that. On your own hardware you can make a re-run reproduce itself exactly, and we show below that it does. What no setting gives you is a re-run of a decision already taken, on weights and an engine build you have since replaced. A prompt log records what was asked. It does not record what was reasoned.

Determinism is forward-only. Switching it on today does nothing for an answer given last year, on a model version and an engine build that no longer exist. The record of what a model considered exists at the moment of inference, inside the serving engine, and nowhere else. Either it is retained then, or it never exists.

That is the operational case for retention, and it is only actionable in one place. Through an API, the working state lives on infrastructure the caller can never touch. On a local deployment, the serving layer is yours, and the state can be retained, losslessly compressed, and kept as the record of the run. This study contains no Revyzor component in either arm, deliberately: it measures the world as it is, and the world as it is cannot repeat itself.

What this means for your organisation

For legal and compliance teams: a prompt log records what you asked. It does not record what the model reasoned. If you need to show, in two years, how an AI-assisted decision was reached, re-running the question will not do it: through an API the re-run reasons differently and you cannot stop it, and on your own hardware a faithful re-run would require the same weights, the same engine build and the same determinism settings you were running two years earlier. The reasoning has to be retained at the time.

5. What we do not claim

• No model ranking. Two tasks and 200 questions do not rank a 31B open-weights model against a frontier model, and the two arms differ in both model and serving path. The stability contrast is the claim; the accuracy parity is an observation, bounded to these tasks.
• Not a same-model comparison. This is local open-weights serving versus cloud API serving, compared under the same protocol. It answers "what does each deployment path give you", not "which model is better".
• Not an indictment of any provider. Batched serving and fleet heterogeneity are how high-throughput inference works everywhere. The API behaviour we measured is consistent with the provider's own documentation.
• The local residual is removable, and we removed it. The 2 answers in 200 that changed locally are an artefact of the default configuration, not a property of the hardware. Enabling batch invariance took it to zero. We report the default-configuration numbers because that is what almost every deployment is running, not because the effect is irreducible.
• Two tasks, one domain. This is a protocol demonstration on legal reasoning tasks. The protocol scales to any task set, and we invite others to run it: it needs nothing but the same questions asked twice.
• Repeatability is obtainable locally, and we obtained it. Our local arm above ran vLLM in its default configuration. vLLM also ships a batch-invariant mode (VLLM_BATCH_INVARIANT=1) that fixes the floating-point reduction order regardless of batch composition. We re-ran the entire local protocol with it enabled, on the same model and the same hardware. All 600 reasoning steps and all 200 answers came out identical across the two runs: 0 changed, against 47 and 2 with the flag off. The flag works, it is one environment variable, and anyone can turn it on. We are saying so plainly because a study that reported only the default-configuration residual would be overstating its case. Four things are unaffected. It is not free: fixing the reduction order gives up the fastest matrix-multiply kernels. Microsoft Research and the University of Washington measured a batch-invariant kernel at 194 TFLOPS against 527 for the standard one, and a single deterministic request in a batch of ten taking throughput from 845 to 415 tokens per second (LLM-42, arXiv:2601.17768, SOSP 2026). Those are their figures, not ours, and we have not reproduced them. It is off by default, so it describes almost no deployment running today. It cannot be applied to a third-party API, because the caller does not control that serving stack: the API arm above is untouched by it. And it works only forwards. Enabling it next year does nothing for a decision taken last year, and it does nothing about the model updates and engine upgrades that land between a decision and the question about it. Retention is not an alternative to determinism. It is what determinism cannot give you: a record of something that has already happened.

Full methodology, per-question diffs, raw run files, and the rerun recipe are available under NDA.