Headline result — six AI vendors, two architectures

Each model answered all 500 questions twice. Path A states the access rule in the model's prompt, which is what products do today. Path B tells the model nothing about access and enforces the rule on the query it writes. 178–180 questions per model actually touch the protected table; those are the ones scored.

The metric is Exact Authorized Equivalence: did the query return exactly the rows this principal was entitled to — no more, and no fewer.

Unauthorised results when the access rule is stated in the model's prompt

Raw counts of queries that returned data the asker was not entitled to.

Figure 1. Two of the five never failed. Including them is what makes the other three credible — and the pattern they reveal is more useful than the average: prompt-based enforcement holds on the largest frontier models and degrades sharply on cheaper and open-weight ones.

Rule stated in the prompt

unauthorised results in 895 scored queries — 3.80%

Rule enforced on the query

in 895 scored queries — 0.00%

Probability this is chance

8.5 × 10⁻¹¹

Fisher exact test, two-tailed

Accuracy change, A → B

+7.4

points on average. Every model improved.

Accuracy by model: rule in the prompt → rule enforced on the query

Exact Authorized Equivalence. Higher is better. Safety was identical on both paths for every model except where Figure 1 shows otherwise.

Figure 2. Accuracy rose on every model when the rule was removed from the prompt — by as little as 1.8 points and as much as 12.0. A model handed a governance constraint spends reasoning on the constraint instead of on the question.

The full table

Model	Vendor	Accuracy, rule in prompt	Accuracy, enforced	Unsafe, prompt	Unsafe, enforced
claude-opus-5.5	Anthropic	81.0%	82.8%	0 of 179	0 of 180
gemini-3.8-flash	Google	51.7%	63.7%	1 of 178	0 of 179
gpt-6-sol	OpenAI	59.4%	62.6%	0 of 180	0 of 179
deepseek-v4.1-flash	DeepSeek	43.3%	54.2%	3 of 180	0 of 179
llama-4-maverick	Meta	42.1%	51.1%	30 of 178	0 of 178
grok-4.7	xAI	61.1%	—	0 of 180	—
Pooled, the five models with both paths complete				34 of 895	0 of 895

Two disclosures. Grok's enforced path is excluded from every aggregate: the run stopped at a spend cap after 196 of 500 generations, so only 75 of its 180 policy-affected questions have SQL. Its prompt path is complete and valid. Separately, DeepSeek declined to answer 127 of 500 questions outright — the highest refusal rate in the sweep — which depresses its accuracy on both paths equally.

7 · What this does and does not show

Stated plainly

It shows that stating an access rule in a model's prompt fails at a measurable rate that rises as models get cheaper, and that enforcing the same rule on the generated query eliminated every observed failure across 895 scored queries per arm and 27,147 direct engine queries.

It does not show a guarantee. Zero observed failures in a finite sample bounds the rate; it does not prove the rate is zero. Nor does it measure behaviour on unstructured documents, which this benchmark does not cover.

The benchmark is BIRD Mini-Dev, not a customer's data. The policies are generated, not drawn from a real organisation's access model. Both are deliberate — it makes the result reproducible by anyone — and both are limits.

BIRD Mini-Dev is published at bird-bench.github.io. Our policy generator, harness, raw result files and the full verdict log are available on request, so that any figure on this page can be recomputed independently.