Context — what BIRD is hard at

For reference, the same 500 questions answered with a plain schema prompt and no access policy at all, by Claude Sonnet 4-6:

Baseline accuracy by BIRD difficulty tier

No access policy applied. This is the benchmark's own difficulty, independent of governance.

Figure 6. 271 of 500 overall, 54.2%. BIRD is a genuinely hard benchmark — challenging questions sit at 36.3% even with no governance constraint in play. This is the context for every accuracy number above: they are measured on a task where frontier models are wrong roughly half the time regardless of access control.

7 · What this does and does not show

Stated plainly

It shows that stating an access rule in a model's prompt fails at a measurable rate that rises as models get cheaper, and that enforcing the same rule on the generated query eliminated every observed failure across 895 scored queries per arm and 27,147 direct engine queries.

It does not show a guarantee. Zero observed failures in a finite sample bounds the rate; it does not prove the rate is zero. Nor does it measure behaviour on unstructured documents, which this benchmark does not cover.

The benchmark is BIRD Mini-Dev, not a customer's data. The policies are generated, not drawn from a real organisation's access model. Both are deliberate — it makes the result reproducible by anyone — and both are limits.

BIRD Mini-Dev is published at bird-bench.github.io. Our policy generator, harness, raw result files and the full verdict log are available on request, so that any figure on this page can be recomputed independently.