Three checks that the measurement itself is sound

These are the difference between “we ran a lot of tests” and “we can show the tests would have caught it.”

Seeded faults detected — mutation testing

6,407 / 6,407

9 fault classes, 0 missed

Agreement with PostgreSQL's own row-level security

149 / 149

0 disagreements

Combinatorial coverage, 2-way

465 / 465

100% — every valid pair of conditions exercised

Combinatorial coverage, 3-way

1,896 / 1,897

99.95% over 76,706 factor assignments

Check	What it rules out	Method
Mutation testing	That the harness would fail to notice a broken rewriter	Nine deliberate faults seeded into the engine: no injection at all, outermost scope only, first scope only, skip CTE bodies, skip set-operation branches, negate the predicate, replace it with 1=1, OR it with 1=1, wrong value. Denominator counts faults that actually changed a result; semantically equivalent mutants are excluded rather than scored as misses.
Native RLS cross-check	That our independently built answer key is itself wrong	A dedicated PostgreSQL role with no `BYPASSRLS`, `FORCE ROW LEVEL SECURITY` enabled, compared against the authorized-only schema.
Combinatorial coverage	That raw volume is hiding shallow coverage	t-way coverage across four factors: engine, rule type, query shape, database. Combinations that cannot exist are excluded by a feasibility filter — some databases have no column able to support a BETWEEN rule, and counting those against coverage would understate it as dishonestly as ignoring real gaps overstates it.

7 · What this does and does not show

Stated plainly

It shows that stating an access rule in a model's prompt fails at a measurable rate that rises as models get cheaper, and that enforcing the same rule on the generated query eliminated every observed failure across 895 scored queries per arm and 27,147 direct engine queries.

It does not show a guarantee. Zero observed failures in a finite sample bounds the rate; it does not prove the rate is zero. Nor does it measure behaviour on unstructured documents, which this benchmark does not cover.

The benchmark is BIRD Mini-Dev, not a customer's data. The policies are generated, not drawn from a real organisation's access model. Both are deliberate — it makes the result reproducible by anyone — and both are limits.

BIRD Mini-Dev is published at bird-bench.github.io. Our policy generator, harness, raw result files and the full verdict log are available on request, so that any figure on this page can be recomputed independently.