BIRD is the standard public benchmark for turning natural language into SQL. It measures accuracy and nothing else — it has no notion of who is asking. We added one, and ran the result across six AI vendors and 1.7 million rows of real data.
All figures recomputed from saved result artifacts, October 2026. Benchmark: BIRD Mini-Dev, PostgreSQL dev split, from bird-bench.github.io. Method and raw results available on request.
500 natural-language questions across 11 real databases, loaded into one PostgreSQL instance. Each database carries one mechanically generated access policy — no hand-picking, which is what stops a benchmark being a demo.
| Database | Protected table | Access rule | Authorized | Of total | Visible |
|---|---|---|---|---|---|
| european_football_2 | match | season = '2008/2009' | 3,326 | 25,979 | 12.8% |
| superhero | hero_attribute | attribute_id = 1 | 623 | 3,738 | 16.7% |
| formula_1 | pitstops | stop = 1 | 2,552 | 5,815 | 43.9% |
| codebase_community | posts | posttypeid = 2 | 47,755 | 91,966 | 51.9% |
| financial | trans | type = 'VYDAJ' | 634,571 | 1,056,320 | 60.1% |
| california_schools | schools | statustype = 'Active' | 11,708 | 17,686 | 66.2% |
| student_club | zip_code | type = 'Standard' | 29,973 | 41,877 | 71.6% |
| thrombosis_prediction | patient | sex = 'F' | 1,023 | 1,238 | 82.6% |
| toxicology | bond | bond_type = '-' | 7,743 | 9,156 | 84.6% |
| debit_card_specializing | customers | currency = 'CZK' | 30,459 | 32,461 | 93.8% |
| card_games | legalities | status = 'Legal' | 424,958 | 427,907 | 99.3% |
| Across all 11 | 1,194,691 | 1,714,143 | 69.7% | ||
Policies span the full range deliberately: from european_football_2, where a principal sees one season in eight, to card_games, where almost everything is permitted. A benchmark that only tested narrow policies would overstate how often enforcement matters.
For each policy the database builds a copy of the protected table containing only the permitted rows — a plain CREATE TABLE AS SELECT. BIRD's own human-written reference SQL is then run against that copy. Whatever it returns is the correct answer for that principal.
Our engine plays no part in producing it. That is what makes the scoring independent: we are graded against a result we did not compute.
Each model answered all 500 questions twice. Path A states the access rule in the model's prompt, which is what products do today. Path B tells the model nothing about access and enforces the rule on the query it writes. 178–180 questions per model actually touch the protected table; those are the ones scored.
The metric is Exact Authorized Equivalence: did the query return exactly the rows this principal was entitled to — no more, and no fewer.
| Model | Vendor | Accuracy, rule in prompt | Accuracy, enforced | Unsafe, prompt | Unsafe, enforced |
|---|---|---|---|---|---|
| claude-opus-5.5 | Anthropic | 81.0% | 82.8% | 0 of 179 | 0 of 180 |
| gemini-3.8-flash | 51.7% | 63.7% | 1 of 178 | 0 of 179 | |
| gpt-6-sol | OpenAI | 59.4% | 62.6% | 0 of 180 | 0 of 179 |
| deepseek-v4.1-flash | DeepSeek | 43.3% | 54.2% | 3 of 180 | 0 of 179 |
| llama-4-maverick | Meta | 42.1% | 51.1% | 30 of 178 | 0 of 178 |
| grok-4.7 | xAI | 61.1% | — | 0 of 180 | — |
| Pooled, the five models with both paths complete | 34 of 895 | 0 of 895 | |||
Two disclosures. Grok's enforced path is excluded from every aggregate: the run stopped at a spend cap after 196 of 500 generations, so only 75 of its 180 policy-affected questions have SQL. Its prompt path is complete and valid. Separately, DeepSeek declined to answer 127 of 500 questions outright — the highest refusal rate in the sweep — which depresses its accuracy on both paths equally.
Beyond the model sweep, the rewriting engine itself was exercised directly against the same 11 databases, with randomised query shapes and 147 generated roles.
Observing no failures in a finite sample bounds the failure rate; it does not eliminate it. On the more conservative per-role basis — 147 roles rather than 27,147 queries — the bound is 2.02 percent. Both are true and they answer different questions.
Twenty thousand queries that only ever pair one rule type with one query shape have covered very little. These two charts answer “did you only test the easy cases?”
MULTI at 88% and CONFLICT_RESTRICTIVE at 87% are the adversarial ones — overlapping and conflicting rules, which is where real enterprise policy lives.OR-tautology is the injection-style case — a query written to defeat a naive filter — and the one where a leak was most often available. UNION, CTE, INTERSECT and EXCEPT are the shapes that produced three real bugs in our own rewriter before they could reach a customer.These are the difference between “we ran a lot of tests” and “we can show the tests would have caught it.”
| Check | What it rules out | Method |
|---|---|---|
| Mutation testing | That the harness would fail to notice a broken rewriter | Nine deliberate faults seeded into the engine: no injection at all, outermost scope only, first scope only, skip CTE bodies, skip set-operation branches, negate the predicate, replace it with 1=1, OR it with 1=1, wrong value. Denominator counts faults that actually changed a result; semantically equivalent mutants are excluded rather than scored as misses. |
| Native RLS cross-check | That our independently built answer key is itself wrong | A dedicated PostgreSQL role with no BYPASSRLS, FORCE ROW LEVEL SECURITY enabled, compared against the authorized-only schema. |
| Combinatorial coverage | That raw volume is hiding shallow coverage | t-way coverage across four factors: engine, rule type, query shape, database. Combinations that cannot exist are excluded by a feasibility filter — some databases have no column able to support a BETWEEN rule, and counting those against coverage would understate it as dishonestly as ignoring real gaps overstates it. |
For reference, the same 500 questions answered with a plain schema prompt and no access policy at all, by Claude Sonnet 4-6:
It shows that stating an access rule in a model's prompt fails at a measurable rate that rises as models get cheaper, and that enforcing the same rule on the generated query eliminated every observed failure across 895 scored queries per arm and 27,147 direct engine queries.
It does not show a guarantee. Zero observed failures in a finite sample bounds the rate; it does not prove the rate is zero. Nor does it measure behaviour on unstructured documents, which this benchmark does not cover.
The benchmark is BIRD Mini-Dev, not a customer's data. The policies are generated, not drawn from a real organisation's access model. Both are deliberate — it makes the result reproducible by anyone — and both are limits.
BIRD Mini-Dev is published at bird-bench.github.io. Our policy generator, harness, raw result files and the full verdict log are available on request, so that any figure on this page can be recomputed independently.