FRONTFACE | Benchmark analysis

Does stating an access rule in the prompt actually enforce it?

Five model vendors, 500 questions each, and two conditions. Stating the rule in the prompt produced 34 unauthorised results in 895 scored queries. Enforcing it on the generated query produced none.

Ashwini Settu · 13 Oct 2026 7 min read

Read the analysis

Can a vendor benchmark its own security claim?

An independent answer key, a negative control, nine seeded faults, and a comparison with PostgreSQL’s own access controls. Four checks that make the measurement open to scrutiny.

Ashwini Settu · 13 Oct 2026 8 min read

Read the analysis

Why query volume is not test coverage

147 access roles, 17 rule shapes, and 10 SQL structures. Measuring the combinations behind the query count revealed three defects in our own rewriter.

Ashwini Settu · 13 Oct 2026 9 min read

Read the analysis

The full report

BIRD-Gov, in seven parts.

The complete benchmark write up, from setup to limits, with every chart.

01The setup500 natural-language questions across 11 real databases, loaded into one PostgreSQL instance. Each database carries one mechanically generated access policy — no hand-picking, which is what stops a benchmark being a demo.
02Headline result — six AI vendors, two architecturesEach model answered all 500 questions twice. Path A states the access rule in the model's prompt, which is what products do today. Path B tells the model nothing about access and enforces the rule on the query it writes. 178–180 questions per model actually touch the protected table; those are the ones scored.
03Enforcement at scaleBeyond the model sweep, the rewriting engine itself was exercised directly against the same 11 databases, with randomised query shapes and 147 generated roles.
04Depth, not just volumeTwenty thousand queries that only ever pair one rule type with one query shape have covered very little. These two charts answer “did you only test the easy cases?”
05Three checks that the measurement itself is soundThese are the difference between “we ran a lot of tests” and “we can show the tests would have caught it.”
06Context — what BIRD is hard atFor reference, the same 500 questions answered with a plain schema prompt and no access policy at all, by Claude Sonnet 4-6:
07What this does and does not showIt shows that stating an access rule in a model's prompt fails at a measurable rate that rises as models get cheaper, and that enforcing the same rule on the generated query eliminated every observed failure across 895 scored queries per arm and 27,147 direct engine queries.

Read the benchmark analysis

Does stating an access rule in the prompt actually enforce it?

Can a vendor benchmark its own security claim?

Why query volume is not test coverage

BIRD-Gov, in seven parts.

Read the benchmark analysis

Does stating an access rule in the prompt actually enforce it?

Can a vendor benchmark its own security claim?

Why query volume is not test coverage

BIRD-Gov, in seven parts.