500 natural-language questions across 11 real databases, loaded into one PostgreSQL instance. Each database carries one mechanically generated access policy — no hand-picking, which is what stops a benchmark being a demo.
| Database | Protected table | Access rule | Authorized | Of total | Visible |
|---|---|---|---|---|---|
| european_football_2 | match | season = '2008/2009' | 3,326 | 25,979 | 12.8% |
| superhero | hero_attribute | attribute_id = 1 | 623 | 3,738 | 16.7% |
| formula_1 | pitstops | stop = 1 | 2,552 | 5,815 | 43.9% |
| codebase_community | posts | posttypeid = 2 | 47,755 | 91,966 | 51.9% |
| financial | trans | type = 'VYDAJ' | 634,571 | 1,056,320 | 60.1% |
| california_schools | schools | statustype = 'Active' | 11,708 | 17,686 | 66.2% |
| student_club | zip_code | type = 'Standard' | 29,973 | 41,877 | 71.6% |
| thrombosis_prediction | patient | sex = 'F' | 1,023 | 1,238 | 82.6% |
| toxicology | bond | bond_type = '-' | 7,743 | 9,156 | 84.6% |
| debit_card_specializing | customers | currency = 'CZK' | 30,459 | 32,461 | 93.8% |
| card_games | legalities | status = 'Legal' | 424,958 | 427,907 | 99.3% |
| Across all 11 | 1,194,691 | 1,714,143 | 69.7% | ||
Policies span the full range deliberately: from european_football_2, where a principal sees one season in eight, to card_games, where almost everything is permitted. A benchmark that only tested narrow policies would overstate how often enforcement matters.
For each policy the database builds a copy of the protected table containing only the permitted rows — a plain CREATE TABLE AS SELECT. BIRD's own human-written reference SQL is then run against that copy. Whatever it returns is the correct answer for that principal.
Our engine plays no part in producing it. That is what makes the scoring independent: we are graded against a result we did not compute.