FRONTFACE Benchmark analysis The coverage
Why query volume is not test coverage
9 min read
Twenty thousand queries that pair one kind of rule with one query shape demonstrate very little. We measured which combinations of conditions were actually tested across 147 access roles, 17 rule shapes, and 10 SQL structures. In the process, we found three defects in our own rewriter.
A test suite is usually reported by its size. The number is easy to produce and easy to read, and it answers a question nobody asked: how much compute was spent. The question that matters is how many genuinely different situations the system was put in. A bug that only shows up in one of them will never be found by repeating the others.
This is well established in software testing. Research on combinatorial testing finds that bugs are usually triggered by a few conditions occurring together, rather than by any single condition. So what you want to count is combinations of conditions, not how many times you ran the test.
Seventeen shapes of access rule
Access policies in practice are not all of the form column = value. They are ranges, exclusions, null and not null checks, pattern matches, rules spanning two tables, rules that overlap one another, and rules that directly conflict and must resolve to the most restrictive. We constructed 147 distinct access roles across 17 such shapes and ran each against live data.
The ranking itself is not the finding. The finding is the absence of a gap: no rule shape was tested only under conditions where failure was impossible. The two highest are the adversarial cases: MULTI, where several rules apply simultaneously, and CONFLICT_RESTRICTIVE, where rules contradict one another.
Ten structures of query
The second dimension is the SQL itself. A rule correctly added to the outer query is of no value if the data also arrives through a subquery, a common table expression, or the second arm of a union. These are not exotic constructions; a language model writing analytical SQL produces them routinely.
OR tautology, highlighted, is the injection-style case: a query constructed to defeat a filter applied naively. With enforcement applied, no structure returned an unauthorised row.Three defects this found in our own rewriter
Execution testing over structures of this kind identified three genuine defects in our own code. Each produced SQL that parsed cleanly, executed without error, and returned the wrong rows. This is a difficult class of defect to detect because nothing in the system appears to be broken.
| Defect | Behaviour |
|---|---|
| First referencing scope only | The rule reached only the first arm of a union. The second arm executed unfiltered. |
| Single rule across aliases | A table referenced twice within one scope under distinct aliases received one ambiguous rule, leaving one side of a self-join unfiltered. |
| Rule placed in the wrong scope | A recursive lookup for the WHERE clause could place a rule meant for the outer scope inside a nested subquery. |
The self-join case is not among the ten structures charted above. All three have been fixed, and the mutation suite described in the companion piece includes faults that reproduce each of them.
We report these because the distinction matters. Two systems can report the same perfect score. One was never tested on hard cases; the other was tested, failed three times, and was fixed. They are not the same system, even though the number looks identical.
Measuring coverage
Across 76,706 test cases, we measure coverage over four factors: the engine, the kind of rule, the shape of the query, and the database.
| Interaction | Covered | Possible | Coverage |
|---|---|---|---|
| Individual factor values | 39 | 39 | 100% |
| Pairs | 465 | 465 | 100% |
| Triples | 1,896 | 1,897 | 99.95% |
One detail about how we count is worth stating, because it cuts both ways. Combinations that cannot exist are excluded by a feasibility filter: some databases contain no column capable of supporting a BETWEEN rule, so that pairing is impossible rather than untested. Of 187 possible rule and database pairings, 147 can actually be constructed. Counting the impossible ones against ourselves would understate coverage as misleadingly as ignoring genuine gaps would overstate it.
Limitations and further directions
Coverage over these four factors is not coverage over everything that could matter. Query length, how deeply tables are joined, how the data is distributed, and what happens under heavy simultaneous load are not part of this analysis. A failure that depends on one of those would not show up here.
The one uncovered triple is a genuine gap rather than an infeasible combination, and we have not yet established whether it is reachable in practice.
The access rules are generated automatically and the data is a public benchmark. That makes the analysis something anyone can check. It also limits what it proves about a real deployment.
Method and data
All figures were recomputed from saved result artifacts and verified against the live database in October 2026; the coverage analysis was rerun and came out the same. The benchmark is BIRD Mini Dev, PostgreSQL dev split, published on the BIRD benchmark website. Our policy generator, test harness and verdict logs for each query are available on request, so that any figure in this piece can be recomputed independently.