FRONTFACE Benchmark analysis The method
Can a vendor benchmark its own security claim?
8 min read
We built a benchmark that measures our own product, which creates an obvious conflict of interest. This article explains four checks we used to address it: an answer key constructed by the database, a negative control, nine seeded faults, and a comparison with PostgreSQL’s own row-level security.
The strongest objection to a benchmark built by a vendor is that the vendor also builds the answer key. If we decide what the correct result is, our product can hardly be shown to fail. Any number we publish is worth exactly as much as the answer to that objection, so it seems right to address it before reporting anything.
There is a second and less obvious problem. A benchmark in which nothing was ever at risk will report a perfect score for a system that does nothing at all. A zero is only informative if something could have been greater than zero.
Constructing an answer key we do not control
For each access policy, the database creates a copy of the protected table containing only the permitted rows. The statement is unremarkable:
CREATE TABLE rbac.customers AS SELECT * FROM public.customers WHERE "currency" = 'CZK';
The benchmark’s own reference SQL for the question is then run against that copy. Whatever it returns is the correct answer for that person. Our engine takes no part in producing it, and is graded against a result it did not compute.
CREATE TABLE AS SELECT. The benchmark’s own reference SQL is then run against that copy, never against the real table. Both experimental conditions are scored against the result. Our engine appears once, in the second condition, downstream of everything that determines the correct answer.We reran this construction from scratch in October 2026, rebuilding all eleven tables of authorized rows in a separate schema and comparing each against the tables the published results were scored on. We compared every row in both directions, not just the totals. Not one row differed.
Establishing that the test can fail
Every query was run a second time with enforcement switched off, for the sole purpose of recording whether a leak was available to be detected.
This is the figure we would want a sceptical reader to check first. A result of zero across a corpus in which only a handful of queries could ever have leaked would carry almost no information.
Three checks on the measurement itself
The answer key can be correct and the harness still blind. We tested that three ways, each rerun live in October 2026.
| Check | Result | What it rules out |
|---|---|---|
| Mutation testing | 9,507 / 9,507 | That the harness would fail to notice a broken rewriter. We deliberately broke the engine in nine ways: not applying the rule at all, applying it only to the outermost scope, applying it only to the first scope, skipping it inside CTEs, skipping it inside UNION and other set operation branches, reversing it, replacing it with a condition that is always true, combining it with OR and a condition that is always true, and comparing against the wrong value. Every break that changed an answer was caught. |
| Native RLS cross-check | 394 / 394 | That our answer key is itself wrong. We set up a PostgreSQL account that cannot bypass the database’s own access control, and turned that control on. This is enforcement written by the PostgreSQL project, not by us. It agreed with our restricted copy on every query the two could be compared on. Six could not be compared. |
| Coverage of condition combinations | 100% / 99.95% | That a big number of tests is hiding shallow testing. We checked every valid pairing of conditions, and all but one three-way combination, across 76,706 test cases covering engine, rule type, query shape and database. |
A note on the mutation denominator, because it is the kind of thing that can be made to look better than it is. Many of the breaks we introduce make no difference on a given query. Reversing a rule changes nothing if the query already filters on that column. Nothing can catch a break that did not change the answer, so counting those as misses would be unfair to the test, and counting them as catches would flatter it. We leave them out. So 9,507 is the number of breaks that actually changed an answer. A further 3,777 had no effect and were set aside.
How we state the result
Observing no failures in a finite sample bounds the failure rate. It does not eliminate it, and we do not describe it as though it did. The form we use is:
Across 27,147 executed queries on live PostgreSQL spanning 11 databases, 147 access roles and 17 rule shapes, we observed no case where an unauthorised row was returned. This bounds the leak probability per query below 0.011 percent at 95 percent confidence.
On the more conservative basis of individual roles, which treats each of the 147 roles as the unit rather than each of the 27,147 queries, the bound is 2.02 percent. Both figures are correct and they answer different questions, so we state the unit whenever we quote one.
Limitations and further directions
The access policies are generated mechanically rather than drawn from a real organisation’s access model, and the underlying data is a public benchmark rather than a customer’s. Both choices are deliberate, because they make the work something anyone can check, and both limit what it demonstrates about a production deployment.
The measurement also covers structured queries only. It says nothing about retrieval over unstructured documents, which this benchmark does not exercise.
We are not aware of a vendor in this category that publishes a failure rate for permission enforcement, or a bound on one. We would rather that changed, and the method above is offered partly in that spirit: the number matters less than whether anyone can check it.
Method and data
All figures were recomputed from saved result artifacts and verified against the live database in October 2026. We rebuilt the answer key tables from scratch and every one came out identical, with not a single row differing. Mutation testing, the native row-level security cross-check and the coverage analysis were each rerun live. The benchmark is BIRD Mini Dev, PostgreSQL dev split, published on the BIRD benchmark website. Our policy generator, test harness and verdict logs for each query are available on request.