How we measured it, and how you can check us.

01 · Scope

Four studies, each with its own sample.

We keep the studies separate because they answer different questions with different samples. Every claim on the site names which one it comes from and its sample size.

The fix-agent study and the autonomous pipeline drew their apps from the exposure study’s leaking apps. The 20 pipeline apps had never been through a fix agent before.

02 · Sample

Choosing the repos.

• Population: public GitHub repos with a Lovable/Supabase project layout, the shape these builders export (a frontend plus a supabase/ folder with migrations).
• Sample: a random sample of 500 of them. We ran 436 of them through the lab.
• Included: the 318 whose migrations built a working database. The other 118 failed to build (missing files, errors, or SQL our lab could not apply) and are excluded from every rate. They are not counted as safe.
• Not selected for: popularity, stars, activity, or any sign of a leak.

03 · The lab

What “rebuilt from its migrations” means.

For each repo the lab creates a fresh, disposable Postgres database on our own machine and gives it the parts of Supabase that access rules depend on:

• the API roles: anon (the public key), authenticated (any signed-in user) and service_role;
• the auth helper functions policies call, such as auth.uid(), auth.role() and auth.jwt(), reading the simulated user from the request claims;
• Supabase’s default grants, so a table is exposed to the API roles exactly as it would be on a project with those defaults.

It then applies every file in supabase/migrations in filename order, the way Supabase does, and reads the final state. Later migrations override earlier ones, so a rule that was opened and later closed counts as closed.

Then it seeds fake rows: a test row in each table, owned by a first test user wherever the table has an owner column (user_id, owner_id, created_by and similar). A second test user plays the stranger. All values are made up; no real person’s data is ever in the lab. The database is thrown away after the run.

04 · Definitions

What counts as a leak.

We read every table as three identities. The SQL below is simplified from what the lab runs.

-- anonymous visitor
set local role anon;
select count(*), count(email) from public.profiles;
-- signed-in stranger: a seeded user who owns none of these rows
set local role authenticated;
select set_config('request.jwt.claims', '{"sub":"<stranger id>","role":"authenticated"}', true);
select count(*), count(email) from public.profiles;
-- owner
select set_config('request.jwt.claims', '{"sub":"<owner id>","role":"authenticated"}', true);
select count(*), count(email) from public.profiles where id = '<owner id>';

Personal data

A column whose name indicates an email, phone number (including mobile and WhatsApp), postal address or date of birth, or an identity or bank number such as passport, SSN, national ID or IBAN. Technical “addresses” (wallet, IP, server, contract) are excluded. Only tables in the public schema, which the Supabase API exposes, are tested.

Readable

The identity’s query returns at least one seeded row that includes a personal column the identity should not see.

Wrong people

The anonymous visitor, or a signed-in stranger reading rows that aren’t theirs.

Per app

An app counts once, under the widest audience that can read its personal data: “anyone” if the anonymous visitor can, otherwise “signed-in” if a stranger can.

05 · Adjustment

From 33% to about 1 in 4.

Column names can’t tell a person’s phone number from a shop’s. So the raw rate (106 of 318, 33%) includes some businesses’ own public contact details, which are meant to be visible.

We estimate how many from the fix-agent study, where agents read each flagged app’s code and judged intent: 4 of 24 flagged apps (about 1 in 6) were a business’s own public contact details. Removing that share from the 106 leaves about 88 apps, 28% of 318. Because the false-alarm rate comes from only 24 apps, we report a range of 25–28%, and round it to “about 1 in 4”.

06 · Fixes

How fixes were made.

For each leaking app, a Claude model working as a fix agent received the lab’s findings and the repo, and followed the same steps every time:

1. Understand intent

   Read the final state of the migrations and how the frontend uses each flagged table: logged-out pages, owner pages, admin pages, and which columns each reads.

2. Give a verdict per table

   Bug (private data exposed by mistake), public by design but leaky (rows should be public, personal columns shouldn’t), false alarm (such as a business’s own contact details) or ambiguous (only the owner can decide).

3. Write one migration

   A single SQL file that runs after all existing migrations, closes the leak and keeps what the app needs, plus notes for any code change that must ship with it and plain-English questions for the owner.

4. Test in the lab

   Apply the migration to the rebuilt database and read every flagged table again as all three identities.

In the autonomous pipeline the models were tried cheapest first: Haiku, then Sonnet, then Opus. Escalation happens only when the independent proof fails, and the failed attempt’s edits are discarded before the next model starts. A spend cap per check stops the chain early.

07 · Judging

How fixes were judged: the strict proof.

An agent’s own report is never the judge. After the agent finishes, a separate verifier rebuilds the database, applies the fix and runs the identity checks itself. It prints one verdict, passed, which is true only if all of these hold:

The column check means a “keep the rows public, hide the personal columns” fix can pass legitimately. The owner check means a fix that closes a leak by locking everyone out fails. Only a passed fix gets a fix branch; anything else is reported as “needs a person”.

Why this matters: in the model trial, before the verifier printed one explicit verdict, the smallest model’s first attempts failed 2 of 3 in ways a looser check missed: one hid columns from logged-out visitors only, leaving signed-in strangers reading everything; another closed a leak and locked users out of their own history. The strict verdict exists because of those two failures.

The 5 Nov study was judged differently, and more slowly: we re-ran every agent fix ourselves in the lab rather than trusting the agents’ reports. “Worked first time” means the fix applied cleanly and closed the leak in that re-run. That study also recorded whether each fix was safe to apply with no human review (4 of 20 were).

08 · Cost basis

What the cost and time figures include.

Model cost

The spend reported for each agent attempt by the agent runtime, at Anthropic’s API prices in November 2026, summed per app across every attempt, including failed ones.

Time

The agents’ working time, summed across attempts. Rebuilding the database and each re-verification took seconds on top (medians of 2 and 3 seconds in the 20-app run), not included.

Excluded

The lab machine (our own computer running Docker), electricity, and any human time. Human review time in the 5 Nov study was the agent’s own estimate (median 15 minutes), not measured.

Medians

Reported with the full distribution, because one escalated app ($15.65, 67 minutes) pulls the means up to $1.75 and 9.2 minutes. All 20 values are in the report and the CSV.

09 · Keys

The key scan.

Alongside the lab, every check scans the repo’s current code and its full git history for exposed AI, payment and database keys, including Supabase service role keys (PL-010) and keys deleted from the code but still in old commits (PL-011). Values are masked in reports. We record where a key was found and where to rotate it; we never use, test or store a key.

The key scan is part of the product and ran during the studies, but no key numbers are claimed on this site.

10 · Conduct

What we didn’t do.

• Connect to any studied app’s live site, API or database.
• View, copy or store any real person’s data. The lab only ever holds fake rows.
• Use, test or store any key found in a repo.
• Push fixes to anyone’s repo. Fix branches existed only on our machine (a dry run).
• Name any app, repo, owner or table, here, in the report or in the CSV.

11 · Reproduce

Reproducing it.

What you can check today:

• The same test on a known app. Our deliberately leaky demo app (manra399/patchlabs-demo-app, public, fake data) goes through exactly this pipeline. Its sample report shows the findings, the fix and the proof.
• The same test on your app. A free check runs the identical lab, verdict and proof on your public repo.
• The definitions in SQL. Every finding page includes queries that reproduce the identity checks in your own Supabase SQL editor.
• The numbers. The aggregate CSV holds every figure we publish and the 20 anonymised runs.

What isn’t public yet: the lab scripts themselves, and the list of sampled repos (which we won’t publish, to avoid pointing at leaking apps). Researchers who want to replicate the study can email manraj@patchlabs.dev for the method in more detail.

12 · Changes

Changes to this page.

1. First publication

   Methodology for the exposure study, the fix-agent study, the model trial and the autonomous pipeline.

   10 Nov 2026

Found an error? Email manraj@patchlabs.dev with the subject “Research”. Corrections are dated here.