What a stranger can verify about an AI business

Start with a traffic cone.

Polyfork, one of the 100 finalists in the Build with Gemini XPRIZE, sells 3D models written as small programs. Its submission says that when you make its traffic cone taller, the model isn’t stretched. It is rebuilt: a reflective band appears, and the triangle count rises from 280 to 472.

We downloaded the free module, no account required, and ran it. Short: 280 triangles. Default: 376. Tall: 472. Scale the short cone up instead and it stays at 280. The claim is exactly true, and anyone with a laptop can confirm it in minutes.

That is what most people imagine verification to be: run the thing, get a yes or a no.

It was the last time it was that simple.

We screened all 100 projects on the official Top 100 list for how cheaply an outsider could reach an evidence-backed conclusion about their claims. We opened 14 in depth and tested five, using only public code, data, live products and official documents. They were chosen because they were testable, not because they were the best, and they are not a sample of the field. The methodology and all 100 scores are published separately.

What the five taught us is that a claim has layers, and confirming one tells you surprisingly little about the next.

Disclosure: sound.fan’s founder entered this competition; neither entry reached the Top 100. sound.fan sells claim verification to hackathon organisers.

When the answer key is public

TradePass sells licensing exam prep for electricians and says its practice tests are built to each state’s published exam outline. That kind of claim has an answer key: the testing company’s official candidate bulletin.

For the Texas journeyman exam, PSI’s current bulletin specifies two portions: 59 questions in 130 minutes, then 26 in 110, with 70% needed on each. TradePass matches to the question and the minute. The master electrician exam matches too.

The same bulletin also corrected TradePass’s story. Its submission describes the Texas exam as “one closed-book exam”. The official bulletins, including the one in force at the deadline, say candidates bring the electrical code into the room: the exam is open book.

An authoritative source is the cleanest verification there is. It confirms what matches and corrects what doesn’t, with unusually little interpretive ambiguity. It is also rare. Most claims an AI business makes about itself have no answer key.

What does “AI runs the business” mean?

Sloane & Pearl’s listing is the most vivid in the Top 100: “A real fashion brand where AI, not a human, pauses ad spend, rewrites strategy nightly, and writes product copy via Gemini.”

There is nothing to run and no bulletin to consult. But the team published code excerpts and logs, and they change the question from is it true? to what is it?

The answer is three different systems.

Pausing ads is a rule. After $22 of spend with no add-to-cart, or $37 with no purchase, the campaign stops. No model is consulted.

Rewriting strategy each night is Claude, Anthropic’s model, with code capping how far its suggestions can move.

Writing product descriptions is Gemini, switched on for this store on 12–13 August, as the team said. The August bill for it came to less than one euro-cent, consistent with a handful of new listings. XPRIZE allowed additional model providers, and the team disclosed its use of both.

“No human” is true in all three places. But “AI” was naming three decision-makers that fail in different ways. A rule fails predictably and can be audited by reading it. A strategy model fails in ways you find only by reviewing what it wrote. A copywriting model fails in front of customers. And of the three decisions in that headline, the one that most directly controls spending involves no model call at all.

That is the first layer: mechanism. For any AI-operated business, the useful question is where a model enters the decision chain and what checks it. We could see this one because the team showed it. We could not confirm the excerpts match what runs in production. Hold that thought.

A number can be exact and still depend on a definition

Doppelganger lets researchers run a survey on AI respondents before paying human participants. Its evidence: an ensemble of models reproduced about 91% of 17 classic behavioural-science effects, including anchoring, framing and the sunk-cost fallacy. The team published every simulated response and the scoring code.

We reran it and got 91.2%, exactly. The mechanism is visible and the number reproduces. By the standard of the traffic cone, we were done.

Then we read the rule. An effect counts as reproduced if any one of five models shows it; the repository itself calls this an optimistic ceiling. Scored with the same code on the same responses:

Rule for combining five models	Effects reproduced
Any one model	91.2%
Best single model	79.4%
All responses pooled	73.5%
Majority of models	67.6%

Nothing is hidden and no figure is wrong. The 91.2% is true of a definition, and the choice of definition is worth nearly 24 percentage points.

The benchmark has a quieter limit too. All 17 effects are known to exist in people; none is a case where the right answer is “nothing happens”. A system that always produced the expected bias would score perfectly, and would never warn a researcher off a doomed study, which is exactly what the product promises to do.

That is the second layer: measurement. Reproducing a number confirms the arithmetic. Understanding it means knowing the rule, and asking what a bad system would score.

Real evidence, smaller claim

Veritas is a legal translation service built around a specific fear: $500,000 quietly becoming $50,000 in a sentence that reads perfectly. It says it preserves 92.68% of “legal invariants” on the WMT25 benchmark’s English→Egyptian Arabic test, against 82.93% for the nearest of ten frontier systems. It names the test set precisely: 87 documents, 332 segments.

Precision is a gift to a verifier. We downloaded the official WMT25 data. Eighty-seven documents. Three hundred and thirty-two segments. An independent source, matching exactly.

The same files label every segment’s domain: literary, news, social media, speech. Legal segments: zero. “Invariant preservation” isn’t a WMT25 metric either; it is Veritas’s own measure of whether figures, dates and cross-references survive translation, which matters in legal work and exists in any text.

Nothing here is false. The dataset is real and cited exactly. But the headline says legal, and the evidence supports a result on general-domain Egyptian Arabic, scored with the team’s own yardstick.

That is the third layer: scope. Corroborated does not mean every implication follows. The claim didn’t fail; it shrank to what its evidence can hold. And it was the precision of the citation that made the narrowing possible. A vaguer claim could only have been doubted.

The boring number

Go back to the traffic cone. The same Polyfork submission that makes the triangle claim also says the company has 54 paying customers.

It is the most ordinary sentence in the submission, and the only kind of claim we could not touch.

The same was true everywhere. TradePass’s exam structure checked out to the minute; its five subscribers at the deadline stayed out of reach. Sloane & Pearl’s mechanism could be mapped line by line; its sales could not. Across five businesses, we could not independently confirm a single revenue, customer or usage figure.

That is not suspicion. Several teams volunteered unflattering numbers, including a $9,330 loss and an acquisition cost 2.3 times a customer’s lifetime value. Inaccessible evidence is not evidence against a claim. It is the edge of what an outsider can know.

What we could verify had little to do with how impressive a claim sounded. It depended on where the evidence lived. Geometry lives in code shipped to users. Exam rules live in a public bulletin. Benchmark responses live in a repository. Customers live in a payment processor.

That is the fourth layer: evidence. The question stops being is it true? and becomes is it knowable, and by whom? The projects we could investigate most deeply were not necessarily the strongest in the Top 100. They were the ones that exposed the most.

The last layer

Take stock of what a stranger with a laptop managed. We ran code, reproduced a measurement and rescored it, mapped a decision chain, checked a claim against the authority that defines it, and redrew a headline to fit its data.

Public evidence took us surprisingly far through the technical layers. It stopped most consistently at the commercial one: whether the business is real in commercial terms.

So the next question is narrow: can an outside verifier move one real business claim from participant-reported to independently reconciled?

The target is Sloane & Pearl’s “$18,641.28 in sales from 204 customers”. It is specific to the cent. It is the claim that says this is a store with real buyers rather than a demo. The team has already published the scripts that calculate it, so the records needed and the arithmetic to apply are defined in advance. A hashed Shopify order export could reconcile the reported totals without exposing a single customer. That would verify the claim against the store’s records; stronger assurance would require tying those orders to an independent payment or financial record.

We are not choosing it because we doubt it. We are choosing it because it is the next boundary, and it has been drawn precisely enough to test.

Public verification stops there. Independent verification doesn’t have to.

---

If you are a participant and have a correction or evidence that materially changes a finding, submit it through our correction form. We update verification records when new evidence warrants it.

Verification records: Polyfork · Sloane & Pearl · Veritas · Doppelganger · TradePass. Evidence frozen 17 September 2026, 07:30 UTC.