Verification Cost, Benchmark Decay and the Repricing of Real-World Data

1. In short

AI models can now produce answers very cheaply. What is hard, and getting more valuable, is knowing whether those answers are actually right.

That changes what data is worth. Bulk data that anyone can buy is becoming cheaper. Fresh, real-world data that has not been seen before, and that comes with proof of where it came from, is becoming more valuable, because it is what lets people check AI properly.

This note follows on from From Models to Reality, which argued that the real advantage in AI is moving from the models themselves to the data behind them. Three months on, prices in the market are starting to show it.

2. Three things that changed this year

Running AI now costs more than building it. Gartner forecasts that in 2026, for the first time, more cloud money will go on running AI models (known as inference) than on training them: about $23.3 billion against $19 billion, or 55% of the total [6].

Figure 1. Forecast worldwide spend on AI-optimised cloud infrastructure in 2026. Source: Gartner [6].

The overall pot is also growing fast. Gartner expects worldwide spending on AI-optimised cloud infrastructure to almost double, from $21.5 billion in 2025 to about $42 billion in 2026, with the share going on running models rising again to 59% in 2027 [6]. More and more of the money in AI is spent on using it, not on making it.

Figure 2. Worldwide spend on AI-optimised cloud infrastructure. Source: Gartner [6].

Each answer gets cheaper, but tasks use far more of them. Epoch AI found that the price of getting the same quality of result from AI falls very fast: typically about 50 times a year, and in some cases up to 900 times [4]. But AI is now used as an agent that plans, uses tools, retries and checks itself. Anthropic reports that a single agent uses about 4 times as much AI work as a chat answer, and a team of agents about 15 times [12]. Gartner puts the range at 5 to 30 times [7]. Cheaper answers, many more of them, and a bill that still grows.

Figure 3. What $100 of AI work would cost a year later, at the slowest, typical and fastest rates of decline Epoch AI measured. Source: Epoch AI [4].

Figure 4. AI work used per task compared with a single chat answer. Source: Anthropic [12].

The tests used to rank AI are wearing out. Well-known tests such as MMLU are now so easy for leading models that they no longer tell them apart [13]. Worse, models sometimes remember test questions from their training data. In February 2026 OpenAI stopped reporting results on SWE-bench Verified, a popular coding test, after finding that leading models could reproduce the expected answers from memory [8]. Researchers found that models could find the broken file far more often on code from that famous test than on code they had not seen before [11].

Figure 5. How often AI models found the broken file using only the bug description. Source: Liang et al. [11].

Put these together and the picture is simple. Getting an answer from AI is no longer the hard part. Knowing whether it is right is.

3. What a correct answer really costs

Buyers do not really want AI text. They want answers that have been checked and found correct. The cost of one correct, checked answer is roughly:

cost of one correct answer = (cost to run the AI + cost to check it) ÷ chance it is right

As running AI gets close to free, almost all of the cost is in checking, and in having an honest measure of how often it is right. Both of those depend on good data.

How expensive checking is depends a lot on the type of work:

• Maths and code. Checking is almost free. A proof works or it doesn't, and the tests pass or they don't. This is where AI is improving fastest.
• Software tasks like using a website or an app. To check the AI, you need a working copy of the software to test it in. Interviews by Epoch AI put a single practice task at roughly $200 to $2,000, a copy of a website at around $20,000, and a full copy of an app like Slack at around $300,000 [5].
• Expert judgement in law, finance or medicine. Checking needs a qualified expert, which takes paid expert hours for every answer.
• Physical work in trades, warehouses, factories and care. There is no cheap way to check at all. Either the part fits or it doesn't, and either the pallet is stable or it isn't. Checking needs real footage of real work.

Figure 6. Reported costs of building environments to test and train AI on software tasks. Source: Epoch AI [5]; the website figure is a SemiAnalysis estimate cited by Epoch.

Figure 7. Illustrative cost of one correct, checked answer, split into running the AI and checking the answer. As the work moves from code to the physical world, checking becomes almost the whole cost.

A worked example

Take a task in physical work, such as judging whether a warehouse worker stacked a pallet safely. Suppose an AI attempt costs $2 to run, a trained reviewer costs $150 to check one attempt against real footage, and the AI gets it right 35% of the time. Each correct, checked answer then costs about $434.

Now imagine running the AI became completely free. The cost would only fall to about $429, a drop of around 1%. For maths and code, where checking is almost free, the same change cuts the cost by about 95%. That is the whole argument of this note in one example: cheaper computing helps most where checking is already cheap, and barely helps where it is not.

Figure 8. Illustrative. Cost of one correct, checked answer today and if running the AI cost nothing, using the same assumptions as the previous figure.

Figure 9. Illustrative. AI progresses fastest where answers are cheap to check, and slowest where checking needs people or the physical world.

This explains something that can look odd. An hour of video of someone doing skilled physical work is not worth much as general training data, because there is plenty of video in the world. But the same hour is worth a lot as a way of checking AI in the one area where no cheaper check exists. The value is not in the pixels. It is in the ground truth they provide.

4. Why tests wear out

Training data can be shared without losing much value. Test data cannot. A test is only useful if the AI has never seen it. Once a test has been published, quoted, used again and again or resold, models start to learn it, and it stops measuring real ability.

So every test has a shelf life:

• Public tests wear out fastest, because every new model is measured on them and their questions spread.
• Private tests last longer, but they still wear out each time they are used or shared.
• Fresh real-world footage can be renewed. A new site, a new worker or a new week of work gives a brand-new test, as long as the date, place and person are recorded with the file and cannot be changed later.

Figure 10. Illustrative. How quickly different kinds of test lose their usefulness. Fresh real-world capture is topped up with each new site or period, so it keeps its value.

Some test designers now accept this openly. LiveBench, for example, adds and updates its questions every month [9].

For real-world work, renewing a test is simpler still. Record the same kind of task at a site the AI has never seen, with a different worker, in a different week, and you have a fresh test by construction. What makes it trustworthy is not the footage itself but the record around it: when and where it was captured, by whom, and who has seen it since.

This is also why buyers of real-world data increasingly ask for terms like "captured in the last few weeks", "sold to no more than one other buyer" and "never sold through a broker". These are not awkward demands. They are ways of making sure the data has not already been seen, so the test results can be trusted. It is the same logic that makes exclusive deals on AI test environments cost about four to five times as much as shared ones [5].

5. The data market is splitting in two

If this is right, there are now two data markets moving in opposite directions.

• Bulk data is used for general training. It is not exclusive, it is often resold, AI-generated data competes with it, and its history is often unclear. Its price is falling.
• Ground-truth data is fresh, exclusive and well documented. It is used to check AI, to build test environments and to measure real progress. Its price depends on how new it is, how many people have seen it, and how good the proof behind it is. Its price is rising.

The dividing line is proof. If a seller cannot show when a file was captured, who captured it and who else has bought it, then a promise of exclusivity is just a promise, and the data is worth bulk prices whatever the contract says. The real asset is not just the footage. It is the footage plus the evidence that it is fresh, unseen and properly consented.

The money involved is already large. Epoch AI's interviews found that contracts for AI test and training environments often run to six or seven figures a quarter, and it cites a report that Anthropic discussed spending over $1 billion on such environments in a single year [5]. Buyers paying at that scale have every reason to insist on proof.

6. Where AI still falls short

Leading AI models are very good at short tasks that are easy to check. They are weaker elsewhere. Stanford's 2026 AI Index found that AI agents jumped from about 12% to about 66% success on a standard set of computer tasks, but still fail roughly one task in three [10]. The same report found that the most capable models are now the least transparent about how they were built [10].

Figure 11. Success on OSWorld, a standard set of computer tasks. Source: Stanford HAI, 2026 AI Index [10].

And those are tasks on a computer, where checking is automatic. The harder gaps are in long tasks, recovering when something goes wrong, work in the physical world, and judgement calls. These are the same areas where checking is expensive and good tests are rare. That is not a coincidence. AI improves where it can be checked.

7. What a trustworthy data record should include

We suggest that any data used to back up a claim about AI ability should come with five pieces of information:

1. When and how it was captured. The original time, device and a digital fingerprint of the file, recorded at capture and never changed.
2. Who has seen it. Every buyer it has been supplied to, and every public use.
3. Whether this is the first sale. And if not, who bought it before.
4. Who made it and who checked it. A consistent, consented identity for the worker and the reviewer, so you can show the reviewer is independent. Harbor Passport is built for this.
5. How it was checked. The version of the scoring guide, who scored it, and which unseen test it was measured on.

With this in place, a claim like "our model scores 80%" can be stated in a form that actually means something: what each correct answer cost, on which unseen test, and how many times that test has been used before.

This builds on the record set out in Attribution After Grounding, which asked what caused an improvement. This note asks what the improvement cost and whether the test that measured it can still be trusted.

8. What this means in practice

For teams buying data or testing AI. Ask for the capture date, the first-sale status and the list of previous buyers before you pay. Keep some real-world tests back that no model, and no vendor, has seen. Report results as cost per correct answer on a named, unseen test, not just a score.

For people selling data. The proof is part of the product. Record capture time, place, consent and identity at the moment of capture, because they cannot be added honestly later. Track every buyer. Exclusive, recent and well-documented footage is worth several times as much as footage that has been passed around.

For the people doing the work. Your skill is what makes the data valuable. A consistent, consented identity lets that skill be recognised and paid for over time, instead of disappearing into an anonymous pile of clips.

9. Common objections

"AI can check AI." For maths and code, yes. For physical and expert work, an AI checker trained on the same kind of data tends to share the same blind spots as the model it is checking, so both get it wrong together. We made a similar point in Continuous Generation, Inference Loops, and the Return of Real Grounding: a convincing wrong answer checked by a convincing wrong checker is still wrong.

"Synthetic data can replace real tests." It can add variety, and it is very useful for training. But whoever generates synthetic data also decides what counts as correct, so it cannot be the final, independent test.

"Bigger models will solve it." Scale helps with training. This note is about the tests that tell us whether scale actually helped.

"Proof of origin is extra cost." For bulk data, yes, and that market will skip it. For ground-truth data, the proof is the product.

10. What we expect next

Over the next 12 to 18 months we expect:

• Fresh, exclusive, well-documented data to rise in price compared with shared data, even as the total amount of footage grows.
• Limits on data age, first-sale terms and broker bans to become standard in AI data contracts.
• AI leaderboards to start reporting cost per correct answer and how fresh their tests are, not just raw scores.
• Tests based on physical work to stay useful the longest, because every new site is a new test.

What would prove us wrong: a cheap, independent way to check physical work that does not share the AI's blind spots. If that appears, the premium for ground truth will fall. We would welcome it, but we do not expect it soon.

11. Where this leaves us

Cheap computing made answers cheap. It did not make them true. The market is starting to put a price on the difference, and that difference is ground truth: data that is fresh, unseen, consented and traceable.

That is where Harbor works. We commission real-world capture in real workplaces, against a scoring guide written before recording starts. We attach the capture time, consent, worker identity and sales history to every file, so buyers can check freshness and exclusivity for themselves instead of taking them on trust. In the half of the market that is growing, that proof is what the data is worth.

References

[1] From Models to Reality · Harbor Research, August 2026

[2] Continuous Generation, Inference Loops, and the Return of Real Grounding · Harbor Research, September 2026

[3] Attribution After Grounding · Harbor Research, October 2026

[4] LLM inference prices have fallen rapidly but unequally across tasks · Epoch AI, March 2025

[5] An FAQ on Reinforcement Learning Environments · Epoch AI, January 2026 (website figure cited from SemiAnalysis)

[6] Gartner forecasts AI-optimised IaaS spending to grow 96% in 2026 · Gartner, August 2026; figures as reported by Tech Times

[7] AI inference is getting cheaper, but your agents are getting more expensive · CIO, August 2026, citing Gartner

[8] Why we no longer evaluate SWE-bench Verified · OpenAI, February 2026

[9] LiveBench: A Challenging, Contamination-Limited LLM Benchmark · White et al., 2024

[10] The 2026 AI Index Report · Stanford HAI, April 2026

[11] The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason · Liang et al., 2025

[12] How we built our multi-agent research system · Anthropic, June 2025

[13] The 2025 AI Index Report: Technical Performance · Stanford HAI, April 2025