Anthropic Wants Independent AI Evaluators. Crypto Already Ran That Experiment.

Chaos detected. Analysis loading.

A crypto outlet published a story recently about Anthropic introducing independent AI evaluators after security incidents. I read it three times looking for a fact.

Here is the inventory: no named evaluation firm. No date. No scope. No access level. No executive quote. No link to a primary source. Four information points total, three of them lifted from the title and abstract, repeating each other.

That is not a news event. That is a placeholder.

But placeholders are worth reading, because they mark where an industry intends to move. And I have seen this exact move before — not in AI. In crypto. For a decade, this industry has run the independent-audit experiment at scale, with real money and real bodies. The results are in. They are not encouraging.

Three pressures converge on Anthropic right now. The EU AI Act's obligations on general-purpose AI models have entered enforcement. California's SB 53 puts a reporting duty on frontier developers for what the statute calls critical safety incidents. Enterprise procurement in finance, healthcare and defense now expects third-party risk documentation before a signature. The calculus is not mysterious: an evaluation program is cheaper than lost market access.

Translate that into crypto vocabulary and it takes four seconds. "Independent AI evaluator" is an auditor. "Security incident" is a post-mortem. "Trust infrastructure" is a badge.

We already have that market. It is called smart contract auditing. And I have sat in enough procurement conversations to know how it actually functions — the audited party selects the auditor, pays the auditor, scopes the engagement, and controls whether the report ever sees daylight. The report's primary function is not preventing failure. Its primary function is being held up during a fundraise.

I watched this happen in 2020, during DeFi Summer, while I was pulling flash loan arbitrage across Compound and Uniswap and publishing threads on oracle manipulation. Protocols bought reports the way they bought logos. Then in May 2022, when I mapped the Terra liquidation cascade hour by hour, I could not find a single audit that had priced the governance failure. It was never in scope. Governance failure is never in scope when the governance is the client.

That is the frame. Now the mechanics.

Autopsy protocol engaged. Four variables determine whether Anthropic's move is governance or theater — and each has a crypto analogue that already resolved.

One: access level decides everything. A black-box API evaluation can only falsify dangers the model already exhibits under test. It cannot exclude latent capability — the ability that exists but was never elicited. That is the capability elicitation problem, and it is unsolved. Crypto's version is the bytecode-only audit. I have reviewed both kinds of report. The bytecode-only audit misses the deploy script, the proxy admin key, the initialization function. The failure lives exactly where the auditor was not authorized to look. White-box access — weights, activations, training trajectories — triggers intellectual property, export control and secrecy constraints all at once. Nobody has resolved that tradeoff. It is the central unresolved problem, and the coverage does not mention it.

Two: no consensus benchmark exists. METR runs long-horizon task suites. The UK AI Safety Institute has Inspect. Apollo runs scheming and planning evaluations. Andon Labs runs autonomous economic behavior experiments. Different task sets, different scaffolds, different scoring. Which means "passed third-party evaluation" has no fixed meaning. Same as "audited" in crypto — it spans a two-day review and six months of formal verification, and the badge renders identically in both cases.

Three: scaffold sensitivity destroys reproducibility. The same model wrapped in a different agent harness can shift tens of points on comparable benchmarks. Run it again next month and the number moves again. On a surveillance desk, this is the reason I distrust single-run evaluations in either industry. Same contract, different fork state, different result. Nobody prints the harness version next to the badge.

Four: the cost is real, and small. Full frontier evaluation — long-horizon agentic red-teaming, large-scale sampling, interpretability probing — runs somewhere between hundreds of thousands and low millions of dollars per pass, and must be re-run every model iteration. Against training spend that is under a tenth of a percent. Cheap. Which is precisely why it will be purchased generously and read carelessly.

Now the variable that actually decides the outcome, and which nothing in the coverage has touched.

Does the evaluator have the right to publish a dissenting report?

That is the whole ballgame. If the answer is no — if findings sit behind an NDA and the public learns only that an evaluation "occurred" — the mechanism is a disclosure instrument, not a risk control instrument. It cannot stop a release. It can only document one. I have asked this question after every smart contract audit I have covered. The answer is always the same sentence: the engagement terms are confidential.

And here is the mechanistic skepticism, turned on my own side of the fence. Crypto's real fix for audit capture was never better auditors. It was cryptography. Proof of reserves. TEE attestation. Verifiable computation. Replacing trust in institutions with verifiable claims. The AI analogue is verifiable inference — proving that this output came from this model with these weights. That would make the PDF irrelevant.

I have to be honest about the arithmetic. ZK rollup proving is already economically brutal; operators bleed unless gas returns to bull-market levels. zkML is two to three orders of magnitude worse. There is no bear market in which verifiable inference ships at scale. So we get the institutional version, the badge, for the same reason crypto got it: it is what is affordable now.

Here is the angle nobody is writing. The evaluation industry is about to be minted, and its product is legitimacy, not security.

Look at the structure. Revenue depends on frontier labs. Labs select the evaluators. No mandated rotation. No regulatory designation of qualified evaluators. No public funding line. Every one of those conditions is what the audit profession was forced to adopt after it failed catastrophically — and only under legal compulsion. AI evaluation is launching from the pre-reform position and calling it independence.

Structurally, that badge is a DAO governance token: a non-dividend claim whose only path to value is a later buyer's belief. No cash flow. Its price is set by the next procurement officer who needs a document in the file. That is not a conspiracy. It is an incentive gradient, and gradients win.

Second unreported angle: the timing. Announcing external evaluation after an incident serves three functions simultaneously — crisis communications, regulatory expectation management, internal process hardening. That does not make it insincere. It makes the announcement the signal and the mechanism the question mark.

Third: new attack surface. Handing an external organization your system prompts, internal tooling, vulnerability lists and red-team findings opens a supply chain path into the lab. A breach at the evaluator becomes a breach at the evaluated. The causal chain in this story may be self-referential in ways nobody has checked.

Watch four signals, none of which appear in the current coverage. Whether the evaluator is named. Whether access is white-box. Whether dissent can be published. Whether the output is machine-verifiable or a PDF.

If the answers are unnamed, black-box, no, and PDF — then Anthropic bought market access and a standard-setting seat, cheaply, and the security language is the packaging. That is a rational purchase. It is not a safety mechanism.

Chaos detected. Still loading.

EOS didn't die; it evolved. Audit badges didn't die either. Do you?