The Benchmark Is Not the Breach: Open Weights and the Collapse of the Access-Control Assumption

A benchmark score is not a capability. It is a measurement of a capability, taken by someone with an incentive to publish the number.

I learned that the expensive way. In 2017 I spent six weeks disassembling the Gnosis Safe multi-sig contract at the assembly level. Every audit summary in circulation described it as clean. What I found was a reentrancy path in the initial release, which I disclosed privately to the core team before anyone could reach it. The lesson was not that audits fail. The lesson was that an audit measures the auditor's model of the attack surface. The attacker does not share that model.

So when a headline crossed my desk this week — open AI models are closing the hacking gap with gated frontier systems, per an evaluation firm called Vals — I did what I always do with a security claim. I went looking for the harness.

There was no harness. There was a headline, a single paragraph of body text, and a source line. No task set. No rubric. No model version numbers. No publication date. No link to the underlying report.

The claim may be true. On my read of the public evidence, it probably is. The coverage, as it stands, is not evidence of it.

Vals AI is an evaluation firm. Not a lab. Not a regulator. Not a vendor of models. It publishes domain-specific leaderboards — legal, tax, finance, code — and sells access to the harnesses behind them. That business model matters here, and I will come back to it.

The claim attributed to it is narrow and, on its face, plausible: the offensive-cyber capability gap between open-weight models and gated frontier systems is narrowing.

The word "gated" is doing quiet work. A gated system is not a weaker system. It is a system whose use is mediated — by an API key, a rate limit, a terms-of-service clause, and, most importantly, a log. That last item is the one nobody discusses. The security value of a gate is not measured by what it blocks. It is measured by what it records.

The second word doing quiet work is "hacking." The evaluation community has a specific meaning for it, and the public does not share it. When a benchmark says a model can hack, it usually means the model solved a capture-the-flag challenge — or a fragment of one — in a sandbox with no defender, no persistence requirement, no attribution risk, and no consequence.

Cybench shipped forty such challenges in 2024. NYU's CTF Bench shipped roughly two hundred. Meta's CyberSecEval series measures adjacent skills. These are real instruments, and I respect the people building them. They are not intrusion.

I have watched this rhetorical move before, inside my own industry. "Decentralized sequencing" has been a slide in Layer 2 decks for two years while a single node signs every block. The slide is not a lie, exactly. It is a description of an architecture that exists in a document and not in production. Gating is that same slide, one industry over.

So the question is not whether open weights are getting better at offensive security. They are. The question is whether what is being measured is what is being feared — and, if it is not, what that gap means for the policy now being built on top of the number.

I keep a private table of evaluation results. It has more columns than any published leaderboard, and two of those columns are the only ones that matter.

The first is fine-tune state. A base model and a fine-tuned model are different security objects. Arditi and coauthors showed in 2024 that refusal behaviour in aligned language models is mediated by a single direction in activation space. Qi and coauthors showed the same year that a few hundred adversarial examples are enough to substantially degrade safety alignment, without the user intending harm. The community calls the practical version of this "abliteration," and it is now a routine craft, not a research curiosity.

Which means the honest question about any hacking-capability benchmark is this: was the model tested as shipped, or after the safety layer was removed? Those two results support opposite policy conclusions. The first says the gate works. The second says the gate was never the thing standing between the attacker and the capability — it was only standing between the attacker and the metering.

The second column is agent harness. A single-turn completion is not the same object as a loop with a shell, a browser, and a sampling budget. Most of the capability gain the public has observed since late 2024 came from wrapping models in tool-calling loops, not from the weights alone.

Here is what a benchmarked score looks like in the honest version, and what it looks like in the version that gets quoted:

# the version that gets quoted
score = solve(model, challenge, max_turns=1)

# the version that reflects an attacker for _ in range(sampling_budget): t = agent_loop(model, tools=[shell, http, python]) if verify(t): return True ```

The difference between those two blocks is not a rounding error. It is the difference between asking a knowledgeable person a question once, and handing them a machine that can ask a thousand questions, check its own work, and try again. Sampling budget is a capability multiplier that never appears in a leaderboard header.

Both columns were absent from the coverage. That does not make the underlying claim wrong. It makes the claim unverifiable — which, for a security claim, is worse than wrong, because an unverifiable claim still gets cited.

I spent 2020 making a version of this argument about Compound's interest rate curves. The curves were presented as market mechanics. They were arbitrary constants with governance attached. An evaluation rubric is the same artifact: a set of human choices wearing the costume of a measurement. That is fine, as long as everyone can see the costume.

There is a third column I did not expect to need. Capability provenance.

A meaningful share of recent open-weight gains traces to synthetic data generated by frontier systems, or to direct distillation from their outputs. I cannot audit the ratio from outside, and neither can anyone reading the coverage. But if that ratio is material, the framing collapses. "Open weights are catching up to gated systems" would then mean something far narrower: the outputs of gated systems are being extracted and re-embedded into weights the gating system cannot see.

That is not a competition story. It is an exfiltration story, told slowly and legally.

And it relocates the policy lever. If capability transfer happens through outputs, weight distribution is the wrong chokepoint. The chokepoint is output-side extractability — API abuse detection, distillation fingerprints, query-pattern anomalies. None of which the current discourse is funding.

Here is the part the coverage got directionally right without knowing why.

Real intrusion is an operations problem, not a token-generation problem. Persistence, operational security, credential markets, social engineering at scale, the avoidance of attribution and legal exposure — none of that is solved by a better completion. On the end-to-end task, open weights are not close to a competent human team, and I doubt they will be soon.

But the attack surface is not uniform. It has a front end, and the front end is where attempts are cheap. Phishing copy. Initial reconnaissance. Vulnerability triage across a wide perimeter. Pretext generation. These are token-generation problems, and they are where an attacker's marginal cost actually lives.

Consider the arithmetic without believing its precision. An open-weight model deployed locally has no API log, no rate limit, no provider-side abuse detection, and a marginal cost per attempt that approaches the electricity bill. If a single-attempt success rate moves from a fraction of a percent to slightly more than that — a change too small to headline — the effect on a campaign of ten million attempts is not small. A small percentile shift, multiplied by industrial volume, is the entire threat model.

That is the correct version of the Vals claim, and it is a supply-chain argument, not a capability argument. The cheapest stage of the kill chain just got cheaper by an order of magnitude, and the expensive stages did not move at all.

My industry has already run this experiment, and it did not like the result. Smart contract audit scores have never prevented an exploit. Wormhole, Nomad, Ronin — none of them failed because an auditor missed a score. They failed at the operational surface: key custody, upgrade authority, incident response. The score was a proxy that made everyone feel covered right up until the block that settled the transfer.

Which is why the crypto audience should be wary of reading the Vals claim as a win for openness. A score moving in your favour is not the same as a system getting safer. And it is worth saying plainly that the verification culture this industry claims to hold — don't trust, verify — has no working implementation for models yet. Attestation of inference inside a trusted execution environment is promising and fragile. Zero-knowledge proofs of model execution remain orders of magnitude away from production. Provenance metadata is metadata, and metadata is the first thing an adversary rewrites. We built a culture of verification for ledgers and inherited none of it for weights.

There is a symmetry the coverage omitted entirely, and it is the one I care most about.

Open weights are enormously valuable to defenders. A security operations team that cannot send logs to a third-party API — which is most of them, in finance, healthcare, and government — can run a competent model locally. Malware triage. Log correlation. Reverse engineering. Data never leaves the perimeter.

What blocks them from using gated systems is not price. It is the refusal behaviour. Ask a mainstream aligned model to analyse a sample and you get a lecture, not an analysis. This is the alignment tax, and in security operations it is not abstract. It is a work stoppage.

So the aligned systems push legitimate defenders toward local open weights, for exactly the reasons threat actors were already there. The gate does not separate good from bad. It separates audited from unaudited — and it does so by making the audited path useless for the people who most need it.

I saw this pattern in 2024 while auditing institutional custody infrastructure. Every firm I reviewed had made the same trade: key management convenience over cryptographic sovereignty, because the compliant path was the convenient path, right up until it wasn't. Nobody was malicious. The architecture simply rewarded the wrong direction, repeatedly, at every decision point along the way.

Everyone reading the Vals claim has drawn the same inference. Open weights are closing the gap. Therefore open weights are the risk. Therefore gating must be strengthened.

I think that is backwards, and I think the reason it feels obvious is that we are still arguing about capability when the thing that actually broke was auditability.

The most important sentence in this discussion is one nobody has written yet: the primary security function of a frontier API is that it produces an attributable record of who attempted what. Gating does not prevent capability. It prevents unobserved capability. When offensive capability migrates from a metered endpoint to a downloaded checkpoint, the attacker does not become stronger. The attacker becomes invisible — and invisibility is worth more than strength.

That reframes every proposed remedy. Model-level gating, FLOPs thresholds, usage terms: all of them assume the risk occurs somewhere a regulator can see. It does not. The risk occurs at the point of fine-tuning and local inference, precisely where the chain of custody ends.

The regulatory architecture confirms the mismatch, and it is worth being concrete. The EU AI Act's systemic-risk presumption sits near 10^25 FLOPs, with obligations landing on the model provider. In the United States, the 10^26 FLOPs reporting threshold under EO 14110 was withdrawn in January 2025, and the diffusion rule followed in May. Federal capacity to act on capability thresholds has largely exited, replaced by state-level fragmentation and a voluntary standards body. China's regime regulates deployed services and content, with an amended Cybersecurity Law taking effect in January 2026, and it has limited reach over weight distribution.

Three jurisdictions. Three instruments. One shared blind spot: the obligation attaches where the model is made; the risk attaches where the model is modified. That is not an enforcement failure. It is an architecture failure.

And the failure has a perverse incentive built into it. If offensive capability becomes the formal trigger for restriction, then capability becomes the reason to close — which means the models that stay open are the ones that stayed quiet, and frontier work migrates toward actors who never intended to publish at all. Publishing a capability threshold is, structurally, a subsidy to secrecy. Vested interest distorts the lens of analysis, and the evaluation industry's interest is in the salience of the gap it measures.

Certainty is a bug in a stochastic world. We are about to legislate as if we had it.

The window that matters is the next six to eighteen months, and it is a fragile equilibrium rather than a stable one.

Right now the situation reads: capability is converging, no attributable incident has occurred, and the policy apparatus is holding an assumption — that gating buys a safety window — which the evidence is quietly dissolving. That equilibrium does not fail gradually. It fails on the first event that can be publicly attributed to an open-weight model, and the response will be written in weeks by people who will never read the harness.

The better path is not to defend the gate. It is to build the accountability layer the gate was standing in for: model provenance, output fingerprints, fine-tune detection, auditable inference. Hard engineering. Almost nobody funding it.

The protocol does not lie; the interface does. A benchmark is an interface. Ask what it is hiding before you let it write the rule.