Inside Coinbase's onramp pipeline, 813 confirmed fraud cases sat buried in 16,140 transactions. The company froze its screening guidance, froze its policy thresholds, and changed exactly one thing: the decision model doing the classification. Then it replayed nine weeks of history β 7,293 users, same rules, new brains. The newest large language models missed more fraud than the models they were meant to replace. One upgrade pushed precision up 11.5 percentage points while recall collapsed 20.7 points. Fewer false alarms, more real criminals walking through the door.
This is not a hack, a bridge exploit, or a token unlock. It is something quieter and, for anyone building financial infrastructure, more unsettling: a compliance system that got worse at its job while every dashboard reported an upgrade.
Onramp is where fiat becomes crypto β the first door a user walks through, and historically the most fraud-dense corridor in any exchange. Coinbase's fraud screening here is not a product feature; it is a legal obligation under anti-money-laundering rules. Get it wrong in one direction and you annoy legitimate customers. Get it wrong in the other and you become a laundering channel. The company said no customer losses were confirmed, which matters legally as much as it matters technically.
The research behind the disclosure, first surfacing in a SR-Fraud paper and later in Coinbase's own blog, used a methodology I wish more teams adopted. Rather than rebuilding the whole screening stack and guessing what changed, the team isolated a single variable β the decision model β and replayed a fixed historical dataset through it. Fixed guidance, fixed policy, one moving part. In software terms, this is an ablation study: you remove the noise so the signal has nowhere to hide.
I spent my early twenties doing something adjacent. In 2017, as a nineteen-year-old economics student in Tokyo, I manually audited ICO smart contracts for three months, hunting for logic flaws in token distribution. What I learned then still holds: the most dangerous bugs are never the ones that crash the system. They are the ones that let it keep running, quietly wrong. A model that misses 20% more fraud does not throw an error. It files a clean report.
The dataset itself deserves scrutiny. 813 fraud cases out of 16,140 transactions is a 5% fraud density β orders of magnitude above the 0.1% to 1% range real payment flows typically carry. This is an enriched sample, deliberately loaded with positives so the test has enough signal to measure. That is legitimate science, but it also means the regression magnitude cannot be pasted onto production traffic without adjustment. Low-base-rate recall is notoriously unstable.
Here is what the numbers actually show. Across the model families tested, the newer generations did not uniformly improve. For the GPT line, the shift was stark: precision climbed 11.5 percentage points while recall fell 20.7 points. In plain language, the model became more conservative β it raised its bar for calling something fraud. That sounds responsible until you realize the bar is now above a fifth of real fraud.
Sonnet's regression was subtler and, in some ways, more troubling. Its ordinary recall dropped 22.2 percentage points. Its amount-weighted recall β which counts a missed $50,000 transaction as heavier than a missed $50 β dropped 22.9 points. The weighted figure falling further than the unweighted one tells you something specific: the fraud slipping through skews large. The model is not randomly blind. It is disproportionately blind to the transactions that hurt most.
This is the first insight most coverage buried: an upgrade that looks neutral on a dashboard can be biased toward the worst-case misses.
Then there is the part that deserves to be the headline. Coinbase fine-tuned a Qwen3.5-9B model β nine billion parameters, a fraction of the frontier systems it was benchmarked against β and it beat Opus 4.5 on four separate metrics. It also cut end-to-end median latency from 1.515 seconds to 0.683 seconds, a 55% reduction. On a narrow, well-defined task with a deterministic reward signal, a specialized small model outperformed a generalist giant and ran more than twice as fast.
This is not an anomaly. It is a pattern the industry keeps rediscovering and keeps forgetting: domain fine-tuning plus a deterministic reward beats general capability on narrow tasks. The reason is mechanical. General models are optimized for helpfulness, safety, and breadth. Those objectives shape a decision boundary tuned for the average of everything. Fraud classification needs a boundary tuned for one thing. When you upgrade a generalist, you may be moving that boundary in a direction that is better on average and worse for you.
The same logic runs through DeFi, in a form I have argued about for years. Lending protocols dress up hand-picked interest rate curves as if they were discovered from market supply and demand. They were not. They were authored β a handful of parameters chosen by a team, then defended as organic. Fraud thresholds work the same way. They are not found in the data. They are written, and every model generation silently reinterprets them.
And here is where the story stops being about accuracy and starts being about accountability. Coinbase stated it could not determine why the regression occurred. Read that again. A system that decides which customers get access to financial services, deployed in production, cannot explain why its own behavior shifted. The policy layer is centralized. The guidance is centralized. The model is a black box even to the people operating it.
There is a supply-chain angle too. Coinbase tested models from Anthropic, from OpenAI, and from Alibaba's Qwen line. Choosing to benchmark a Chinese open-weight model alongside American frontier systems, and then to deploy it, is a quiet statement about vendor dependency. No serious financial institution wants to be locked to a single model provider whose internal changes it cannot see. The dual-track architecture β a general model for breadth, a fine-tuned model for the critical path β is where this is heading.
We spend enormous energy in this industry debating data availability layers β arguing whether rollups need dedicated DA when 99% of them do not generate enough data to justify one. Meanwhile, the data pipeline that decides who gets a bank account runs on a proprietary dataset that no outsider can inspect. Tracing the code back to the conscience is easy when the code is on-chain. It is nearly impossible when the code is a model and the model is a vendor's secret.
Coinbase deserves real credit for one thing: it published a negative result. Open books, open ledgers, open hearts β that is the ethos. In an industry that markets every upgrade as progress, voluntarily disclosing that newer models performed worse is an act of institutional honesty. But transparency in the disclosure and opacity in the mechanism are not the same virtue.
The contrarian reading is that this story is not about failure at all. The framing β 'new models miss more fraud' β is the clickbait layer. The durable finding is that Coinbase built and deployed a specialized model that beats frontier systems on cost and latency. That is the transferable lesson for every team running automated decisions: stop assuming the biggest model is the right model, and start measuring.
There is a second contrarian point that cuts against the researchers themselves. The 5% fraud density means the recall regression may be overstated for real traffic. In low-base-rate conditions, recall becomes statistically fragile; a handful of missed cases can swing the percentage wildly. The honest conclusion is not 'new models are worse.' It is 'we do not know how much worse, and the proprietary dataset prevents anyone from checking.'
And a third, uncomfortable one: this is a fully centralized decision system at a company whose entire industry preaches verifiability. There is no on-chain record, no external auditor, no reproducible benchmark. Building bridges where others build walls applies here too β but the wall, this time, is a dataset kept behind a corporate door.
Coinbase's own recommendation is the right one: test any configuration change against historical replay before it ships, and watch amount-weighted recall, not just headline accuracy. The broader lesson is harder. In a market that rewards the appearance of an upgrade, the discipline is refusing to trust it. The audit is not the end, but the beginning. The models will keep getting bigger. The question is whether we keep getting more careful.
