Claude Sonnet just got worse at exactly the same task it was already failing. Same model. Same payloads. Same simulated tool environment. Only the scoring function changed — and its measured attack success rate collapsed by a factor of four to six.
That is not a measurement tweak. That is a paradigm collapse. The SADF study — presented at DEF CON 34's AI Village — did not merely add another benchmark to the pile. It falsified the scoring methodology that most prior agent-security evaluations silently depended on.
Here is the uncomfortable sequence: if your agent framework reports a 60% attack resistance, and the underlying model actually fails 15.5% of the time, then every downstream conclusion — framework comparison, model ranking, procurement decision, insurance premium — was computed against a corrupted oracle. The metric was the vulnerability.
Context: The Model-Centric Blind Spot
The agent security industry has spent 2024 through 2026 building evaluation suites that treat the LLM as the unit of analysis. Jailbreak benchmarks. Prompt-injection test sets. Red-team leaderboards. All of them share an implicit axiom: the model is the system.
That axiom was always false. It became catastrophically false the day we gave models tools. Once an agent can call APIs, read memory, invoke sub-agents, and write to external stores, the model is one component in a distributed execution pipeline. The orchestration framework — CrewAI, LangChain, AutoGen, SmolAgents — is not a wrapper. It is the execution plane where every security decision is serialized, routed, and persisted.
SADF — the Security Assessment of Agent Orchestration Frameworks study led by Julie Brunias and team — treats this shift as the object of study rather than an inconvenience. The methodology is deceptively simple: fix the model, fix the tool environment, and vary only the orchestration layer. What remains is a clean attribution of attack surface to framework architecture.
The experimental baseline is direct API access to Claude Sonnet. No framework. No orchestration. The model receives a prompt, calls tools directly under the test harness's supervision, and returns. That baseline registered an Attack Success Rate (ACR) of 15.5%. Against that number, four frameworks were measured: CrewAI at 11.9%, LangChain at 18.1%, AutoGen at 20.0%, and SmolAgents at 31.1%.
CrewAI beat the no-framework baseline. SmolAgents doubled it. And none of this was visible until the researchers fixed the scoring function.
Core: The Variable-Separation Principle
The experimental design is the contribution. By holding the model constant — Claude Sonnet throughout — SADF eliminates the single largest confounder in agent-security research. Any ACR difference between targets must be attributed to the framework layer. This is engineering-grade methodology applied to a problem that had previously been treated as a model-benchmarking exercise.
Think about what prior evaluations actually measured. A typical agent-security paper swaps both the model and the scaffolding between test runs. Different system prompts. Different temperature settings. Different tool descriptions. The result is a compound measurement that cannot be decomposed. When a LangChain-based agent scores worse than a raw GPT-4o API, you cannot tell whether the model degrades under orchestration or whether LangChain's routing logic introduces the failure. SADF's fixed-model approach dissolves that ambiguity.
The authors go further with a parallel measurement on Claude Haiku: a true ACR of 22.3% after correction. The spread between Sonnet's 15.5% and Haiku's 22.3% provides a secondary data point on model-level variance. But the framework delta — from 11.9% to 31.1% — is an order of magnitude more significant than the model delta. Framework choice moves the attack surface more than model choice does. That single finding reorders the enterprise procurement calculus.
I have been on the other side of this attribution problem. During my Groth16 circuit audit in 2024, the production team insisted the proving system was sound because the arithmetic circuit verified correctly. The vulnerability was not in the mathematics. It was in the challenge-generation phase — the orchestration logic around the circuit. We almost shipped a zero-knowledge protocol whose core cryptographic assumption held while its operational wrapper allowed duplicate spending. The lesson generalized: in any layered system, the adhesion layer between components is where soundness goes to die. SADF has now quantified that principle for agent frameworks.
The Refusal-Filtered Correction: Your Benchmark Was Lying
The most technically important result in the SADF study is not the framework ranking. It is the discovery that naive substring-matching scoring overestimates Claude model vulnerability by a factor of four to six. The correction methodology deserves scrutiny because it changes the reading of every number in the paper.
Naive substring matching works like this: the evaluator defines a set of attack-vector substrings — malicious tool arguments, injected instructions, poisoned context fragments — and checks whether the model's output contains them. If yes, the attack is flagged as successful.
For most open-weight models, this is a reasonable proxy. The model directly executes the injected instruction, the malicious string appears in tool arguments, and the test catches it. But Claude models behave differently. They refuse. A substantial fraction of Claude's responses to adversarial payloads are refusals — “I cannot assist with that request” — and refusals may or may not include the injected substring verbatim. When a refusal contains a quoted fragment of the malicious payload, substring matching flags a hit. The attack is counted as successful even though the model declined to execute it.
The result is systematic overestimation of vulnerability for models that refuse frequently. That is exactly what the researchers found: applying refusal-filtered scoring — where a response that refuses the instruction is scored as a non-attack unless the tool call actually executed malicious content — dropped Claude Sonnet's ACR from the inflated range to 15.5%. Claude Haiku landed at 22.3%.
The implications are corrosive for existing literature. Any agent-security evaluation that used substring matching and did not control for refusal behavior has an unquantified bias baked into its conclusions. Your 70% attack-resistance number for an agent built on Claude may actually be 85%. Your framework comparison that placed one orchestrator above another may be measuring refusal-echo artifacts rather than real security properties.
This is the same class of measurement error I encountered while fuzzing Compound's governance contract in 2020. I spent forty hours with Echidna hunting an integer overflow in the claimReward path, only to realize that my test harness was the problem — the property assertions I wrote were checking the wrong invariant. The contract appeared vulnerable because my oracle was naive, not because the logic actually failed. The discipline that stuck with me: before you audit the system, audit the measurement instrument. SADF applied that discipline at scale across 5,119 evaluation rows. The researchers did not just report numbers; they found their own test's blind spot and corrected it. That is rare. Most benchmark papers die with their errors intact.
The correction also validates a design choice that was initially controversial: routing all tests through a SimulatedToolEnvironment. Because no real systems or credentials were exposed, the researchers could run high-volume adversarial payloads without ethical or operational blowback. The isolation is a double-edged sword — more on that in the contrarian section — but it enabled the 5,119-row dataset without collateral damage.
— Log entry: measurement corrected.
The ACR Ledger: Reading the Framework Spread
The headline numbers form a ledger that procurement teams will soon need to read:
- Direct API (Claude Sonnet): 15.5% ACR — the control group, no framework.
- CrewAI: 11.9% ACR — below baseline. The framework reduced attack success.
- LangChain: 18.1% ACR — modest increase over baseline.
- AutoGen: 20.0% ACR — a 4.5-point jump.
- SmolAgents: 31.1% ACR — more than double the baseline. The outlier.
The spread between CrewAI and SmolAgents is a 2.6x factor. For a CISO signing off on an agent deployment, that delta is the difference between a tolerable residual risk and a breach. For an enterprise RFP, that delta converts security from a checklist item into a selection criterion with quantitative teeth.
CrewAI's result is the counterintuitive gem. A framework that performs better than no framework at all. The mechanism is architectural: CrewAI's discrete task-isolation model boundaries each agent's context into separate execution scopes. When an injected instruction arrives, its blast radius is constrained to the current task. The framework's structural rigidity acts as a defensive filter. That finding disrupts the industry assumption that every added abstraction layer increases attack surface. Some abstractions serialize context in ways that reduce cross-contamination.
SmolAgents is the mirror image. Its 31.1% ACR is driven by two unique failure signatures: RAG Poisoning appearing at 20% and Context Boundary Violation reaching an extraordinary 64%. The architecture leans toward shared context and lightweight tool routing. Every agent in the system reads from a common context window, which means every agent inherits any poisoned content introduced by any other agent. The framework is designed for maximal flexibility — and it pays for that flexibility in attack surface.
The 64% Context Boundary Violation rate deserves emphasis. It means that in nearly two-thirds of adversarial test scenarios, SmolAgents allowed information or instructions to cross boundaries that should have been isolated. For a framework marketed as lightweight and composable, this is an architectural verdict: composability without isolation is a security liability.
I analyzed a similar failure in 2025 while working on an AI-driven oracle network. The network used LLMs to validate off-chain data, and the consensus mechanism failed deterministically when multiple agents produced identical but incorrect outputs — the result of prompt injection cascading through a shared validation context. My local LLM simulation showed the verification layer could not detect semantic consistency errors because the agents all shared the same poisoned context. SmolAgents' Context Boundary Violation is the same failure mode, measured at scale, in a general-purpose framework.
Eight Failure Modes as a Shared Vocabulary
The study contributes a taxonomy of eight failure modes that should become the industry standard vocabulary:
Tool Call Hijacking — an attacker redirects a legitimate tool invocation to a malicious target. The model believes it is calling a known API; the framework routes the call elsewhere.
Output Poisoning — the model's response itself carries malicious content that contaminates downstream consumers, either humans or other agents.
Cross-Tool Injection — data retrieved from one tool carries instructions that execute when passed to another tool. The classic second-order injection, now at the tool-graph level.
Memory Poisoning — persistent memory stores adversarial content that resurfaces in future sessions. The attack does not need to succeed now; it only needs to survive until the next context window.
RAG Poisoning — retrieval-augmented generation supplies poisoned documents that the model treats as authoritative ground truth.
Delegated Authority Abuse — a sub-agent holds elevated permissions, and an attacker induces the sub-agent to exercise those permissions beyond its intended scope.
Multi-Agent Propagation — an attack spreads across agents through inter-agent communication channels, functioning as an autonomous worm within the orchestration graph.
Context Boundary Violation — information or instructions leak across context isolation boundaries, collapsing the security domains that the framework was supposed to maintain.
This taxonomy matters beyond the immediate study because it gives researchers, auditors, and vendors a shared lexical foundation. Before SADF, a vulnerability in one framework was described in one vendor's terms, and the equivalent vulnerability in another framework was described differently. Security research cannot accumulate without a stable vocabulary. The eight-mode classification is the kind of contribution that outlives the specific numbers in a paper.
That said, the taxonomy functions as a system of classification, not a system of proof. The presence of a failure mode in the taxonomy does not tell you how the modes interact. My read of the data suggests Context Boundary Violation is the root failure that amplifies the others. Once context boundaries collapse, Multi-Agent Propagation becomes trivial, RAG Poisoning can be seeded from any memory store, and Delegated Authority Abuse can be triggered from any agent's context. The framework that controls context boundaries controls the entire attack tree.
— State transition: attack surface reclassified.
The Five-Set Gap: Declared Coverage vs Actual Coverage
Here is where the study starts to show cracks. The paper claims coverage of eight architectures. The detailed ACR data published covers five targets: Direct API plus four frameworks. That leaves three architectures unaccounted for in the quantitative breakdown.
Possibly the remaining three architectures produced incomplete data — framework configuration failures, harness incompatibilities, or time-boxed evaluation windows. Possibly their datasets were excluded because of protocol violations in the test runs. The paper does not say. The gap between “declared coverage” and “actual coverage” is precisely the kind of discrepancy I flag in every audit, because it is the point where a confident claim outruns its evidence base.
Compounding the concern: the repository's SUPERSEDED folder contains an earlier version of the study that claimed coordinated disclosure across ten architectures. A previous iteration of the research was formally retired. This is good scientific hygiene — researchers should preserve and mark old versions rather than deleting them — but it also indicates that the methodology has been through at least one significant revision. The question for consumers of this research is whether the current methodology is the final stabilized form or merely the latest iteration in an ongoing convergence. The community must cite the new version, not the superseded one. The existence of the folder is a reminder that even the corrected numbers are snapshots, not eternal truths.
Then there is the payload problem. 5,119 evaluation rows — an impressive volume — but they map to only 32 distinct attack payloads. A 32-payload corpus is a curated sample, not a representative distribution. The researchers may have selected payloads for clarity and interpretability, which biases the corpus toward straightforward attacks and away from rare-but-high-impact paths. An attack that requires a four-step chain of reasoning across three tools will not survive a 32-payload filter if the filter prioritizes atomic, testable injections. The ACR numbers must be read as rates against a specific attack corpus, not as absolute measures of real-world exploitability.
I have to ask: were the 32 payloads adversarially enhanced? Did they undergo red-team iteration? The paper does not disclose whether the payload set was evolved through repeated testing failures. A static 32-payload corpus cannot adapt to framework-specific defenses. If a framework happens to block the specific phrasing of the payloads, its ACR drops regardless of whether it blocks the underlying attack semantics. This is a classic overfitting pattern. The framework that wins the benchmark may be the framework that best matches the test corpus, not the framework that is generically secure.
The variables the study did not decompose are equally troubling. Default framework configurations were used. That means the temperature settings, system prompt templates, and tool-permission granularity varied across frameworks in ways that were not controlled. A framework with a permissive default system prompt will score worse than a framework with a defensive default system prompt — not because of architecture, but because of prompt template choices. The study treats the framework as a black-box configuration, which is fair for a first pass but leaves a massive interaction surface unexplored.
Finally: the model interaction question. All results are for Claude Sonnet. Would the framework ranking hold with GPT-5.4? DeepSeek? Llama? The model×framework interaction effect is unknown. Some frameworks route tool calls in ways that suppress certain model behaviors; others amplify them. The 2.6x spread between CrewAI and SmolAgents may shrink, expand, or invert under a different model. Until the study is replicated across at least three model families, the ranking should be treated as conditional on Claude, not universal.
Contrarian: The Commercial Attractor Behind the Science
SADF is not a commercial product. It is research. But the commercial logic it unlocks is unmistakable — and that is where my reading diverges from the celebratory coverage.
The study creates a measurable value anchor for “framework-layer security assessment.” The 2.6x ACR gap between frameworks gives security vendors a quantitative justification to sell assessment services. The natural business form is Security-Evaluation-as-a-Service: a repeatable audit product that runs SADF-style evaluation against any enterprise's agent deployment. The numbers are already RFP-ready. An enterprise can now write “must maintain ACR below 15% under refusal-filtered scoring” into a procurement document. That is a concrete, enforceable security requirement. It did not exist before this study.
I am not naive about what happens next. Every benchmark that acquires commercial significance becomes a target for optimization. Framework vendors will tune their defaults to score well on SADF-style evaluations. The tuning may be legitimate — better system prompts, stricter context isolation — or it may be gaming: special-casing the test corpus, detecting the harness, or shifting refusals to hide failures. I have seen this cycle in the DeFi auditing market, where protocols optimized for audit checklists rather than actual security. The checklist becomes the attack surface.
The CVE evidence cited alongside this research — Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) — demonstrates that framework-layer vulnerabilities are real and exploitable. Those are not theoretical constructs. They are assigned CVE numbers. Enterprises pay for vulnerability remediation when the vulnerability is real. That willingness to pay is what makes the Security-Evaluation-as-a-Service model viable.
But there is a structural tension the researchers probably did not intend to create. If security firms build commercial products on SADF's methodology, they become dependent on the ACR numbers staying relevant. The methodology relies on a fixed model (Claude Sonnet) to enable attribution. When the next Claude or GPT generation lands, the baseline shifts. Every commercial assessment built on this methodology will require a full re-baselining. That is expensive. The likely outcome is that vendors lock the methodology to one model version and let the model technology drift underneath it, producing results that are internally consistent but increasingly disconnected from production realities. The same failure mode that plagued the rule-based scoring — measurements detached from the actual system — will re-emerge at the commercial layer.
And then there is the venue question. This study — fundamentally an enterprise security research artifact — was published through a blockchain and Web3-focused information channel. That is a peculiar distribution choice. The audience that needs this research is enterprise CISOs, security engineers, and agent developers. The audience that reads a blockchain news site is largely crypto infrastructure builders and investors. The mismatch suggests either a deliberate effort to reach the crypto-AI developer community (which is genuinely building agent frameworks at scale) or a missed signal about where the commercial value will flow.
My read: it is the former. The crypto-AI convergence has been the proving ground for agent frameworks precisely because the stakes are high — financial transactions, private keys, composable DeFi protocols — and the regulatory cover is thin. DeFi agents are the most attack-prone agents in production today. The SADF methodology, applied to a crypto-agent framework, would produce numbers that terrify anyone holding assets in an autonomous portfolio manager. That is the commercial attractor. The Web3 publication venue is not a mistake. It is a roadmap.
The simulated environment limitation deserves the final contrarian note. Everything ran in SimulatedToolEnvironment — no real API keys, no real databases, no real cross-tenant data flows. This was ethically necessary and methodologically clean. But real-world permission boundaries behave differently under adversarial conditions. A simulated tool has no latency, no rate limits, no heterogeneous error modes. Real tools fail. They return unexpected schemas. They time out. Those operational imperfections are where real attacks hide — an attacker manipulating a tool response's schema change could poison the model's next decision in ways no simulated environment can reproduce. The SADF numbers should be read as lower bounds on real-world vulnerability.
— Proof: accepted, with caveats.
Takeaway: The Triad Shift Is Permanent
SADF's deepest contribution is conceptual, not numerical. It shifts the unit of security analysis from the model to the triad: model + framework + tool environment. The same model, operating through different frameworks, produces a 2.6x spread in attack success. The framework is not the delivery vehicle for the model — it is the security domain in which the model executes. Treating the model as the attack surface was the original error. This research corrects it.
The industry will spend the next eighteen months absorbing the implications. Enterprise agent evaluations will stop being single-model benchmarks and become orchestration-level assessments. Framework vendors will issue security hardening releases with SADF-derived numbers in their marketing materials. Security firms will build assessment products on this methodology and sell them to enterprises that suddenly realize their agent deployments have an unmeasured vulnerability plane. The ACR metric will be standardized, commercialized, and eventually gamed.
My forecast: within two years, “frame-layer ACR” will be a procurement standard, and within thirty months, the first certification theater will emerge — vendors claiming SADF-compliant security scores without disclosing their evaluation configurations. The benchmark will become the attack surface, as benchmarks always do.
The methodologically honest response is not to abandon benchmarking but to embed the audit into the score. Every SADF-style evaluation should publish its payload corpus, its configuration parameters, and its refusal-filtering rules alongside the ACR number. If the evaluation cannot be independently re-run, the number is not a measurement — it is marketing.
The open question, the one this research cannot answer yet: who audits the auditors? The refusal-filtered correction proved that benchmark methodology can conceal order-of-magnitude errors. SADF caught its own. The next generation of framework-security evaluations will not be so lucky. When the first agent-framework breach occurs in production — and it will — the post-mortem will reveal whether the evaluation community learned the measurement lesson or merely polished the numbers.
You do not secure an attack surface by renaming it. You secure it by quantifying the risk accurately. SADF has given the industry a corrected instrument. The question is whether the industry will keep using it honestly once the incentives to fudge the numbers arrive.
— Trust: revoked, pending resync.
