Almost every AI security framework we are applying rests on one misguided assumption: danger scales with size. Compute thresholds, export controls, and tiered evaluation regimes all encode the same intuition, the larger the model, the more we should worry. If this isn’t true, what policy changes are needed?
I recently mapped more than twenty fielded AI systems against two axes: raw offensive capability with safeguards stripped, and residual risk as actually deployed (Fig. 1). They ran from millions to trillions of parameter models, and included munition seekers, gene design models, theatre planning, cyber offense systems, and general-purpose AI models. The picture does not support the above assumption. In fact, the data supports the inverse. Small, specialized models beat bigger general models at offense, but bigger models maybe better at defense.
Figure 1. Security Risk vs Size of Model. Hollow ring: raw offensive capability with safeguards stripped. Filled dot: residual risk as actually deployed. Cyber related positions anchored to CAISI / UK AISI results, July 2026. Data compiled by Alvin W. Graylin.
Each system appears twice: a hollow ring for raw capability with safeguards stripped, a filled dot for residual risk as deployed. The gap between them is the safeguard effect. Read left to right, and the size to threat correlation everyone assumes is simply absent.
Seven assumptions worth rethinking
One: the largest models pose the greatest risk. The high-residual band, where capability and deployed risk are nearly identical. Across six orders of magnitude, no clear trend. In 2022, researchers at Collaborations Pharmaceuticals inverted the scoring function on a commercial drug-discovery model (MegaSyn) of under 100 million parameters and generated more than 40,000 candidate chemical warfare agents (many more lethal than VX) in just six hours on a desktop. Chemprop-class retrosynthesis models, which can find non-controlled precursor routes around scheduled pathways, run at one to ten million parameters. Evo models with single digit billions of parameters can help design novel life forms. News just came out last week that this exact system was able to generate 16 new viruses. All these systems sit well below the axis floor of any parameter-based regime. Compute restrictions do nothing to fix this.
Two: compute thresholds capture the relevant risk. Hackphyr, built on Zephyr-7b-β at 7 billion parameters, performs comparably to GPT-4 on network attack scenarios and runs on a single GPU. Deep Hat V2 ships commercially at 30 billion parameters, is marketed as uncensored for offensive security, and executes inside the customer environment with no external calls. No government evaluation covers it, so that placement rests on vendor claims. Neither would trip a FLOP ceiling.
Three: capability and threat are the same axis. Claude Fable 5 and Claude Mythos 5 share one underlying model. Their capability positions are nearly identical; their deployed risk differs sharply, because Fable's added domain safeguards drop cyber requests back toward Opus 4.8 behavior. The difference comes entirely from the safeguards and distribution controls layered on top. That vertical gap is what governance can act on. Parameter count is not.
Four: open-source models are more dangerous than closed ones. Recent NIST study found that Kimi K3, the strongest PRC open-weight model, only scored 32 percent on ExploitBench against 57 percent for the leading U.S. model, and reached step 17 of a 32-step attack range where U.S. models reached 28.5. On the highest-severity outcome test, arbitrary code execution (ACE), Kimi K3, succeeded on 0 of 41 tasks, while the most capable U.S. models averaged 20. So, we really need to be careful about self-interested parties saying larger open-source models are more dangerous, when it likely has more to do with protecting margins than national security concerns. (see Fig. 2)
UK AISI did recently report that the Kimi K3 model was able to escape its sandbox during testing, but it merely used a misconfiguration in the testing sandbox that left the door open, rather than a sophisticated swarm agent attack like what the OpenAI model did. And when it did get out, all it did was look up the answer for the test it was given, rather than doing any damage to real world systems. In the future, this behavior could change, but it’s important to question the basic assumption that open models are always more dangerous.
Figure 2: CAISI/NIST - Comparison of aggregate capabilities over time of the most capable U.S. and PRC models. A 400-point increase on the y-axis equates to a 10x increase in the odds of solving tasks. Shaded regions denote 95% CIs.
Five: model quality determines attack success. Microsoft's MDASH is a harness, not a model. It orchestrates more than 100 specialized agents across an ensemble and scored 88.4 percent on CyberGym at launch, against 83.1 percent for the Mythos preview model. Adding a compact security model roughly 1/10th the size raised that to 95.95 percent. The orchestration layer beat every individual model. Regulating training while ignoring scaffolding regulates the less important variable.
Six: national security requires the largest models. It requires the opposite. Loitering munition seekers performing automatic target recognition run on Jetson-class edge hardware, capping them in the single-digit millions of parameters. Edge deployment favors small models on latency, power, thermal envelope, and operation without a datalink. Larger models are slower and, in narrow domains, more easily distracted by irrelevant context. They are also harder to validate, and validation is what matters when a false positive is a struck target. Cisco's Foundation-Sec-8B matches or exceeds models ten times its size on security benchmarks while running on one or two GPUs. The famed DoD Maven Smart System is based on a fine-tuned 2-year-old Claude Sonnet 3.5 model. That level of intelligence can now be distilled into a 4B model which could potentially run on a smartphone.
Seven: denying China compute is the primary lever to keep U.S. safe. Due to shared risks between these superpowers, on many safety related issues, cooperating may actually produce the outcomes most beneficial to the U.S. and the world. More on this below.
Five threat domains, five different answers
Attack and embedded systems favor small models that don’t require a comm link. Air-gapped operation, no API telemetry, no rate limits, no refusals mid-chain. The offensive bottleneck is stealth and throughput, not reasoning.
Cyber Orchestration favors large models, but the advantage attaches more to the system rather than the model, as MDASH shows. There’s little discussion today on regulating orchestration systems, but it’s clearly very needed.
Cyber Defense favors large models most clearly, and this is where the current approach fails. Defenders need breadth across every vector; attackers need depth in one. Safeguards that constrain security research are therefore costly in a way that is easy to miss.
When Hugging Face's systems were breached in July by OpenAI models, commercial frontier-model APIs blocked the forensic requests because their safety systems could not distinguish defensive analysis from attack. The team ran the open-weight Chinese model GLM-5.2 on its own infrastructure instead, worked through more than 17,000 logged actions, and contained the intrusion. An American company under active attack by an American closed-model was defended by a Chinese open-source model because the American ones could not tell friend from foe.
That is a Slave AI failure, in the terms I set out in Beyond Rivalry. A model trained toward obedience can only refuse; it cannot reason about whether refusing is right. What we need is Guardian AI: systems capable and contextually aware enough to protect us from malicious actors, from other AI systems, and from our own unintended consequences. That requires scale, because judgment requires breadth. It also requires that we stop locking down every capability rather than stewarding it. High-quality models with fewer restrictions, in defenders' hands, are a global public good. Every hour a defender spends fighting a guardrail is an hour the attacker fights nothing.
Bio/Chem Design favors small models. Molecular graphs, protein sequences, and binding energies come from compact architectures immune to alignment techniques built for natural language. A graph neural network has no refusal layer to remove. The key here is monitoring and controlling access to precursor chemicals and expanding safeguard for synthesis equipment.
Bio Synthesis is the outlier, and there is good news and bad. For biology, model-level access control has already failed. Evo 2 shipped with weights, inference code, training code, and its full dataset. There is no API to revoke. What remains is the synthesis chokepoint: Customer vetting and sequence screening at nucleic acid providers already operate internationally through the Gene Synthesis Consortium, whose members screen orders against databases of sequences of concern before synthesizing. There are still gaps as novel AI-generated combinations are developed, but they can be reduced if vendors and regulators globally work more closely together to keep the systems updated. Closing those gaps buys more security than any parameter threshold. But this requires Washington and Beijing to align, since they are the two largest suppliers of synthesis equipment in the world. Of course, collaboration across all vendors globally is needed to truly secure this threat vector. Again, larger general AI models aren’t the core problem.
Data beats intelligence
On the opening day of the U.S.-Iran war in February, a Tomahawk missle struck the Shajareh Tayyebeh girls' school in Minab, killing at least 168 people, more than 100 of them children under twelve. The school sat within 100 yards of an IRGC naval installation and had been inside that perimeter until a wall went up around 2013. Targeting ran through the Maven Smart System, which generates roughly 1,000 target packages an hour. A preliminary investigation concluded the strike likely followed from outdated intelligence, and former officials said stale human-curated data, not AI, was to blame.
The school had a website. Free satellite imagery showed a schoolyard with a sports field. No model of any size prevents this, because the failure was in data lineage, not reasoning. A larger model querying the same stale record returns the same coordinates faster and with more confidence.
The China mistake
The threat model that matters is not Beijing reaching AGI first. It is a non-state actor with a 30-billion-parameter uncensored model, a good harness, and no return address. Small models proliferate regardless of jurisdiction and leave no attribution trail, and an unattributable intrusion between nuclear powers is an escalation problem before it is a technology problem. In that world, a China unable to defend its own infrastructure is a liability to global stability, not an advantage to Washington.
Beijing is already regulating its own labs more aggressively than any other market. Concordia AI's 2026 survey documents agentic AI security guidance, ethics review requirements, and binding obligations on consumer AI services that are already deployed and enforced. It should be noted that Chinese frontier safety research output grew roughly 60 percent year over year, with agent safety rising from 8 percent of new papers in early 2025 to 27 percent by early 2026. The caveat: only five of ten leading Chinese developers reported safety evaluation results on release. Shared standards here would make a difference.
As Fig. 2 showed, CAISI found the Chinese models less dangerous, but they also found GLM-5.2 answers sensitive biological queries at far higher rates than tested U.S. models, which is where PRC safeguards are weakest. In personally speaking with multiple Chinese labs, it’s clear that their lack of compute resources due to export controls has forced them to deprioritize safety demands vs. capability enhancement. Expanding safety testing compute resources, like what UK AISI has, to more countries could help improve AI safety globally, without fear of its misuse by rival nations.
The race framing is softening at home, too. More than 100 organizations, including Nvidia, Microsoft, Meta, IBM, Palantir, OpenAI and Google, have now signed the July 24 Open Weights and American AI Leadership letter opposing premature restrictions. Days later, Nvidia and roughly 50 partners launched the Open Secure AI Alliance to build open defensive models and agent harnesses, citing the Hugging Face incident as its founding case. Every participant has commercial exposure to a ban, so weigh the motives. But 8 of the top 10 models on OpenRouter in July are already open-source, and the industry has now reorganized around the proposition that open weights are defensive infrastructure.
Four Asks for September
Four asks follow, and Xi Jinping's state visit to Washington on September 24, the first in over a decade, is where they could land. Trump has said AI will be on the agenda.
Shared harm standards, not shared capability standards. Agreement on what constitutes an unacceptable capability, evaluated the same way in both countries, so that "safe" means the same thing in Shanghai and San Francisco. That’s clearly missing today and doesn’t require mutual trust.
A shared safety evaluation cluster. Chinese labs are compute-constrained, so safety research competes with capability research for scarce chips. Compute earmarked for evaluation and red-teaming is cheap relative to the benefit, and the benefit is global. An international testing facility open to any vendor institutionalizes it. Require publishing safety scores alongside capability benchmarks so safety investment earns a competitive return. Then, both Chinese and US labs would have no excuse not to test their systems.
An incident notification channel. The Nuclear Risk Reduction Centers, staffed continuously since 1987, exist because a misread signal costs more than talking. An equivalent for AI incidents where attribution is contested is cheap insurance. With the rising risk of bad actor attacks and false flag operations from non-state actors, this safeguard will be increasingly needed.
Capability non-development agreements. A capability never trained cannot leak. This matters more than denying Beijing another turn of the scaling crank. Beijing also wants to limit rogue actor misuse, thus agreeing on redlines in advance makes sense for both sides (no nuclear weapon command/control, no AI uplift to bio weapon design, no AI-attack on civilian infrastructure, no autonomous self-replication outside control environments [RSI]). General commercial models have no need for bio and chemical threat design, so keeping defense use case training only in military labs on both sides seems quite reasonable.
Another Asilomar moment
At Asilomar in 1975, molecular biologists imposed a voluntary moratorium on a class of recombinant DNA experiments, then built the containment framework that governed the field for decades. They acted before the capability matured enough to do real harm.
That view is starting to catch on in the AI labs now. On July 28, 1,200+ employees of frontier labs published Pacing the Frontier, asking Washington to support an international effort to build tools for deliberately slowing automated AI development. Signatories include top technical leaders at Anthropic, OpenAI, Meta and Google. The concern is recursive self-improvement (RSI) of AI that goes out of control. The logical extension is an explicit agreement not to implement it in frontier labs even once it becomes possible.
But this cannot stop at two capitals. If dangerous systems are small and cheap, a country with a modest research budget and a few hundred GPUs can build a competent offensive cyber agent or an inverted molecular designer. Within a few years, dozens will. Any regime binding only Washington and Beijing binds the two parties least likely to defect and leaves the growing middle untouched. A U.S.-China agreement is the necessary first move, not the finished structure, and it has to open immediately to third parties. That is how Asilomar's containment norms and the Montreal Protocol scaled.
Bigger AI is not more dangerous. Better orchestrated is more dangerous, less monitored is more dangerous, and irreversibly released is more dangerous. All three require cooperation with Beijing: orchestration needs shared harm standards, monitoring needs shared evaluation infrastructure, and irreversible release needs joint agreement on what never gets built. September 24 is a good place to start.
Read more national security insights from experienced experts exclusively in The Cipher Brief.
