Traditional security frameworks were designed for systems that do not learn. AI systems do. That distinction is not a footnote — it is where control breaks down.
—
The assumption buried in every control framework
OFFICIALLY IT IS a security framework. But NIST SP 800-53, ISO 27001, and the CIS Controls are, at their core, a set of instructions for governing systems that hold still long enough to be described. Network edges can be mapped. Code can be reviewed. Dependencies can be scanned. The model of threat is essentially mechanical: a static artefact with known components, exposed to known attack classes.
AI systems violate that assumption at the architecture level.
A model does not merely execute instructions written by its developers. It learns from data — data that arrives continuously, from external sources, through pipelines that traditional monitoring was not built to observe. When researchers documented 23.77 million secrets leaked through AI systems in 2024, the 25 per cent year-on-year increase was not a story about weak passwords or unpatched servers. It was a story about attack surfaces the compliance frameworks organisations rely on simply do not describe. Organisations can satisfy every requirement in a formal audit and still carry AI-specific risk they cannot see, measure, or report to a board. The frameworks are not wrong. They are incomplete in a way that compounds quietly, then suddenly.
—
The build pipeline as attack surface
The Ultralytics AI library compromise is the most instructive case study available. Attackers did not find a vulnerability in the codebase. They did not exploit a misconfigured server or a weak credential. They compromised the build environment — the infrastructure that compiles and publishes code — injecting malicious content after code review had completed but before the package was published to public repositories.
Organisations with comprehensive dependency scanning installed the poisoned package. Organisations running software bill of materials analysis installed it. Organisations with mature vulnerability management installed it. Every control that should have caught a supply chain compromise was in place. None caught this, because none were designed to monitor the build pipeline for post-review injection. The attack exploited a gap in the control model's imagination, not a gap in its implementation. This is a gamble that every organisation using external AI packages is currently taking, whether or not it knows so.
The Hacker News documented four distinct attack classes now operating against AI systems: prompt injection, which manipulates how models interpret input; model extraction, which reverse-engineers proprietary systems through crafted queries; LLMjacking, where stolen API credentials enable attackers to use frontier models at the legitimate account holder's expense; and data exfiltration through agent tooling, where natural language instructions cause AI systems to perform unauthorised actions. What these share is a common property — they exploit the behaviour of AI systems rather than vulnerabilities in their code. Traditional controls address code. They have almost no visibility into behaviour.
The supply chain risk controls in NIST SP 800-53 focus on vendor assessments, contract requirements, and software bill of materials. Sensible requirements for the systems those controls were designed to govern. They provide no guidance on whether a pre-trained model's weights have been backdoored, whether a fine-tuning dataset has been poisoned, or whether a retrieval-augmented generation pipeline is ingesting corrupted external data. Those questions did not exist when the standards were written.
—
What the attack surface actually looks like now
The traditional perimeter was the network edge. For an AI system operating in a modern enterprise, the surface extends considerably further.
It includes training data — the corpus from which the model learned its initial behaviour. It includes fine-tuning datasets used to adapt a general model for enterprise use. It includes retrieval-augmented generation pipelines that pull external content into model context at inference time, potentially ingesting corrupted documents from sources the organisation does not control. It includes model weights — the numerical parameters encoding learned behaviour, which can be backdoored to activate under specific conditions. It includes the tooling that agents use: APIs, databases, external services that an autonomous agent can query and act upon based on natural language instructions. And it includes inference infrastructure, where prompt injection attacks can manipulate what the model treats as authoritative.
OpenAI's published analysis of prompt injection defences illustrates how hard this surface is to govern. The company has implemented instruction hierarchy training, automated red-teaming, and extensive monitoring. Its assessment of those defences is candid: prompt injection remains a "frontier security challenge" the industry has not solved. OpenAI notes it has not yet seen significant real-world adoption of prompt injection attacks by malicious actors. That is a temporal advantage, not a structural one. The attack pattern works. The defences are incomplete. When systematic exploitation begins, it will not announce itself in advance.
NIST's $20 million collaboration with MITRE to establish research centres studying AI threats against critical infrastructure is, among other things, an institutional acknowledgement of inadequacy. Research funding follows recognised problems. The problem is already operational.
—
The agentic dimension
Standard security tooling — vulnerability scanners, SIEM platforms, endpoint detection — was built to observe code executing on known infrastructure. It was not built to detect model behaviour drift, identify poisoned training data, or flag anomalous agent actions occurring through natural language interfaces. Deploying that tooling against AI systems produces a monitoring posture that appears complete and is structurally blind to the most significant risk classes.
Agentic systems make this harder still. OpenAI's security design for ChatGPT Atlas includes a "Watch Mode" that alerts users before sensitive operations and requires human confirmation for high-stakes actions. Reasonable design for a consumer product. It does not scale to an enterprise with hundreds of agents operating simultaneously across departments, processing transactions, drafting communications, and accessing sensitive systems. At that scale, pre-approval human oversight becomes operationally infeasible. The choice narrows to constraining agent autonomy — limiting the productivity gains that justified the investment — or accepting that some actions will execute without pre-approval and building monitoring robust enough to catch problems after the fact.
Neither maps onto existing governance frameworks without explicit design. Both require a decision from leadership rather than quiet deferral to inherited process. The organisations currently treating agent governance as a future problem are already running with unmanaged risk. Agents are already operating.
The regulatory picture adds pressure without adding clarity. The White House executive order of December 11 establishes federal preemption over state AI laws and creates a task force to challenge requirements including mandatory safety audits and training data transparency. California's SB 53, which takes effect January 1, requires frontier AI developers to publish catastrophic risk assessment frameworks covering cyber offence, biological threats, and loss of control scenarios. Both rules are live. They point in opposite directions. For strategic ambiguity over AI compliance to have deterred bad investment decisions, it needed teeth. Feckless ambiguity — two contradictory federal and state regimes with no resolution timeline — does not hold the same promise.
—
The design shift that matters
The required shift is architectural, not primarily technological. Traditional security thinking starts at the perimeter and works inward. AI security thinking needs to start at the data and work outward: what enters the training pipeline, what enters fine-tuning, what the retrieval system can ingest, what instructions the agent can receive and from whom. The control boundary is not a network edge. It is a trust boundary running through every point where external content or instruction can influence model behaviour.
Concretely, this means monitoring for model behaviour rather than only code behaviour — baselines against which drift can be detected. It means identity and access management designed for non-human agents, with authentication and audit logging that operates at machine speed rather than adapted from human workflows. It means supply chain evaluation that goes beyond SBOM analysis to include provenance assessment for pre-trained models and fine-tuning datasets. And it means explicit governance decisions about where human oversight remains mandatory and where it will be replaced by automated monitoring — decisions currently being made implicitly, by default, as deployment outpaces governance. Who makes those decisions, who funds the controls, and who absorbs the cost when they fail are questions that need answers before the next build pipeline is compromised, not after.
The failure mode if these shifts are deferred is not hypothetical. The Ultralytics compromise, the 23.77 million leaked secrets, and OpenAI's candid acknowledgement that prompt injection defences are incomplete are not early warnings of a coming problem. They are evidence the problem is already operational, in systems that passed their compliance audits.
—
The durable implication
Security frameworks lag technology. That is not a critique of standards bodies; it is how they work. The problem is not that NIST has not yet updated its guidance for AI-specific threats. The problem is that the lag creates a category of risk organisations can carry indefinitely while believing their controls are adequate — because the audit says so.
The organisations best positioned as AI systems become more deeply embedded in operating infrastructure are not those waiting for frameworks to catch up. They are those that identify where current controls describe the system they used to run rather than the system they are running, and redesign accordingly. That is not a compliance exercise. It is a governance decision about what risks are acceptable to carry, and for how long.
The trust boundary has moved inside the model. The controls need to follow it there. ■
