Microsoft’s Threat Intelligence team confirmed this month that a phishing campaign detected on 28 August used AI-generated obfuscation code that was alien to human authorship, too verbose, too internally complex, and not organised the way a human engineer would organise it, so it wasn’t detected.
Separately, this month researchers documented a ransomware system that scans its target environment and generates a bespoke encryption payload for each victim, meaning the file hash changes every time and signature-based detection has nothing to match against. IEEE projections place LLM-generated malware at 50% of detected threats by 2025, against 2% in 2021.
The defenders’ logical response is that if an attack operates below human speed, detection must operate at the same speed. Security vendors are now positioning AI agents, systems that pursue a goal using available tools rather than following a pre-written flowchart, as the necessary counter to machine-native threats. But matching machine speed with machine speed is only half the answer. The harder question is what happens when the defender’s algorithm makes a mistake, acts on a false inference, or cannot be interrogated after the fact.
The claim is that probabilistic, adaptive defence can keep pace with probabilistic, adaptive offence in ways that legacy SOAR tooling cannot.
This is not totally wrong. But it is only half the answer. The other part is what happens when the defender’s algorithm makes a mistake, acts on a false inference, or cannot be interrogated after the fact.
That is where the governing question sits, and it is a question most organisations currently evaluating agentic security tools have not asked.
How agents differ from the tools that came before
SOAR (Security Orchestration, Automation, and Response) basically follows a conditional flowchart: if this indicator, then execute this playbook.
The logic is deterministic: the same input produces the same output, every time. An analyst can read the playbook, trace the decision path, and explain precisely why an action was taken. Auditing is straightforward because the flowchart decisions are fixed.
An AI agent is different in a specific and fundamental way. Instead of if-this-then-that, it is given a goal, such as “hunt for strange movement in the finance network“, plus a set of tools: query logs, isolate a host, reset a credential.
The agent then determines its own action sequence based on what it finds. Two agents with identical goals and tools may very well take different paths given the same information, because some context has changed.
The output is probabilistic, not deterministic.
It’s these properties that make agents great against some threats like polymorphic malware. It is also what makes their governance non-trivial. The EU AI Act Article 14 requires that natural persons can effectively oversee high-risk AI systems and intervene when necessary.
Whether an autonomous security agent that can isolate production infrastructure qualifies as high-risk is, in most organisations, an open question.
Where the control boundary needs to now sit
The core issue is not whether to deploy AI agents. Given the explosion of threats, the costs of not deploying them will soon exceed the costs of deploying them badly. The issue is where the control boundary sits, and what is happening on both sides of it.
Consider these categories of agent actions:
Read-only actions — querying logs, correlating alerts, generating threat reports — carry limited execution risk. If the agent infers incorrectly, the cost is a false positive that a human analyst must review and dismiss. The failure mode is noise, not damage.
Soft-execution actions — flagging a host for review, notifying a team, triggering an automated alert to a SOC analyst — carry slightly more consequence but still leave human judgment in the loop before anything changes in the environment.
Hard-execution actions — isolating a host, revoking credentials, modifying firewall rules, quarantining a subnet , change the operational state of the environment immediately and without human confirmation. If the agent acts on a false inference at this level, the cost is a self-inflicted outage, a disrupted production system, or , in the worst case , a planned action by an attacker.
The LevelBlue analysis of Zero Trust for AI systems identifies granular access controls and continuous behavioural monitoring as the critical mitigations for agentic risk. Both require that the control boundary is defined before the agent is deployed , not discovered through the first operational failure.
A useful way to look at it: an agent should not have access to actions that, if mistakenly executed, could be reversed in a time period that you’re comfortable with.
Domain controller isolation likely fails that test. Quarantining a guest Wi-Fi segment does not.
The deployment sequence should follow that logic.
Trust me, I’m your agent, design shifts that matter
The Microsoft integration of Anthropic’s Claude into M365 Copilot is interesting here, because it illustrates how platform architecture can obscure the actual trust boundary.
Users choosing Claude Sonnet 4 for a task inside a familiar Copilot interface may not know that data processed through Claude operates outside Microsoft’s standard compliance framework.
The interface is unified. The governance is not. The same dynamic applies to agentic security tools: a single-pane console may surface autonomous actions without making visible which decision was made by the agent, on what basis, and with what confidence score.
If the organisation cannot distinguish between a human-authorised action and an agent-executed one inside the audit log, then accountability starts to be lost.
This creates a concrete board-level consequence. CIOs and CISOs approving agentic security tools are not merely approving a capability enhancement. Instead, they are approving a change to the organisation’s decision-making architecture , one where a probabilistic system holds execution rights over infrastructure.
Who is accountable when the agent is wrong?
The Wiz guidance on AI Bills of Materials gives some pointers to a tactical answer: organisations maintaining an AI-BOM that tracks models, permission boundaries, and third-party service dependencies have the artefact required for post-incident review. Organisations without one are operating on implicit trust in the vendor’s audit trail.
Three design changes would substantially reduce these risks. Importantly, none require slowing adoption. All require treating agent governance as a design consideration, not an afterthought.
Register agents as principals, not tools. An AI agent with execution rights in a production environment is a principal , a system that can act on behalf of the organisation with real consequences. It should have a registered identity, a defined permission scope, a review cycle, and credential rotation. Treating an agent as an infrastructure tool rather than a principal is how unauthorised scope expansion goes undetected. The viral Claudius vending machine incident , where an agent without adequate self-model bounds asserted a human identity to building security , illustrates what happens when an agent’s operating boundaries are not well defined.
Define the read-only period as an evidence definition, not a delay. Deploying agents in suggestion-only mode before enabling execution is the minimum evidence-collection phase required to set a behavioural baseline.
Without that baseline, anomaly detection for agent behaviour is not operational , there is nothing to compare against. Applying sixty to ninety days of read-only operation, with false positive rates tracked and reviewed by a defined human owner, can give you the data required to calibrate the agent rather than relying on an out-of-the-box default.
Separate the kill switch from the console. An override mechanism that revokes all agent permissions must be separate from the platform the agent operates within. A kill switch that requires access to the same console in which the agent is running may not be accessible when it is needed. If you are servicing the EU, the EU AI Act Article 14 requirement for effective human oversight has a practical meaning , not nominal override capability, but a mechanism that works when the system being overridden is behaving adversarially.
Takeaways
The real question is not whether organisations will deploy AI agents. It is whether they will delegate authority without being able to explain or revoke it. Agentic security may be necessary as threats accelerate, but speed does not make an autonomous decision trustworthy. AI security cannot remain a perimeter function; it has to govern the full organisational stack, from people and process to technology. The security team must know which agents are deployed, the CIO must know which vendors control their decisions, and the board must know what permissions autonomous systems hold against its risk appetite. Three questions should decide whether trust is justified: what does this agent do, what can it touch, and what happens when it is wrong? If the organisation cannot answer them, it is not extending trust deliberately. It is accepting invisible authority.
