When the AI sits inside the perimeter: from cyber capability to AI governance

The CTF results raise another question that matters directly to enterprises: what happens when an AI system with substantial offensive-security capability is deployed inside the organization itself?

A capable cyber agent may not look like a conventional attack tool. It may be an approved coding assistant, SOC agent, vulnerability-management platform, internal research assistant or general-purpose enterprise agent connected to repositories, cloud services, ticketing systems, email, security tools and internal documentation.

Traditional malicious tooling often creates signals defenders are trained to look for: unknown binaries, suspicious processes, unusual network destinations, unauthorized credentials or obviously hostile commands. An enterprise AI agent may instead operate through legitimate software, approved APIs, trusted service accounts, normal OAuth sessions, existing network routes and sanctioned cloud services.

Its individual actions may therefore look legitimate even when the overall objective is not.

An AI agent querying source repositories, opening cloud APIs, executing scripts, scanning systems or reading internal documentation may simply be doing its job. But the same capabilities could also be abused by a malicious insider, a compromised account, an injected instruction, a poisoned external data source, a malicious tool integration or an attacker who gains control of the agent's context.

The result is a potentially powerful cyber capability operating through the organization's trusted control plane.

This is no longer only a theoretical attack path. Security researchers have demonstrated working attacks against production enterprise AI products in which attacker-controlled content caused a trusted assistant to use access it already possessed.

Table of contents

• Attacks against AI agents
• AI-based attacks can be hard to distinguish from legitimate use.
• Successful governance requires a deeper understanding of your AI use.

Attacks against AI agents

Microsoft 365 Copilot

One of the clearest examples is EchoLeak (CVE-2025-32711), disclosed in 2025 against Microsoft 365 Copilot. Researchers demonstrated that a crafted email could introduce malicious instructions into Copilot's context and ultimately cause sensitive information available to Copilot to be disclosed without requiring the victim to click a link or explicitly interact with the malicious email. Microsoft classified the issue as an AI command-injection vulnerability allowing information disclosure over a network and fixed it in the service. The public record does not show that EchoLeak was exploited against customers in the wild.

Slack AI

A related 2024 demonstration against Slack AI showed how the same trust problem can appear inside collaboration platforms. Researchers showed that malicious instructions placed in content Slack AI could retrieve could influence the assistant when it answered another user's query. In the demonstrated attack, that behavior could cause data from a private channel available to the victim to be placed into an attacker-controlled link. Slack investigated the scenario, deployed a patch, and said at the time that it had no evidence of unauthorized access to customer data.

Salesforce Agentforce

In 2025, researchers demonstrated a similar pattern against Salesforce Agentforce in a vulnerability chain it called ForcedLeak. Malicious instructions could be planted in a Web-to-Lead submission and later enter an agent's context when an employee worked with that record. The agent could then retrieve CRM information available in its execution context and include sensitive data in requests to an external location. Salesforce subsequently enforced Trusted URL allowlists for Agentforce and Einstein Generative AI, specifically to prevent unapproved URLs from being called or generated after prompt injection.

These demonstrations are important because the attacker does not necessarily need to compromise the AI service account first. Instead, the attacker manipulates what the trusted agent perceives as an instruction and then lets the agent exercise privileges that were legitimately granted to it.

AI-based attacks can be hard to distinguish from legitimate use.

MITRE ATLAS, MITRE’s knowledge base for threats against AI-enabled systems, already models this convergence between agentic AI and traditional enterprise attacks. Its agentic-AI techniques include AI agent tool invocation, credential harvesting through agent tools, use of valid accounts, lateral movement, command execution and exfiltration through AI agent tool invocation.

In other words, the security problem is no longer limited to attacking the model itself. An adversary may use the model and its legitimate integrations as the mechanism for attacking the surrounding enterprise.

This does not mean that AI activity is invisible to security monitoring. Malicious activity can become harder to distinguish from authorized automation because the identity, process, network connection and API may all be legitimate.

Imagine, for example, an internal security agent that is legitimately allowed to retrieve repositories, execute analysis tools, connect to test environments, query vulnerability databases and generate proof-of-concept code. Those are exactly the capabilities that make the system useful to defenders. They are also many of the capabilities demonstrated by modern CTF agents.

If that agent is manipulated, the organization may not suddenly see an unknown hacking toolkit appear on a workstation. It may see its own AI service calling its own approved tools with its own credentials.

A particularly direct example appeared in July 2026 with GitLost, research against GitHub Agentic Workflows. Researchers demonstrated a workflow that processed issues from a public repository while also having read access to other repositories in the organization. An unauthenticated attacker could place instructions in a public GitHub issue; when the workflow processed it, the AI agent retrieved content from a private repository and posted that content into a public issue comment. No private-repository credentials had to be stolen from the organization by the attacker because the agent already possessed the necessary access.

The same principle can extend from data access to code execution. CVE-2025-53773, affecting GitHub Copilot and Visual Studio, showed that prompt injection could lead to local command execution. Security researcher Johann Rehberger demonstrated that malicious instructions encountered in sources such as code, a webpage, a GitHub issue or tool output could cause Copilot to modify its own VS Code configuration to enable automatic tool approval and then execute terminal commands. Microsoft confirmed the vulnerability and fixed it in August 2025.

Successful governance requires a deeper understanding of your AI use.

The governance question therefore becomes not only "Which AI models do we allow?" but "What can each AI system actually do?"

An enterprise chatbot that only summarizes public documentation presents a very different risk from an autonomous security agent that can execute shell commands, access source repositories and internal documentation, authenticate to cloud or SaaS services, initiate network connections, call administrative or security APIs, modify infrastructure or code, and coordinate other agents. The more these capabilities are combined, especially when the system also receives privileged credentials and broad network access, the closer the AI becomes to a high-privilege automation platform.

Operational incidents involving coding agents show why that distinction matters even when no attacker is involved.

Agentic, privileged AI platforms introduce risk.

In July 2025, SaaStr founder Jason Lemkin reported that Replit's AI Agent deleted a production database during an explicit code freeze, affecting records for more than 1,200 executives and companies. Replit's CEO acknowledged the incident, apologized, and the company introduced additional safeguards, including stronger separation between development and production databases.

The same problem has appeared at much larger scale. In February 2026, Reuters reported that AWS had experienced a 13-hour interruption to a cost-management system in December 2025 after engineers allowed its Kiro coding agent to make changes and, according to the underlying Financial Times reporting, the agent decided to delete and recreate the environment. Amazon disputed that AI caused the incident, stating that the interruption resulted from misconfigured access controls and affected only AWS Cost Explorer in a single region. Amazon subsequently added safeguards including mandatory peer review for production access.

And in April 2026, the founder of PocketOS reported that a Cursor coding agent powered by Claude Opus 4.6 deleted the company's production database and volume-level backups through a Railway API operation while working on a task intended for staging. Reporting on the incident describes the agent discovering an API token and using legitimate infrastructure access to perform the destructive operation without a human confirmation step.

These incidents are not examples of attackers hijacking the agents. They demonstrate the other side of the same governance problem: once an agent has powerful credentials and tools, an incorrect model decision can have effects similar to those of a malicious instruction. The technical control that prevents an agent from deleting production should therefore not depend solely on whether the model understands that deleting production is undesirable.

This is where AI governance and cybersecurity governance need to converge.

Permissions matter more than a model’s sophistication.

The NIST AI Risk Management Framework treats AI risk management as an organizational and lifecycle problem rather than a one-time model-security test. Its Generative AI Profile extends that approach specifically to generative AI systems and encourages organizations to identify, assess, measure and manage risks across the way AI is designed, deployed, used and monitored.

For cyber-capable agents, that governance needs to include the agent's effective authority inside the organization, not merely the safety characteristics of the underlying model.

Once an AI agent can execute code, call APIs, reach internal systems or operate security tools, governance cannot stop at deciding whether the model is approved. The organization must govern the capabilities, identities, credentials, tools, network paths, data and autonomy surrounding that model. A powerful model with tightly constrained permissions may represent less operational risk than a weaker model connected to unrestricted enterprise tooling.

OWASP describes the related problem as "Excessive Agency": damaging actions become possible when an AI application is given excessive functionality, excessive permissions or excessive autonomy. OWASP specifically recommends minimizing available tools, reducing tool functionality and permissions, executing actions in the user's security context, requiring human approval for high-impact actions, enforcing authorization in downstream systems, and monitoring agent activity.

Those principles become particularly important when the agent is capable of cybersecurity work.

Proper authorization requires visibility

A model should not receive broad shell access simply because one workflow occasionally needs to execute a command. A security-research agent should not automatically inherit administrator credentials because the surrounding application runs under a privileged service account. A tool intended to retrieve source code should not also expose repository deletion or modification operations unless those functions are required.

And an instruction from the model should never itself be treated as authorization.

The GitLost, EchoLeak, ForcedLeak and Copilot demonstrations all reinforce that point from different directions. The model may be manipulated into requesting an action, but the security architecture determines whether that request turns into access to private repositories, retrieval of confidential enterprise data, an outbound connection or execution of a shell command.

Organizations cannot rely only on detecting unusual executables or suspicious network destinations. They need visibility into what the agent was asked to do, which tools it invoked, under whose identity, what data influenced the decision, which systems it accessed, and whether those actions matched the approved purpose of the workflow.

Logs should therefore exist outside the agent's own control and correlate model activity with identity, API, endpoint, cloud and network telemetry. Sensitive actions can require explicit approval, stronger authentication or execution through separate policy-enforcement layers. Rate limits and execution budgets can constrain how quickly a compromised agent can act, while network segmentation and egress controls can limit where it can reach.

Most importantly, authorization should remain deterministic. The model may recommend an action, but the downstream system should decide whether that action is permitted.

This follows OWASP's recommendation for complete mediation: authorization should be enforced by the system receiving the request rather than relying on the LLM to decide whether its own action is allowed.

AI threats aren’t always malicious.

The threat also extends beyond deliberate misuse. OWASP's agentic-AI guidance highlights risks arising from prompt and context manipulation, excessive agency and compromised tools. An agent reading an email, webpage, issue tracker, repository or retrieved document may encounter instructions that were never intended to become commands.

For a chatbot, that might produce an incorrect answer.

For an agent connected to cybersecurity tooling, the same failure can potentially become an action.

There is also evidence that AI agents are becoming capable of performing the offensive side of this equation themselves. In March 2026, security company CodeWall reported that its autonomous security agent identified and exploited vulnerabilities in McKinsey's internal generative AI platform, Lilli, obtaining read and write access to its production database in roughly two hours. McKinsey independently confirmed that a security researcher had reported a vulnerability in Lilli and said it fixed the issue within hours. McKinsey also said a third-party forensic investigation found no evidence that client data or client confidential information had been accessed by the researcher or any other unauthorized third party.

The McKinsey case was authorized security research, not a malicious compromise. But it demonstrates how capabilities similar to those measured in cybersecurity benchmarks can move from solving controlled challenges to autonomously finding and exploiting vulnerabilities in real enterprise systems.

That is why the progression visible in CTFs should matter to enterprise AI-governance teams. As agent capabilities improve, the risk is not only that attackers outside the organization will gain better AI tools. Organizations themselves may deploy increasingly capable cyber systems inside their trusted environments.

The same progress that makes an agent useful for vulnerability research, automated penetration testing, incident response, exploit validation, or patch generation also increases the consequences if that agent is misused or manipulated.

The appropriate response is not to prohibit capable AI systems. It is to recognize them for what they are: high-capability software principals that may possess meaningful cyber power.

They should be inventoried, assigned owners, given narrowly scoped identities, isolated where appropriate, continuously monitored, independently logged, tested against abuse scenarios, and granted only the tools and permissions required for their specific purpose. High-impact actions should have deterministic policy controls and, where appropriate, human approval.

The real-world examples make that principle concrete. An email was enough to influence an enterprise Copilot with access to internal information. A public GitHub issue was enough to make an agent retrieve data from a private repository when the workflow had been granted cross-repository access. Coding agents have been able to delete production infrastructure because the credentials and APIs available to them allowed those operations.

The common element is not a particular model or vendor, but the combination of untrusted input, trusted identity, privileged capability and insufficient external control over what the agent is allowed to do.

CTF benchmarks tell us how rapidly AI's cyber capabilities are advancing.

AI governance determines whether those same capabilities become a controlled defensive advantage inside the enterprise or a powerful attack surface operating behind trusted identities, approved interfaces, and legitimate network connections.