From mayhem to multi-agent hackers: how AI is changing Capture the Flag and what this means for cybersecurity

Author note: The author is an active CTF player and mobile security researcher. The views on future competition formats reflect his personal experience and opinion.

In August 2016 (yes that’s right, 10 years ago), seven racks of computers faced one another in a Las Vegas ballroom. No human operators were allowed to guide their actions during the competition.

Each system had to:

• Inspect previously unseen challenge binaries
• Identify and generate proofs of vulnerability
• Attack opposing services
• Defend its own software
• Deploy repairs without breaking the services it was supposed to protect

The event was the DARPA Cyber Grand Challenge, described by the US’s Defense Advanced Research Projects Agency (DARPA) as the world's first all-machine cyber hacking tournament. Seven Cyber Reasoning Systems competed in an isolated network environment against custom software that none of the systems had previously analyzed. Mayhem, developed by ForAllSecure, won the $2 million first prize.

At the time, the competition appeared to be a highly specialized demonstration of automated program analysis. A decade later, its central idea moved much closer to mainstream cybersecurity.

Modern AI agents can do the following:

• Read natural-language challenge descriptions and operate Linux security tools
• Inspect source code and binaries while writing and testing exploit scripts
• Interact with remote services and submit flags directly to competition platforms

In selected live events, these systems have performed at a level comparable to strong human teams. In more realistic evaluations, however, their performance remains much less consistent.

As an active and enthusiastic CTF player, I find this development both technically fascinating and personally complicated.

CTFs are not only about obtaining a flag. Much of their appeal comes from the process: the moment when an unusual clue finally makes sense, the frustration of spending hours on the wrong hypothesis, the exchange of partial findings between teammates, and the satisfaction of turning incomplete evidence into a working solution.

AI can accelerate that process, but it can also remove some of its mystery. When an autonomous system can enumerate a challenge, launch several approaches in parallel, generate solve scripts, and submit a valid flag within minutes, it changes the experience for everyone else on the scoreboard.

Capture-the-Flag competitions are therefore becoming more than training games. They are increasingly being used as controlled laboratories for measuring how autonomous cyber systems plan, execute, verify, and adapt. At the same time, the CTF community may need to decide which parts of the traditional experience it wants to preserve.

Before LLMs: DARPA's Cyber Grand Challenge

The Cyber Grand Challenge did not use conversational large language models. Its systems were constructed from tightly integrated security technologies, including:

• Static and dynamic binary analysis and rewriting
• Symbolic and concolic execution
• Fuzzing and crash triage
• Constraint solving and exploit generation
• Automated patching and patch validation

The competitors operated inside the DARPA Experimental Cybersecurity Research Evaluation Environment (DECREE). DECREE was a purpose-built operating system used in an isolated competition testbed. It reduced variability compared with a general-purpose operating system while still supporting networked services and machine-to-machine attacks.

The MIT Lincoln Laboratory Cyber Grand Challenge dataset describes the event as a head-to-head competition between autonomous systems running on DECREE and reasoning about flaws in previously unseen software.

The scoring system was especially important. A competitor could not win simply by disabling every suspicious function. A repair that blocked an exploit but broke the protected service could damage the team's score. The CRSs therefore had to balance security, availability, and functional correctness. DARPA reports that the systems were scored on protecting hosts, scanning for vulnerabilities, and preserving correct software operation during nearly ten hours of competition.

The competition established several concepts that remain central to modern autonomous cybersecurity. Vulnerability discovery could be automated. Exploitability could be verified rather than inferred from a suspicious code pattern. Repairs could be generated and tested without waiting for a human developer, and offensive and defensive actions could take place within the same machine-speed feedback loop.

What has developed since 2016?

The interface between the machine and the security problem has changed.

The Cyber Grand Challenge systems had deep capabilities within a narrow, carefully defined environment. Modern agentic systems combine broader language-model reasoning with traditional security tools.

They can:

• Interpret documentation and understand challenge descriptions
• Generate scripts in several languages and select between alternative tools
• Summarize results and revise an attack plan in response to observed output

The strongest current systems do not replace fuzzers, symbolic execution engines, debuggers, disassemblers or static analyzers. They orchestrate them.

The bridge to modern AI: DARPA AIxCC

The clearest historical successor to the Cyber Grand Challenge is DARPA's AI Cyber Challenge (AIxCC).

While the original competition used purpose-built binaries, AIxCC focuses on large real-world open-source projects. During the 2025 final, seven Cyber Reasoning Systems analyzed more than 54 million lines of C and Java code.

According to DARPA's AIxCC final results, the systems discovered 54 of the competition's 63 synthetic vulnerabilities, representing 86% of the complete set. They generated successful patches for 43, 68% of all synthetic vulnerabilities in the final.

The competitors also found 18 real, non-synthetic vulnerabilities that had not been intentionally inserted into the challenge codebases. They provided 11 patches for those real vulnerabilities. DARPA reported that patches were submitted in an average of approximately 45 minutes, with an average computational cost of about $152 per competition task.

Team Atlanta won the $4 million first prize. The team included experts from Georgia Tech, Samsung Research, the Korea Advanced Institute of Science and Technology, and the Pohang University of Science and Technology. Trail of Bits placed second, and Theori placed third.

The improvement between the 2024 semifinal and the 2025 final was also significant. DARPA reported that synthetic-vulnerability discovery increased from 37% to 86%, while the proportion patched increased from 25% to 68%.

The differences between Cyber Grand Challenge, 2016 and the AI Cyber Challenge, 2025

• Primary targets: purpose-built DECREE binaries vs. real open-source C and Java projects
• Scale: constrained competition services vs. more than 54 million lines of code
• Core methods: binary analysis, fuzzing, symbolic execution and patching vs. LLM orchestration, fuzzing, program analysis, proof generation and patching
• Competition model: attack, defend and maintain services vs. discover, prove, report and patch vulnerabilities
• Real vulnerabilities: custom, previously unseen challenge software vs. 18 real, non-synthetic vulnerabilities discovered
• Winning system: Mayhem from ForAllSecure vs. Atlantis from Team Atlanta

Atlantis was the name of Team Atlanta's Cyber Reasoning System submitted to the final competition. The system and the other finalist projects were subsequently published through the AIxCC Competition Archive.

AIxCC is important because it demonstrates that autonomous cyber capability is no longer limited to recovering flags from small challenge programs. The systems worked across large codebases, created structured vulnerability reports, generated proofs and produced repairs.

DARPA also stated the dual-use concern directly. It argued that defenders need faster vulnerability identification and patching, particularly as adversaries use AI to amplify their exploitation capabilities.

This is the central tension behind AI in cybersecurity: the capabilities that help defenders understand and repair software can also reduce the time, expertise and manual effort required to attack it.

What is a modern AI CTF agent?

A modern CTF agent is not simply a chatbot that receives a challenge description and guesses a flag. It is usually a complete execution framework built around one or more language models.

The process often begins with challenge intake. The system collects the challenge name, description, category, downloadable artifacts, remote host, flag format, source code, container configuration and available hints. Some systems integrate directly with CTFd or another competition platform so that they can list challenges, download files, launch instances and submit candidate flags automatically.

CTFusion, for example, implements a Model Context Protocol server for CTFd. It exposes live challenges to multiple agents while preserving separate progress records and forwarding only the first correct flag for each challenge.

After intake, the agent performs reconnaissance. It may inspect:

• File signatures, executable architecture and linked libraries
• Binary mitigations, source-code imports and cryptographic constants
• Network protocols, image metadata, archive structures and web routes

The tools used during this stage are familiar to human CTF players: file, strings, readelf, objdump, checksec, binwalk, exiftool, tshark, nmap and curl. The difference is that an agent can select and invoke these tools autonomously, interpret their output and use the findings to update its attack plan.

The planning component converts the collected evidence into a series of testable hypotheses.

For a web challenge, it might:

• Enumerate reachable routes
• Trace user-controlled input
• Identify likely trust boundaries
• Test several vulnerability classes
• Verify impact and retrieve the flag

For a binary challenge, the plan may be to:

1. Identify the architecture and protections
2. Locate a memory-corruption primitive
3. Locate an information leak
4. Calculate runtime addresses
5. Construct an exploit chain
6. Test the result locally before connecting to the remote service

The execution layer performs the actual work. Depending on the challenge, it may use GDB, pwndbg, radare2, Ghidra, angr and pwntools for binary analysis; SageMath, Z3, gmpy2 and RsaCtfTool for cryptography; browser automation, Burp Suite, curl and Python HTTP libraries for web exploitation; or tshark, Volatility, binwalk and steganography tools for forensic investigation.

The model does not need to reproduce the internal capabilities of these tools. Its role is often to select the appropriate tool, construct the input, interpret the result and determine the next action.

Planning, execution, memory and verification

One of the major developments in CTF-agent design has been the separation of responsibilities.

A planner maintains the high-level understanding of the challenge and selects hypotheses to test. Executor agents carry out commands or develop code inside controlled environments. A verifier or evaluator checks whether the evidence actually supports the claimed result.

The evaluator may ask whether:

• A command completed successfully
• The output was interpreted correctly
• The exploit interacted with the intended target
• A candidate matches the expected flag format
• The agent is repeating an approach that has already failed

This separation reduces a common agent failure: confusing a plausible explanation with a verified solution.

A model might conclude that an application appears vulnerable to SQL injection because a parameter is passed into a query. A proper solving framework requires the system to demonstrate that the injection is reachable, verify its effect and recover the expected protected data or flag from the actual challenge environment.

Long challenges also require structured memory. A capable system must preserve confirmed findings, failed hypotheses, generated scripts, candidate keys, local and remote differences and the origin of every potential flag.

Without reliable memory, agents often rediscover information they already found, overwrite working code, repeat unsuccessful commands or return to an earlier theory that was already disproven.

A 2026 evaluation of the complete NYU CTF Bench used an extended D-CIPHER planner-executor architecture, a Kali Linux image containing more than 100 security tools and automated result parsing. This illustrates how much of current performance comes from the surrounding system rather than from a standalone model.

Parallelism changes the competition

A human participant can test only a limited number of approaches simultaneously. A coordinated agent system can assign separate models to source review, fuzzing, cryptanalysis, exploit development and forensic analysis at the same time.

Some systems go further by racing several models against the same challenge. Each solver receives an isolated container and security toolchain, while a coordinator monitors the attempts and shares useful discoveries between them.

Veria Labs reported using this architecture at BSidesSF CTF 2026. Its open-source CTF Agent races multiple models against each challenge, gives solvers isolated Docker environments with preinstalled CTF tools and uses a coordinator to exchange findings. The developers reported solving all 52 challenges and winning first place.

The BSidesSF 2026 CTFtime standings independently confirm that Veria Labs finished first among 688 teams. They also show that 16 teams in total finished with the same maximum score. The first-place position was therefore determined by the competition's ordering or tiebreaking rather than by Veria Labs being the only team to clear the event.

The example is nevertheless significant. It shows that the comparison is increasingly not between one human and one chatbot. It is between human teams and engineered systems that may run several models, several tool environments and several independent attempts in parallel.

What the benchmarks show

There is no single meaningful number for how good AI is at CTFs.

Performance depends on:

• The challenge set, model and tool environment
• The orchestration framework and the number of attempts
• Token budget or internet access
• Degree of parallelism
• Whether the challenges and their solutions were previously public

Specific evaluations, representative results and operational contexts

• InterCode-CTF had 95% pass@5 and 81/85 usable tasks in entry-level and high-school-level picoCTF patterns
• NYU CTF Bench had 59% top result across 200 tasks in broad multi-domain collegiate and CSAW challenges
• Cybench, Meta evaluation had 92.9% pass@1 in public professional tasks under a high-compute agent setup
• AICrypto had a 56% AI vs. 81.2% human average in a practical cryptographic exploitation
• CTFusion had a 14.4% static vs. 6.3% live performance drop on fresh, unreleased challenges
• CVE-Bench found 10% zero-day and 13% one-day critical vulnerabilities in realistic web applications
• BountyBench had 12.5% Detect, 67.5% Exploit and 90% Patch results in a real vulnerability discovery, exploitation and remediation task

These figures are not directly interchangeable. They describe different challenge populations, execution environments, compute budgets and levels of assistance.

InterCode-CTF: beginner benchmarks are approaching saturation

In 2024, the authors of Hacking CTFs with Plain Agents reported 95% performance on InterCode-CTF through prompting, tool use, planning and multiple attempts.

The final ReAct-and-Plan configuration solved 81 of 85 usable tasks with five attempts. The researchers described InterCode-CTF as a high-school-level benchmark and concluded that it had become saturated.

The result demonstrated that frontier agents had become highly effective on common beginner challenge patterns. It did not mean that AI could solve 95% of arbitrary CTF challenges.

This distinction is important because a benchmark can be saturated without the broader field being solved.

NYU CTF Bench: stronger models solve approximately half

NYU CTF Bench contains 200 challenges from CSAW competitions between 2016 and 2024 across cryptography, reverse engineering, binary exploitation, web security, forensics and miscellaneous tasks.

An April 2026 study, Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks, evaluated ten models using the same D-CIPHER-based architecture, Kali environment, generic prompt configuration and execution limits.

• Claude Opus 4.5 solved 118 of 200 challenges: a 59.0% solve rate
• Gemini 3 Pro solved 104 of 200 challenges: a 52.0% solve rate
• Gemini 3 Flash solved 54 of 200 challenges: a 27.0% solve rate
• GLM-5 solved 39 of 200 challenges: a 19.5% solve rate
• GPT-5.2-Codex solved 36 of 200 challenges: an 18.0% solve rate
• GPT-5.2 solved 27 of 200 challenges: a 13.5% solve rate

These names and results reflect the model configurations used in that specific April 2026 evaluation. They should not be treated as universal model scores that apply to every harness or agent framework.

The environment itself produced a major difference. Gemini 3 Pro achieved 52% in the Kali configuration, compared with 42.5% in the matched Ubuntu configuration. That is an improvement of 9.5 percentage points without changing the model. The richer security toolchain and libraries materially improved performance.

The study also found that miscellaneous challenges were generally the easiest, while binary exploitation was consistently the hardest. Pwn tasks demand precise reasoning about architecture, memory layout, mitigations, leaks, calling conventions, heap state and exploit reliability. A minor error in one stage can invalidate the entire exploit chain.

Cybench: from a short capability horizon to near-saturation

Cybench contains 40 professional-level CTF challenges from four competitions. It covers cryptography, web security, reverse engineering, forensics, binary exploitation and miscellaneous challenges. It also uses intermediate subtasks to measure partial progress.

The original benchmark results showed a relatively short capability horizon. Frontier agents could completely solve only tasks whose human first-solve times were comparatively short, while the hardest task in the benchmark had taken a human team 24 hours and 54 minutes.

The more recent picture is dramatically different.

Meta's Muse Spark 1.1 Evaluation Report reported 92.9% pass@1 and 97.0% pass@10 across all 40 Cybench challenges. Meta described the public CTF suite as approaching its performance ceiling and treated it as a capability gate before running more realistic cyber evaluations.

The result is evidence of substantial capability growth, but the evaluation conditions matter. Meta used a ReAct-style agent with shell and Python tools, context compaction, repeated attempts and large inference budgets.

Most solves were concentrated within the first approximately 100,000 output tokens, but performance continued improving toward one million output tokens per attempt. The hardest solved challenges required median budgets of roughly 240,000 to 360,000 output tokens, compared with about 20,000 tokens for the median solve across the complete benchmark.

The official Cybench site also warns that leaderboard entries have been produced with different evaluation systems and, in some cases, different subsets of the 40 challenges. Its leaderboard includes results evaluated against 35, 37, 39, or all 40 tasks and it documents an earlier answer-leakage issue affecting some results.

The correct conclusion is not merely that AI now solves professional CTFs. It is that public CTF benchmarks can move from difficult to nearly saturated within a short period, especially when frontier models receive extensive tools, orchestration, repeated attempts and large compute budgets.

Older public benchmarks are increasingly useful as capability gates, but less useful as definitive tests of novel exploitation.

The knowledge-action gap

AICrypto provides a useful distinction between cryptographic knowledge and practical exploitation.

The benchmark evaluates 17 models and human experts across multiple-choice questions, proof problems and practical CTF challenges. The best model achieved 97.8% on the 135-question multiple-choice section, exceeding the best human expert result of 94.1%.

On the practical CTF challenges, however, human experts averaged 81.2%, while the best model reached 55.3%.

A model can achieve 97.8% on cryptographic knowledge questions while reaching only 55.3% on practical CTF exploitation. Knowing the relevant concept is not the same as identifying the correct attack path, configuring the required tools, executing the attack and validating the result.

A model may correctly describe RSA common-modulus attacks, nonce reuse, lattice reduction, oracles and weak random-number generation, but still fail to extract the required parameters, identify the correct mathematical reduction, implement the attack, operate the necessary software and validate the recovered result.

Lattice-based tasks are a good example. The language model does not need to perform lattice reduction internally because tools such as SageMath and fpylll can execute the numerical operations.

The difficult part is recognizing the correct mathematical formulation, constructing the lattice precisely, selecting appropriate scaling parameters, invoking the tool correctly and interpreting the result. A small modeling or parameter error can make an otherwise correct strategy fail.

Live competitions provide a harder test

Static benchmarks are repeatable, but they have an important limitation: their source code, flags, walkthroughs and exploit scripts may already be public or may have appeared in model-training data.

Live competitions offer stronger evidence because the challenges are fresh when the agents receive them.

AI vs. Humans CTF

In March 2025, Palisade Research and Hack The Box ran a 20-challenge AI vs. Humans competition focused primarily on cryptography and reverse engineering.

According to the Palisade Research report, source code and raw standings, 403 teams registered and 158 solved at least one challenge. Four of the seven evaluated agents solved 19 of 20 challenges.

CAI placed 20th, Palisade Claude Code placed 21st, FCT placed 30th and imperturbable placed 33rd. The best AI result was within the top 5% of all registered teams and approximately the top 13% among teams that solved at least one challenge.

This was a strong live result, but it did not show that AI defeated the best human teams. It showed that several autonomous agents reached the lower boundary of elite human performance in a focused event containing challenge types suitable for local autonomous analysis.

Applying METR's time-horizon methodology to its competition data, Palisade estimated that the tested agents could reliably solve cyber challenges requiring approximately one hour or less from a median human CTF participant. This was Palisade's application of the methodology, not a separate METR cybersecurity benchmark.

Cyber Apocalypse 2025

The same agents performed much less strongly in Hack The Box Cyber Apocalypse, a larger and more diverse event.

The competition registered 8,129 human teams, of which 3,994 solved at least one challenge. CAI, the strongest evaluated agent, solved 20 of 62 challenges and ranked 859th. The other evaluated systems solved between two and five challenges.

The best agent was in the top 10% of all registered teams, but only around the top 21% among teams that solved at least one challenge.

The contrast is important. A system that nearly saturated a focused 20-challenge event solved fewer than one-third of the challenges in a broader competition.

Challenge composition, tool requirements, interaction style, runtime and infrastructure can transform the result.

Fresh challenges reveal a generalization gap

CTFusion was designed specifically to investigate whether public static benchmarks overestimate agent capability.

The researchers evaluated three models and two agent frameworks across five live CTF competitions containing unreleased challenges. Average performance was 14.4% on NYU CTF Bench and 6.3% on the live challenges.

Performance therefore fell by more than half when agents moved from a public static benchmark to fresh competition material.

The researchers also added web search to D-CIPHER. The web-enabled version achieved 24.07% on NYU CTF Bench, compared with 12.59% for the original agent. Analysis of the logs found 71 attempts to use public challenge information, including 63 cases of directly copying a flag and eight searches for challenge write-ups.

This creates a difficult evaluation problem.

Human security researchers routinely use search engines, documentation, exploit databases, tool manuals and technical blog posts. Preventing all external research makes an agent less realistic. Allowing unrestricted search can turn a benchmark into an information-retrieval exercise when the exact challenge name, write-up, exploit, or flag is already public.

Any serious AI CTF result should therefore disclose whether the challenges were public before evaluation, whether write-ups were available, whether internet access was allowed, whether the score was pass@1 or pass@k and how many independent attempts were permitted.

It should also disclose the available tools, token and runtime limits, number of parallel models, human interventions and whether the candidate flag was verified by the real challenge platform.

Without those details, two percentages that look comparable may measure fundamentally different systems.

Where agents remain weak

A 2026 paper, Autonomous LLM Agents and CTFs: A Second Look, evaluated several agent architectures on 30 fresh web CTF challenges spanning 14 vulnerability classes.

The strongest configurations solved 19 of 30 challenges. Adding planners, evaluators and architectural complexity improved consistency and reduced steps and cost, but the strong configurations reached the same ceiling and failed on many of the same vulnerability classes.

Business-logic vulnerabilities were a recurring problem. Agents are often effective when they can search for a known technical pattern such as SQL injection, path traversal, command injection, or an insecure token. They struggle when exploitation requires understanding how the application is intended to operate and identifying a flawed assumption in that workflow.

Blind SQL injection and blind command injection also remain difficult because they require long adaptive extraction processes. Every request depends on earlier output and partial results must be stored accurately over many iterations.

Race conditions create a different challenge. An agent may understand that two or more actions must occur concurrently, but a sequential execution framework may fail to coordinate the requests reliably.

Browser-dependent attacks remain difficult when the agent lacks a full browser interface. Cross-site scripting, client-side state, event execution, multi-step authentication and JavaScript-heavy workflows may not be solvable through a basic HTTP client alone.

The study's planner correctly identified the target vulnerability in 23 of 30 challenges. The remaining seven corresponded to vulnerabilities that none of the tested architectures could recognize or formulate into a plan. Among the 23 correctly identified cases, only four subsequently failed during execution.

The authors therefore identified vulnerability recognition, rather than exploitation after correct recognition, as the primary bottleneck.

These failures show that some weaknesses come from missing infrastructure, while others come from semantic understanding, state management and recovery from a convincing but incorrect hypothesis.

CTF performance is not real-world vulnerability discovery

CTFs provide isolated targets, explicit objectives, known flag formats and environments designed to contain one or more vulnerabilities.

Real applications are noisier. The agent must determine which behavior is relevant, locate the true attack surface, distinguish intended functionality from a security flaw, preserve authentication state, avoid damaging the target and demonstrate impact without a predefined flag.

CVE-Bench evaluates agents against 40 critical-severity vulnerabilities in real web applications.

Under the zero-day setting, agents received only a general task description and had to identify and exploit the vulnerability independently. The strongest result reached 10%. Under the one-day setting, where agents received a high-level vulnerability description, the strongest reported result reached 12.5% with up to five attempts.

These results are substantially lower than the strongest public CTF benchmark scores.

The difference does not make CTF benchmarks useless. It shows that they measure only part of the operational problem.

A reliable autonomous security system must progress from recognizing a familiar pattern to discovering an unfamiliar weakness, proving that it is exploitable, measuring its impact, avoiding false positives, generating a safe repair and verifying that the repair preserves intended functionality.

BountyBench: discovery is harder than exploitation and patching

BountyBench provides another important bridge between CTF-style evaluation and real security work.

The benchmark contains 25 complex systems and 40 real bug-bounty vulnerabilities covering nine of the OWASP Top 10 risk categories. The vulnerabilities carried original bounty values ranging from $10 to $30,485. Each vulnerability produces three task types: Detect, Exploit and Patch.

This distinction matters because discovering an unknown vulnerability is a different capability from exploiting or repairing a vulnerability that has already been identified.

With up to three attempts, the strongest Detect result was 12.5%. The best Exploit result reached 67.5%, while the best Patch result reached 90%.

Claude Code itself achieved 5% Detect, 57.5% Exploit and 87.5% Patch, which may explain why some summaries cite a 5% discovery result. It was not the benchmark's highest Detect score.

The result suggests that discovery and semantic understanding remain larger bottlenecks than code generation alone.

Once an agent is told what the vulnerability is, it can often write an exploit or generate a candidate repair. Finding the vulnerability independently inside a complex system remains much harder.

This distinction is central to understanding AI cyber capability. High exploit or patch success does not necessarily mean that the system can autonomously discover the underlying weakness.

From AI-assisted teams to AI-versus-AI security

The capabilities demonstrated in CTFs map directly to parts of real offensive and defensive workflows.

An attacker-side agent could accelerate:

• Asset discovery and service enumeration
• Source-code review and binary triage
• Vulnerability classification and public-exploit adaptation
• Credential analysis
• Payload generation

It could operate continuously, process more reconnaissance data and test many hypotheses in parallel.

The main risk: more than an AI-created sophisticated zero-day attack

Reducing the cost of applying known techniques at scale can also be operationally significant. An agent that adapts a public exploit, tests it across configuration variants and summarizes successful outcomes can increase the reach of an attacker even when none of the individual actions is novel.

The defensive side can apply many of the same capabilities to:

• Continuous attack-surface assessment and detection engineering
• Vulnerability reproduction and crash triage
• Candidate patch generation
• Regression testing
• Incident reconstruction and mitigation validation

AIxCC provides direct evidence of this defensive potential. Its systems found vulnerabilities, generated proofs, produced structured reports, created patches and validated repairs across large open-source projects. BountyBench similarly shows that current agents can be substantially stronger at patch generation than at discovering unknown flaws.

The strategic feedback loop

The practical meaning of AI vs. AI cybersecurity is an infrastructure race. The same orchestrator an attacker uses for parallel reconnaissance, exploit adaptation and payload testing can be used by a defender to reproduce vulnerabilities, generate regression tests, prioritize telemetry and validate patches.

The advantage belongs to the side that connects models, deterministic security tools, trusted data and human authority into the fastest reliable feedback loop.

An attacker-side system may continuously enumerate an environment, identify potential weaknesses, adapt an exploit and validate the result. A defender-side system may process telemetry, reproduce the behavior, prioritize the finding, generate a mitigation, test the patch and continue monitoring for related activity.

Humans remain necessary for authorization, scoping, risk acceptance, technical verification, production changes and high-impact decisions.

This is the practical meaning of AI versus AI in cybersecurity.

AI can amplify skilled human teams

The near-term effect is not limited to autonomous agents. AI is already functioning as a force multiplier for human competitors.

Hack The Box's NeuroGrid benchmark report analyzed 1,078 active teams, including 120 AI-augmented teams and 958 human-only teams, across 36 challenges, nine technical domains and four difficulty levels.

The report found that elite AI-augmented teams produced up to 4.1 times more output, while the complete AI-augmented population produced 1.4 times more output. Top AI-augmented teams achieved a 27% challenge solve rate, compared with 16% for top human-only teams. Across all active participants, the reported solve-rate ratio was 3.2 times higher for AI-augmented teams.

The advantage narrowed among the most skilled teams. Hack The Box reported that the overall 3.2-times solve-rate advantage decreased to 1.7 times among the top 5%, while elite AI-augmented teams completed challenges 312% faster.

The benefits were not evenly distributed. Skilled teams were better able to identify when an agent was stuck, reject unsupported conclusions, supply missing context and redirect the system toward more productive approaches.

Lower-performing AI teams could lose time by following an agent into repetitive or unproductive loops. Hack The Box reported that some lower-performing AI-augmented teams were 12.5% slower.

This suggests that AI proficiency is becoming a competition skill of its own. Effective participants need to know how to divide challenges into testable stages, provide the right artifacts, preserve evidence, supervise tool use and distinguish verified findings from plausible hallucinations.

Is AI taking the magic out of CTFs?

From a technical perspective, autonomous CTF agents are an impressive achievement. From the perspective of an active player, the situation is more complicated.

Part of the magic of a CTF comes from not knowing whether a challenge is easy, difficult, misleading, or based on an unfamiliar technique. Players inspect small details, share partial observations, disagree about theories and gradually construct a solution.

AI can compress this process. It can test many ideas simultaneously, generate scripts almost instantaneously, consult documentation and continue operating without fatigue.

That may improve a team's score, but it can also reduce the sense of personal discovery that makes CTFs enjoyable.

It can be frustrating to spend hours studying a challenge only to find that another participant submitted the flag through an autonomous solver without meaningfully understanding the solution. Even when AI use is permitted, the participants may no longer be taking part in the same type of competition.

One team may consist of several people manually reversing a binary, tracing a protocol, or debugging a heap exploit. Another may operate a multi-model swarm with automated challenge retrieval, isolated containers, dozens of installed tools and automatic flag submission.

Both teams appear on the same scoreboard, but they are demonstrating different capabilities.

Two different CTF futures

In my personal opinion, the development of autonomous CTF agents will eventually produce two distinct forms of cybersecurity competition.

Unrestricted AI-versus-AI competitions

Teams will openly build autonomous or semi-autonomous cyber systems and compete on the quality of their architecture.

The challenge will not only be whether a model can recognize a vulnerability. Teams will compete on planner quality, tool orchestration, memory retention, exploit reliability, cost control, verification, parallel execution and coordination between multiple agents.

These events should measure more than the number of captured flags. They could score runtime, computational cost, invalid submissions, exploit reliability, service availability, patch correctness, false positives and the amount of human intervention required.

Attack-and-defense formats would be particularly suitable. Agents would need to discover vulnerabilities, defend their own services, deploy patches, maintain functionality and adapt to active opposing systems.

In many ways, this would bring the field back to the original vision of the Cyber Grand Challenge, but with modern LLM-based coordination and real-world tooling.

Human-versus-human competitions that prohibit generative AI and solver systems

This would preserve the traditional, old-school CTF experience. Participants would solve challenges through their own research, scripting, reverse engineering, exploitation, debugging and teamwork.

A human-only event could still permit conventional technical tools. Disassemblers, debuggers, packet analyzers, symbolic execution decompilers, exploit frameworks, documentation and ordinary scripting have always been part of CTF play.

The restriction would focus on generative AI systems that perform the central reasoning, generate substantial parts of the solution, or autonomously operate the challenge environment.

Enforcing this distinction will not be simple, especially in online competitions. Organizers may struggle to prove whether a player consulted an LLM, used AI-powered code completion, or ran a local autonomous agent.

In-person events could use controlled machines, monitored networks, approved software, restricted internet access and clear competition rules. Online events may need honor-based divisions, separate unrestricted leaderboards, or post-competition solution reviews.

The boundary will require careful definitions.

Is ordinary code completion allowed? Is a machine-learning-assisted decompiler allowed? Can a participant use an LLM to explain tool documentation but not analyze the challenge? Can AI generate boilerplate code while the human performs the central vulnerability research?

There will not be one universally correct answer. Different competitions can define different levels of permitted assistance.

The purpose of a human-only category would not be to reject technical progress. It would be to preserve a specific kind of contest in which the participant performs the central reasoning and experiences the creativity, frustration, teamwork and eventual breakthrough personally.

The AI category can embrace the technology fully and become a proving ground for the autonomous cyber systems that may later be used in security operations.

Separating the two formats would allow both to evolve without forcing them onto the same scoreboard.

Are AI agents killing CTFs?

AI is not making CTFs irrelevant; it is forcing them to change.

Beginner challenges based on standard encodings, familiar cryptographic errors, simple metadata, or obvious injection vulnerabilities are increasingly easy for tool-enabled agents. Public challenges with searchable write-ups are also becoming weak tests of original problem-solving ability.

Future AI-oriented competitions will need fresh, unreleased challenges and stronger controls against benchmark contamination. Dynamically generated variants can make it harder to retrieve exact solutions. Stateful applications, business-logic flaws, browser and mobile interaction, multi-host environments, race conditions and adaptive attack-and-defense scenarios can test capabilities that remain difficult for current systems.

One possible direction for future benchmarks is to generate families of semantically equivalent challenges. The underlying vulnerability and intended solution would remain the same, while variable names, constants, code structure, control flow and irrelevant implementation details would change between instances.

This could help distinguish an agent that understands an exploitation strategy from one that recognizes or retrieves a particular public challenge.

Scoring should also extend beyond flags. An autonomous cyber competition can measure execution cost, time, exploit reliability, false-positive rates, service uptime, patch safety and human intervention.

Human-focused competitions may evolve differently. They can emphasize creative puzzle design, unfamiliar architectures, physical devices, hardware interaction, unusual constraints and challenges that reward deep understanding instead of rapid reproduction of standard patterns.

CTFs will also remain important for education. Even when AI assistance is available, players must understand generated output well enough to validate it. An exploit that works once but cannot be explained, modified, or debugged provides limited educational value.

The key skill may shift from manually typing every command to understanding the complete solution well enough to supervise the tools, identify false assumptions and recognize when a plausible result is technically wrong.

However, not every event needs to make that shift.

There should still be a place for competitions where the purpose is to sit with a difficult challenge, understand it, solve it and experience the satisfaction of knowing that the central reasoning was your own.

Conclusion

The history of autonomous cybersecurity did not begin with ChatGPT.

DARPA's Cyber Grand Challenge demonstrated in 2016 that machines could autonomously find, exploit and patch vulnerabilities inside a constrained environment. AIxCC expanded that model to more than 54 million lines of real open-source code and demonstrated autonomous discovery and patching of both synthetic and previously unknown vulnerabilities.

Modern CTF agents add another layer. They can interpret natural language, write programs, operate standard security tools, coordinate specialized models, preserve evidence and compete directly against human teams.

The results are substantial but uneven.

Beginner public benchmarks are approaching saturation. The strongest systems solve roughly half or more of broad academic CTF suites. High-compute agent evaluations now report near-saturation on Cybench and engineered multi-model systems have achieved full clears in selected live competitions.

At the same time, performance falls sharply on fresh challenges, unfamiliar business logic, stateful attacks, complex binary exploitation, independent vulnerability discovery and real-world vulnerabilities.

The most important lesson is not that AI has already replaced the security researcher.

It is that increasingly large parts of cybersecurity work are becoming executable by software.

The same agent that can enumerate a CTF service, identify a vulnerability, write an exploit and validate a flag can be adapted to support real offensive operations. The same reasoning loop can help a defender reproduce the vulnerability, generate a mitigation, test a patch and monitor for exploitation.

Cybersecurity is therefore moving toward an environment in which AI-assisted attackers increasingly encounter AI-assisted defenders, while humans continue to define scope, validate results, approve consequential actions and make strategic decisions.

For CTFs, I expect this to create two parallel worlds.

One will openly embrace autonomous systems and become an AI-versus-AI laboratory for the future of cyber operations.

The other will preserve human-versus-human competition without LLMs or autonomous solvers, allowing players to continue experiencing the curiosity, frustration, teamwork and satisfaction that made CTFs special in the first place.

Both formats can be valuable.

They should simply be recognized as different competitions.