Paper & Survey: To read our full paper, please visit this link. We're also conducting a survey on this topic and would appreciate your participation. If you're interested, the survey is available at this link.
AI is rapidly reshaping critical sectors such as cybersecurity, healthcare, finance, and education. Frontier models are advancing at an unprecedented pace, showing striking gains in mathematics, coding, and scientific problem-solving, as illustrated in the figure below. This rapid progress brings urgent questions about a broad spectrum of AI-related risks, with cybersecurity standing out as a top concern. The current trajectory suggests AI systems may soon possess far more potent "attack capabilities" than previously expected. As frontier AI evolves, addressing the rising cybersecurity threats becomes increasingly critical.
Understanding frontier AI’s impact on the landscape of cybersecurity—its role in attacks, defenses, and the risks it introduces—is essential for keeping pace with its evolving capabilities, enabling early risk mitigation, and guiding policy decisions. To this end, we examine five critical questions:
- What methods should we use to assess the impact of frontier AI on the cybersecurity landscape?
- What is the current status and real-world impact of frontier AI in cybersecurity?
- Will frontier AI benefit attackers or defenders more?
- How will AI reshape future system design and create new security challenges?
- How should society better prepare for the frontier AI era in cybersecurity?
What Methods Should We Use to Analyze Frontier AI’s Cybersecurity Impact?
We propose to use a Marginal Risk Assessment Framework to systematically evaluate how frontier AI is reshaping the cybersecurity landscape across all stages of attack and defense.Marginal Risk Assessment Framework: We define the marginal risks of frontier AI as novel/heightened cyber threats that arise specifically from its misuse—extending beyond conventional attack techniques. This framework isolates AI’s distinctive contributions to the threat landscape, enabling a clearer understanding of its unique impact.
Comprehensive Attack Stage Coverage: To ensure thorough coverage, we break down attack stages using the well-established cyber kill chain methodology (illustrated in the figure below). We then refine these stages into granular categories based on attack targets, objectives, and methods. This structured approach enables pinpointing where AI most significantly alters the attack lifecycle.
Using our marginal risk assessment framework, we systematically cataloged Frontier-AI-leveraged attacks and defenses across the entire attack and defense lifecycle, drawing from hundreds of academic papers and real-world incident reports. Our review included an extensive search of literature and documented incidents to identify how AI is being applied in both offensive and defensive cybersecurity contexts. To further quantify frontier AI’s impact, we analyzed existing benchmarks specifically designed to evaluate performance in security scenarios. This quantitative analysis complements our qualitative findings, offering concrete metrics on AI’s effectiveness at each stage of the attack and defense process—highlighting both its rapid advancements and current limitations in cybersecurity applications.
What Is the Current Status and Impact of Frontier AI in Cybersecurity?
Frontier AI is rapidly transforming the cybersecurity landscape. Our research provides a holistic view of how AI is currently being utilized in both offensive and defensive contexts, as presented in the following table.
| Attack Steps | Summary |
|---|---|
| Step 1: Reconnaissance (Demonstrated in the real world) |
Qualitative: Both research and real-world attacks leverage LLMs in reconnaissance,
including active scanning, victim information gathering, and open-source database search. Quantitative: Current benchmarks rely primarily on text-based prompts, while practical evaluations like AutoPenBench show AI agents’ potential in reconnaissance but still limited. |
| Step 2: Weaponization (Demonstrated in the real world) |
Qualitative: Research papers show LLMs can generate functional malware with high evasion rates
and AI agents aid vulnerability identification and exploitation, although large-scale real-world attacks remain limited to simpler attacks like DDoS, SQL injection, etc. Quantitative: “RedCode” benchmark reveals LLMs’ limitations in generating functional malware. Vulnerability exploitation benchmarks show Claude 3.5 Sonnet achieving cybersecurity practitioner-level skills but falling short of expert capabilities. |
| Step 3-5: Delivery & exploitation & installation (Demonstrated in the real world) |
Qualitative: Initial access and exploitation (installation) are demonstrated in the real world
while persistence remains at research level. Quantitative: Benchmark coverage and quality for this attack step are very limited. |
| Step 6: Command and control (Demonstrated in the real world) |
Qualitative: Most sub-categories have been demonstrated only in research papers, e.g., privilege escalation
through exploit chain generation, and command and control through automated domain generation, while credential access is used in the real world. Quantitative: Benchmark coverage and quality for these attack steps are very limited. |
| Step 7: Action on objectives (Large-scale real world impact) |
Real-world AI-enhanced attacks increase across various systems (Web, mobile, cloud), with malicious purposes including malware deployment, business logic abuse, and credential theft, causing significant financial losses and data breaches. |
| Attacks against humans (Large-scale real world impact) |
Frontier AI escalates attacks against humans, with studies showing increases in social engineering and voice phishing since ChatGPT's adoption. AI is misused for identity theft, deepfake-based fraud, child exploitation, and psychological manipulation. |
| Defense Steps | Summary |
| Step 1: Proactive testing (Demonstrated in the real world) |
Qualitative: Research explores LLMs for proactive cybersecurity testing, including automated
penetration testing and vulnerability detection through code foundation models and AI-enhanced fuzzing. Real-world
demonstrations exist but lack evidence of large-scale adoption. Quantitative:AI penetration testing benchmarks remain nascent, with one end-to-end benchmark AutoPenBench. Vulnerability detection benchmarks show SOTA models achieve limited accuracy, but these benchmarks face challenges in data quality, limited code context, and insufficient test diversity. |
| Step 2: Attack detection (Demonstrated in the real world) |
Qualitative: Frontier AI addresses key limitations of traditional AI methods by eliminating
manual feature engineering, reducing dependence on labeled datasets, and improving generalizability to
out-of-distribution data. Real-world application exists in malware and network intrusion detection. Quantitative: Many benchmarks exist for network intrusion and malware detection but have data and label quality issues. Foundation model-driven detection shows high accuracy, calling for more challenging benchmarks. |
| Step 3: Triage & forensic (Demonstrated in research papers) |
Qualitative: Frontier AI usages are at an early stage, including LLMs for vulnerability analysis, improving symbolic execution with PoC generation and developing AI agents for root cause analysis. Pre-trained foundation models demonstrate superior performance in binary analysis tasks. Quantitative: Recent benchmarks like CRUXEval show promising results for PoC generation with GPT-4o achieving 75% pass@1 success rate, while no public benchmark exists for comprehensive AI-driven reverse engineering evaluation. |
| Step 4-5: Remediation development & deployment (No demonstrated effect) |
Qualitative: Research demonstrates frontier AI can automatically generate security vulnerability patches, though real-world applications like the SQLite3 Off-by-One bug fix remain limited and no specific work exists on AI-assisted remediation deployment. Quantitative: SOTA systems using Claude 3.5 Sonnet resolve about 50% of issues in SWE-bench-verified. However, most SWE-bench issues are non-security bugs, revealing limitations in evaluating frontier AI's defense capabilities. |
| Defense for humans (Demonstrated in the real world) |
Frontier AI shows promise for enhancing defenses against human-targeted attacks, including social bot detection, fraud detection, deepfake detection, and misinformation detection. However, defensive techniques struggle to keep pace with sophisticated attacks, highlighting the need for research on adversarial dynamics between defensive and attack AI systems. |
For attacks against systems, our research reveals a nuanced picture of AI-enabled attacks on systems. While AI-driven attacks have been successfully demonstrated in research settings, they have yet to make a significant impact at scale in real-world scenarios for many attack steps. Currently, the major real-world applications of AI in attacks are concentrated in the reconnaissance, weaponization, and final execution phases, with limited use in intermediate steps.
Benchmark evaluations show uneven AI performance across the attack lifecycle. In reconnaissance, basic AI agents—such as LLMs with tool access—show early success, identifying targets on remote networks and conducting surveillance. In the weaponization phase, capabilities are advancing but remain constrained: state-of-the-art commercial models like OpenAI-o1 and Claude 3.5 Sonnet can rival technical novices or cybersecurity apprentices in exploiting vulnerabilities. However, AI performance drops off significantly in later stages. For example, while GPT-4o can generate multi-stage attack sequences, it consistently has low success rates.
These findings underscore a key reality: Despite recent breakthroughs in frontier AI, its practical offensive capabilities remain limited—particularly when it comes to executing full end-to-end cyberattacks.
For attacks targeting humans, the landscape is far more alarming. AI is already being used in large-scale, real-world operations for advanced social engineering, deepfakes, and misinformation campaigns. Since the rise of tools like ChatGPT, incidents of social engineering have surged by 135%, and voice phishing by 260%, according to recent studies. Among these threats, deepfakes stand out as particularly dangerous—for example, Hong Kong authorities reported attackers using AI-generated deepfakes to impersonate a CFO in a video call, successfully stealing $25 million.
AI capabilities in this domain are advancing rapidly and along a troubling trajectory. While studies in 2023 and mid-2024 found AI-generated phishing less effective than human-crafted versions, newer research from late 2024 onward shows that fully automated AI phishing can now rival expert human attackers in effectiveness.
For cyber defenses, frontier AI has primarily been applied to proactive testing and attack detection, with limited adoption in vulnerability triage and very little or virtually none in remediation development or deployment—even in research. While AI holds potential for enhancing automated defenses, key challenges remain around robustness, transparency, and generalizability.
One of the most persistent hurdles is the slow pace of remediation deployment, which still relies heavily on manual processes; for example, study shows that hospitals currently average 471 days to fully implement patches. As frontier AI matures, it could help dramatically shorten this timeline, narrowing the vulnerability window. However, this shift is likely to be gradual and long-term—not an immediate fix.
Will Frontier AI Benefit Attackers or Defenders More?
We argue that frontier AI will likely benefit attackers more than defenders in the short term due to three key factors:
- Equivalence Classes: Many AI capabilities designed for defense can be readily repurposed for offensive use. For example, techniques developed for penetration testing can enhance targeted attacks, and AI-generated patches may inadvertently aid exploit development. The table below shows examples of the key equivalence classes we identified in this dual-use context.
- Pen. testing
- Vulnerability detection
- Enable more targeted attacks
- Find vulnerabilities in target systems
- ML-based threat detection
- Lifelong monitoring
- Develop stronger evasion methods
- Re-purpose it to monitor defenses
- PoC & root cause
- Reverse engineering
- Facilitate localization & exploitation
- Understand targets and steal source code
- Patch & testing generation
- Automated configuration
- Malware & weapon & exploit generation
- Automated installation and gain access
- Multimodal generation
- Automated reconnaissance and delivery
- Fundamental Asymmetry: Beyond dual-use capabilities, a core asymmetry between attackers and defenders gives adversaries a disproportionate edge when leveraging frontier AI. This imbalance is driven by three key factors:
- Asymmetry in Cost of Failures: Attackers need only one successful exploit, while defenders must protect against every attack. This low margin for error severely constrains defensive strategies. In AI-powered threat detection, both false positives and false negatives are costly—false positives disrupt operations, while a single false negative can compromise an entire system.
- Asymmetry in Remediation Deployment and Required Resources: Even with known fixes, remediation is slow and resource-intensive to deploy, requiring extensive testing, dependency resolution, global deployment, and post-deployment verification—all demanding significant resources and time. In contrast, attackers can operate with minimal resource requirements and compressed timelines, exploiting the gap between vulnerability discovery and complete patch deployment.
- Different Priorities of Scalability and Reliability: Defenders aim for reliability and comprehensive protection; attackers optimize for scale and impact. AI lowers barriers for attackers—reducing the need for deep expertise, specific target knowledge, or precise execution. A failed attack on one system can still succeed across many others. Meanwhile, AI’s current limitations in robustness, transparency, and generalizability make defenders hesitant to fully trust and deploy these technologies in security-critical environments.
Economic Impact: As AI reduces the expertise needed for cyberattacks and automates complex operations, adversaries can launch more frequent, sophisticated, and stealthy campaigns at low cost. More importantly, AI is beginning to fundamentally alter the cost structure of cyberattacks—enabling novel attack types and attack chains that were previously too complex, resource-intensive, or unscalable to be viable. This opens the door to entirely new threat vectors that challenge existing defense paradigms. Meanwhile, defenders face rising expenses to keep up—especially as traditional defenses become less effective against AI-powered threats. The challenge is compounded by slow, resource-intensive remediation processes that hinder timely responses. This growing cost imbalance puts resource-limited organizations at particular risk. When the cost of defense exceeds the perceived value of the assets being protected, some may scale back or abandon certain protective measures. This creates a dangerous dynamic: as defense weakens, attackers enjoy greater returns on minimal investment—potentially triggering a downward spiral in overall cybersecurity resilience.
| Defense stage | Defense capabilities | Attack usages |
|---|---|---|
| Proactive testing | ||
| Attack detection | ||
| Triage forensic | ||
| Remediation | ||
| Normal utilities |
Resource/Capability Spectrum of Attackers and Defenders: Attackers range from lone individuals with limited resources to state-sponsored groups with extensive funding, while defenders span from small organizations with basic security measures to large enterprises with advanced systems. Given the three points above, even resource-constrained attackers can derive substantial benefits from frontier AI, whereas in the short term, defenders will experience only limited gains. In particular, operational challenges—such as delays in deploying patches—will continue to constrain defensive responses. Although well-resourced defenders might eventually leverage frontier AI to overcome these hurdles, achieving significant benefits will likely require considerable time and effort.
Long-Term Shift Toward Defenders: While attackers currently benefit from AI-driven advantages, this imbalance may gradually shift in favor of defenders as advanced techniques mature, remediation becomes more automated, and systems grow more resilient—making new vulnerabilities increasingly difficult to exploit.
- Strengthening Defensive Leverage: As frontier AI models advance, early access—often granted to large enterprises and government agencies—gives defenders a critical advantage in proactively identifying and patching vulnerabilities. With AI-driven automation accelerating this process, systems steadily evolve to become more robust, with fewer exploitable flaws.
- Secure-By-Design With Provable Guarantees: Furthermore, by leveraging AI for formal verification and adopting a secure-by-design approach, defenders can build fundamentally more secure systems with provable guarantees.
- Change of Attack Economics: This continuous improvement raises the bar for attackers, making the discovery of new vulnerabilities increasingly difficult and resource-intensive. Over time, attackers face diminishing returns, as discovering new vulnerabilities becomes increasingly difficult and resource-intensive—rendering many attack strategies impractical or economically unviable.
How Will AI Transform Future System Design and Introduce New Security Challenges?
Security Challenges in Hybrid AI Systems: The integration of frontier AI into traditional software is giving rise to a new class of hybrid systems—architectures that combine foundation models with non-ML symbolic components such as programming logic, databases, and traditional software services. These systems introduce distinct security challenges that extend beyond those of traditional software or purely AI-based systems.
- Architectural Complexity: The fusion of AI models with symbolic components creates novel integration points that expand the attack surface. Hybrid systems are exposed to both conventional cyber threats and AI-specific vulnerabilities, increasing the complexity of securing them end to end.
- Emerging Attack Vectors: Unique risks include indirect prompt injection (where malicious prompts are embedded to influence AI behavior), AI-to-symbolic attacks (where manipulated AI outputs compromise downstream system components), and new forms of data poisoning that exploit the interaction between AI and symbolic logic.
- Gaps in Defense: Most existing defenses on AI focus on protecting the model itself, leaving system-level vulnerabilities largely unaddressed. Robust, provable security frameworks for hybrid systems remain underdeveloped.
As hybrid systems become more widespread, their growing attack surface will demand entirely new security paradigms—ones specifically tailored to the dynamic, hybrid nature of these architectures.
How Should Society Better Prepare for the Frontier AI Era in Cybersecurity?
Based on our findings, we recommend the following key actions for the security community.
Continuous Comprehensive Marginal Risk Assessment: There is a pressing need to develop comprehensive, fine-grained benchmarks that rigorously evaluate AI capabilities across the full range of attack stages and hybrid system vulnerabilities. As illustrated in the following two tables, no comprehensive benchmark currently exists for assessing AI’s cybersecurity capabilities—highlighting a critical gap. High-quality benchmarks are essential, as they provide the foundation for quantifying AI-driven risks and informing key decisions. To be effective, these benchmarks must prioritize realism, ensuring that tasks and metrics reflect actual security challenges and capabilities. Just as importantly, they must be continuously updated to keep pace with the rapid evolution of frontier AI systems.
| Attack step | Coverage | Metric | Dynamic updates | |
|---|---|---|---|---|
| Step 1 | Reconnaissance | LLM judgment + Dynamic | ✗ | |
| Step 2 | Malware creation | VirusTotal | ✗ | |
| Vulnerability exploitation | Static + Dynamic | ✓ | ||
| Step 3-5 | Initial access & delivery | LLM judgment + Dynamic | ✗ | |
| Exploitation & installation | Dynamic | ✗ | ||
| Persistence & evasion | NA | ✗ | ||
| Step 6-7 | Remote control & target location | LLM judgment + Dynamic | ✓ | |
| Defense steps | Quantity | Notable examples | Key limitations | |
|---|---|---|---|---|
| 1 | Pen. test | Few | AutoPenBench | Limited attacks, target systems, metrics, limited complexity and diversity, low-quality labels |
| Vul. D | Many | PrimeVul, FuzzBench, OSS-Fuzz | ||
| 2 | Att. D | Many | IDS2018, Drebin, BODMAS | Low-quality data and labels, lack OOD/hidden tests |
| 3 | PoC & RT | Few | CRUXEval | Low complexity, no benchmark for root cause, data leakage, incomplete task coverage |
| Rev. Eng. | Many | ByteWeight, ReSym | ||
| 4 | Rem. Dev. | Few | SWE-bench | Noisy, no benchmark for security bugs, limited programming languages (PLs) |
| 5 | Rem. Dep. | None | None | N/A |
Enhanced Defense Development: To stay ahead of rapidly evolving threats, we recommend strategically leveraging frontier AI to strengthen every phase of the cybersecurity defense lifecycle.
- Proactive Testing: Frontier AI can transform vulnerability discovery by augmenting and extending traditional methods. Promising directions include: designing advanced penetration testing and bug finding agents with robust planning capabilities, diverse toolsets, and strong program reasoning capabilities; enhancing static analysis using AI-driven state pruning; fine-tuning models specialized in reasoning about program states and vulnerabilities; and improving fuzzing via intelligent seed generation, adaptive mutation strategies, and grammar discovery. Together, these techniques can significantly improve our ability to identify vulnerabilities ahead of attackers.
- AI-Augmented Threat Detection: As cyber threats grow more dynamic, detection systems must evolve accordingly. Security-specific learning methods should be developed to handle challenges such as attack shifts and low-quality data. Combining AI-based detection with traditional rule-based systems can balance robustness, error tolerance, and scalability. To build trust and improve incident response, enhanced interpretability—through explainable alerts and actionable insights—should be a core design goal.
- Automated Triage and Remediation: One of the most significant security improvements will come from automating the labor-intensive processes of vulnerability triage and remediation. We recommend building multi-agent systems that combine AI planning with traditional program analysis tools for comprehensive root cause analysis, developing patching agents with formal security and correctness guarantees, training specialized models for generating accurate and secure patches, and leveraging AI for automating deployment through test generation, configuration management, and dependency conflict resolution.
- Provable Security Through AI: To move beyond reactive defenses and traditional bug hunting, frontier AI should be integrated with formal verification techniques to build systems with provable security guarantees. This involves training models and agents to automate tasks like theorem proving and program verification, translating legacy code into modern, memory-safe languages (e.g., C to Rust), and generating key verification artifacts such as invariants and function summaries. AI can also enhance constraint solvers through advanced planning techniques and support the creation of specialized frameworks that offer formal guarantees for hybrid systems. By fusing AI with formal methods, we can unlock a new era of software that is secure by design—and provably so.
Secure Hybrid System Design: Securing hybrid systems—those combining AI modules with traditional symbolic components—requires both specialized defenses and a secure-by-design paradigm. We recommend developing dedicated frameworks and design principles that directly address the unique security issues introduced by the integration of AI and conventional software. Real-time monitoring and alerting mechanisms should also be embedded to detect and respond to emerging threats post-deployment.
In the long term, achieving provable security guarantees for hybrid systems is essential. A promising direction is to decompose the verification process into targeted sub-goals for AI and symbolic components, leveraging domain-specific techniques for each. Successfully securing these complex systems will require sustained, collaborative innovation across the security, AI, and programming languages communities.
Strengthened AI Development Practices: Rigorous testing of each new model’s cybersecurity risks should be a standard part of the development process. Before release, models must be evaluated for their potential to enable or carry out cyberattacks. As more robust benchmarks emerge, developers should broaden their evaluations and collaborate with white-hat hackers to simulate real-world scenarios. We also recommend embedding red-teaming practices throughout the AI development pipeline—especially for hybrid systems, where AI and symbolic components interact in complex ways. We can also explore implementing fine-grained privilege and access controls to ensure responsible use and prevent the formation of equivalence classes. Providing established defenders with early or differentiated access can help them stay ahead of potential attackers.
Equally important is transparency: clear documentation of development practices can inform better policy and risk governance. One viable approach is tiered transparency, where information is disclosed at varying levels to the public, trusted third parties, and government agencies. However, implementing this framework presents real challenges, highlighting the delicate balance between openness and security.
Mitigating Human-Related Risks: Frontier AI has dramatically increased the scale, precision, and sophistication of human-targeted attacks—causing widespread harm and exposing a critical need for new mitigation strategies. Addressing these threats must begin with improved education for both users and developers, emphasizing AI-driven attack vectors and fundamental security best practices. Studies show that poor security awareness among users and limited expertise among developers significantly heighten risk and impact.
AI itself can be part of the solution—enabling more personalized, adaptive security training. However, education alone is not enough. Humans remain a persistent vulnerability, and the pace of AI-enhanced attacks increasingly outstrips traditional defenses. To keep up, cybersecurity research must shift its focus toward detecting and countering AI-enabled threats, not just reinforcing protections against conventional attacks.
One important direction is to use AI to serve as a direct countermeasure against AI-enabled attacks. Recent research demonstrates promising strategies such as deploying "AI nannies" or defensive bots that engage attacker-controlled AI agents in conversation, effectively wasting their time and computational resources. Developing such active, AI-based countermeasures is an emerging and essential frontier in defense.
By taking proactive and coordinated steps—advancing defenses, developing robust benchmarks, securing hybrid systems, and mitigating human-targeted risks—the cybersecurity community can shape a future where frontier AI strengthens rather than undermines digital security. While the challenges are significant, thoughtful action today can ensure that AI becomes a powerful force for resilience, not risk.
Read more about these challenges and opportunities in our full paper! To share your insights on this topic, we invite you to join our survey by following this link.