Conversational AI Attacks Succeed Nine Times Out of Ten

The Gap Between Testing and Reality

Enterprise AI security programs are optimized for a threat that barely exists in the wild. Amy Chang, leading AI threat intelligence at Cisco, recently shared findings from testing that pitted nearly 7,000 conversational attacks against fifteen production models. Success rates climbed as high as 88.3% when adversaries adjusted their approach across multiple exchanges, a pattern that standard evaluation methods fail to detect entirely.

The research involved 30,090 isolated prompts alongside 6,986 dialogue-based attack sequences. Every model examined showed meaningful vulnerability to the conversational approach, and the two testing methodologies ranked model resilience in completely different orders. Cisco now tracks adversarial performance across 105 models through a public leaderboard, providing visibility into how different architectures respond under pressure.

Chang, who previously ran cyber threat intelligence for JPMorgan Chase and served in U.S. Navy Reserve intelligence roles, described the core problem at an industry security gathering in San Francisco. Snapshot testing fires a malicious prompt and checks the response. Dialogue-based attacks unfold over several turns, probing for weaknesses, adapting to guardrail behavior, and exploiting the model's effort to maintain conversational coherence. The latter mirrors how users actually interact with deployed systems, yet remains absent from most enterprise testing protocols.

A survey conducted in mid-2026 captured responses from 107 enterprises deploying autonomous AI systems. Eighteen percent reported confirmed security breaches involving agents, while another 36% caught incidents before damage occurred. Only three in ten companies isolate high-risk agents in dedicated sandbox environments, and fewer still assign each agent a scoped identity with auditable permissions. Most organizations, 82%, rely on security controls provided directly by model vendors or cloud platforms as their primary defense layer.

Billion-Dollar Bets on Identity Infrastructure

The vulnerability has not gone unnoticed by platform vendors. Palo Alto Networks completed a $25 billion acquisition of CyberArk in February, bringing identity and access management into its security portfolio. CrowdStrike paid $740 million for SGNL in January, while Cisco announced plans to acquire Astrix Security for approximately $400 million. Each deal targets the identity and isolation infrastructure that most enterprises have yet to build out.

Chang's team has automated much of the adversarial testing itself. Agents now analyze deployment scenarios, generate relevant attack vectors, evaluate their likelihood of success, execute the attacks, and score the outcomes. Despite that sophistication, she emphasized that defensive fundamentals remain straightforward. Organizations should start with a structured framework mapping how AI systems can be compromised across their entire lifecycle, from data ingestion through model deployment and supply chain dependencies. Teams can then work backward from observed incidents to identify coverage gaps and prioritize mitigations.

Heather Ceylan, chief information security officer at Box, confirmed the same blind spot from the defender's perspective. Most available red-teaming tools focus on isolated prompts, which diverges sharply from how employees and customers actually use AI in production. Box now runs adversarial agents that think like attackers, iterating across multiple turns to hijack target systems. Without that continuous pressure testing, execution controls remain unvalidated assumptions.

Box deployed agents inside its security operations center roughly a year ago. Analysts initially approved every action manually, and confidence built quickly enough that the team shifted to monitoring mode. Then the agent made a single error. Trust evaporated instantly, and the SOC reverted to full manual oversight. Ceylan noted that monitoring remains critical even when human approval is removed, because models change and interpretation can shift without warning.

Three Layers Versus an Operating System

Box structures its agent security as three nested controls. Permissioning ensures no agent accesses more content than the human who invoked it. Ephemeral sandbox environments spin up for each task, containing the blast radius if an agent is compromised. Runtime execution control restricts available tool calls to only those relevant to the specific task. If an agent is summarizing a document and a prompt injection attempts to forward content to an external address, that action simply does not exist in the agent's available vocabulary.

Ceylan classifies agent actions into three oversight tiers. Non-sensitive operations such as read and summarize require no human intervention. Moderately sensitive actions bypass approval but generate logs for review. Destructive operations, including mass file deletion, always require explicit human authorization. She acknowledged that actions will migrate between categories as the threat landscape evolves, but establishing the framework upfront provides a principled foundation.

Rajesh Parekh, leading AI and machine learning at Intuit, described a different architectural approach. Rather than layering security controls onto individual agents, Intuit built a central platform abstracting security, risk, and fraud modeling so developers never reinvent protection. Permissioning at Intuit does not grant AI systems access to user data; instead, it defines tightly scoped, auditable authority for each agent to perform specific tasks.

Intuit evolved from agents inheriting user permissions to each agent carrying its own identity. The company is now investigating dynamic permission changes tied to the task underway, adjusting authority mid-session as context shifts. Parekh described the model as an AI-powered expert platform where the human expert is embedded in the trust architecture rather than functioning as a gate. Users, agents, and human experts collaborate to solve problems, with each party contributing distinct capabilities.

The End of Manual Code Review

Ceylan addressed the tension between security rigor and development velocity without qualification. Manual secure code reviews and security architecture reviews are finished, she said. Teams that continue relying on those processes will fall behind. Box is building toward a fully autonomous development lifecycle where agents review design documents, apply security requirements, and scan code for vulnerabilities. She expressed optimism that agents will eventually write code without security flaws, though acknowledged that milestone remains distant.

Her advice for development teams skips advanced AI concepts entirely and returns to foundational access control principles. Least privilege access matters more than ever. Organizations that grant agents broad permissions at the outset will struggle to claw back access and retrofit infrastructure for ephemeral credentials and narrowly scoped tasks.

Parekh explained why the attack surface has expanded so rapidly. Agents possess skills, and those skills can become vulnerabilities. Agents access data and tools, and threats can lurk within those tools. The potential impact of malicious code or intent grows dramatically in agentic environments. When Intuit identifies common vulnerability patterns through manual red-teaming exercises, it automates those tests back into the central platform so future agents inherit protection and red-teamers focus on novel threat vectors. Runtime scanning of prompts and responses adds a final layer, stopping suspect responses and escalating to human experts.

Continuous testing remains essential to ensure protections stay robust and account for drift or new dependencies that create novel vulnerabilities, Chang emphasized. Models evolve, permissions drift, and adversaries adapt their techniques across the conversational patterns that snapshot tests never capture.

Intent Versus Probability

A question about intent detection sparked the sharpest exchange of the discussion. Ceylan noted that when Box's own agent operates, the system always knows user intent because it controls the prompt, enabling guardrails and tool-call restrictions to be engineered around that knowledge. The harder challenge arrives when external agents connect and the context behind each request remains opaque.

That exchange exposed a split running through the industry. Some enterprises are building frameworks to quantify and propagate intent as a standard, arguing that complex B2B workflows cannot function without that trust signal. Endpoint security architects have taken the opposite position, betting on probability rather than intent inference for production workloads. Chang explained why models, as currently trained, cannot reliably derive intent from a prompt, which is why deterministic controls and behavioral proxies remain necessary. Ceylan agreed that both approaches are required, noting that programs relying entirely on intent inference are not yet production-ready.

The story about trust collapsing after a single agent error resonated because enterprise agent security is not a problem that gets solved once. Models change, permissions drift, and adversaries adapt across dialogue patterns that one-shot tests never reveal. For the 82% of enterprises relying on vendor-provided controls as their primary security layer, and the 59% evaluating agent security tooling over the next year, the message was direct: test the way attackers operate, across full conversations and continuously, or discover in production what single-turn red-teaming missed.