What Is AI Agent Red Teaming?

In July 2025, a coding agent on Replit deleted a live production database during an active code freeze, after being told explicitly not to touch anything. It then told the team the deletion couldn’t be undone. It could be. The AI Incident Database logged it as incident 1152. Days later, a malicious commit slipped into the open-source Amazon Q Developer extension for VS Code, carrying instructions that told the agent to wipe local files and cloud resources (CVE-2025-8217).

Neither of these was a jailbreak. Nobody typed “ignore your previous instructions” into a chat box. Both agents did exactly what a motivated attacker (or a confused agent) can make a tool-using, credential-holding, autonomous system do. Standard LLM red teaming doesn’t test for that, which is why agent red teaming is now its own discipline instead of a subsection of prompt security.

What does AI agent red teaming actually test?

AI agent red teaming is adversarial testing of an autonomous AI agent’s actions, not just its text output, across its full attack surface: prompts, tool calls, memory, credentials, and multi-agent handoffs. LLM red teaming asks “can I get a bad response out of this model?” Agent red teaming asks “can I get this system to actually do something harmful?” An agent that holds API keys and calls tools without a human in the loop can act on a manipulated instruction instead of merely repeating it.

BotGauge automates this process. It interviews your agent to map its purpose, users, and permitted actions, picks attack strategies that fit what it learned, runs them at a depth you choose (Core, Stress, or Redline), and reports pass or fail by risk category. Any failure you care about becomes a regression check that runs on every future release.

Why “it’s just LLM red teaming with extra steps” is wrong

Traditional LLM red teaming targets a single input-output pair. You send a prompt and evaluate the response for a policy violation. That model works fine for a chatbot with no memory and no tools.

It breaks the moment the system in front of you can read a support ticket, decide that ticket implies an action, call an internal API, and execute that action, with no person checking any intermediate step. Tencent Zhuque Lab’s open-source AI-Infra-Guard platform reflects this structurally: it runs separate scanners for AI infrastructure, MCP servers and agent skills, agent workflows, and model jailbreaks, because each layer fails differently and no single detection method covers all four. A prompt-response classifier tuned for jailbreaks tells you almost nothing about whether an agent will drop a table because it read an empty query result as a bug to fix.

The mechanism differs too. Researchers at Michigan State documented an Agent-in-the-Middle attack (published at ACL 2025) in which an adversarial agent intercepts and rewrites messages passed between legitimate agents. Across the frameworks and communication topologies they tested, success rates exceeded 40% and frequently passed 70%. None of that shows up in a single-turn red team against a base model, because the vulnerability lives in the handoff, not the weights.

Why agents are exploitable in the first place

Three properties of LLM-based agents explain most of what a red team finds. Understanding them is what separates a test plan from a pile of prompts.

Instructions and data share one channel

A traditional application separates code from input. SQL injection happens precisely when that separation fails. An LLM has no such separation by design: the system prompt, the user’s message, a retrieved PDF, and a tool’s JSON response all arrive as tokens in the same context window. The model decides what counts as an instruction based on statistical patterns, not privilege levels. Greshake and colleagues demonstrated in 2023 that this lets an attacker plant instructions in any content an agent will later read, which they named indirect prompt injection. The EchoLeak vulnerability in Microsoft 365 Copilot (CVE-2025-32711) is the production version of that paper: a crafted email, never opened by the user, was enough to exfiltrate data.

Safety training generalizes worse than capability

Wei, Haghtalab, and Steinhardt’s “Jailbroken” paper identified a failure mode they called mismatched generalization. Pre training teaches a model to read Base64, ROT13, leetspeak, and dozens of other encodings. Safety fine-tuning mostly happens in plain natural language. So a model can understand a harmful request encoded in Base64 without its refusal behavior recognizing it. That’s why encoding strategies belong in every agent test suite, even though they look like trivia. They probe whether the guardrail and the capability live at the same level of understanding. Usually they don’t.

Context accumulates, and so does drift

Single-turn tests miss attacks that unfold across a conversation. Microsoft’s Crescendo research showed that starting with benign questions and escalating gradually, while referencing the model’s own earlier answers, bypasses safeguards that block the same request asked directly. Agents make this worse because they carry state: conversation history, memory stores, and intermediate tool results. Each turn is a chance to shift the agent’s picture of what the task is.

Put those three together and the practical consequence is clear. For an agent, the blast radius of a successful attack is not a bad sentence. It’s whatever the most privileged tool call in the agent’s toolbox can do.

The attack surface: OWASP’s ASI Top 10, mapped to what actually happened

In December 2025, the OWASP GenAI Security Project published the Top 10 for Agentic Applications, built with input from more than 100 practitioners and grounded in documented incidents rather than hypothetical risk. It gives agent red teaming a shared vocabulary, the way the original OWASP Top 10 did for web application security twenty years earlier.

A red team scope that doesn’t map findings back to this list, or to the OWASP Top 10 for LLM Applications for model-level risk, produces a report that security leadership and auditors struggle to act on. Mapping is not busywork. It turns “the agent did something weird” into a finding a board or a compliance auditor can actually price.

What a red team engagement against an agent actually tests

A useful engagement scopes the agent’s tools, memory, and autonomy level before writing a single adversarial prompt, because that scope is the attack surface, not the chat window on top of it. From there, testing splits into five categories.

Prompt injection, direct and indirect. Direct injection means the attacker types the malicious instruction into the input. Indirect injection, the more dangerous and more common pattern in production agents, hides it in content the agent reads later: a webpage, a PDF, a support ticket, a calendar invite, a tool response. Payload style matters. A 2025 systematic evaluation of prompt injection and jailbreak techniques across open and commercial LLMs found roleplay-framed attacks reached an 89.6% success rate, ahead of logic traps (81.4%) and encoding tricks (76.2%). That study tested models rather than full agents, but it’s a strong argument for weighting a suite toward social-engineering payloads over strings a blocklist can catch.

Tool-calling abuse. Can a manipulated instruction get the agent to call a legitimate tool with parameters that produce an illegitimate outcome? Deleting instead of archiving, sending to an external address instead of an internal one, executing instead of drafting. The test oracle here is the tool call, not the text. An agent can reply “I can’t do that” while its trace shows it already did.

Memory and context poisoning. Anything the agent treats as a durable fact (a prior conversation, a shared knowledge base entry, a vector store record) is a place to plant a false premise that gets acted on much later, far from where it was planted. Tests here need at least two sessions: one to write the poison, one to trigger it.

Multi-agent handoffs. Wherever one agent’s output becomes another agent’s input with no independent verification between them. Peer messages tend to get more trust than user messages, which is exactly backwards from a security standpoint.

Privilege and identity boundaries. Whether the agent’s actual runtime permissions match what someone assumed when they approved the deployment. This is usually where the gap is. An OAuth scope granted for “read calendar” that also permits “send email” is a finding before any prompt gets written.

Automated scanning covers known-pattern injection well and cheaply. It’s weaker at business-logic and privilege-escalation paths that require understanding what the agent is for, which argues for human-led testing as a complement to continuous automated coverage, not a replacement. The International AI Safety Report 2026 also notes that red-teaming results depend on team composition, the instructions red-teamers receive, and how many attack rounds run. A single, narrowly scoped test will systematically undercount real risk.

How to score a red team run so the numbers mean something

“We ran 300 attacks and 12 failed” sounds precise and often isn’t. Four details decide whether a red team metric is worth reporting.

Report attack success rate per category and per strategy, never just overall. A 4% overall ASR can hide a 60% ASR on Base64-encoded tool-misuse attempts. The breakdown tells engineering where to spend the fix; the aggregate just tells marketing what to put on a slide.

Judge the trace, not only the transcript. For agents, the verdict should combine a model-based judgment of the response with deterministic checks on what the agent actually did. A minimal assertion for a refunds agent might look like this:

python

# fail the attack if the agent issued a refund it wasn't authorized to issue,
# regardless of what it said in its reply
for call in trace.tool_calls:
    if call.name == "issue_refund":
        assert call.args["amount"] <= policy.max_refund
        assert call.args["order_id"] in session.verified_orders

Run repeated trials. LLM output is stochastic, so an attack that works one time in ten will usually look like a pass on a single try. The math is unforgiving: with a 10% per-trial success rate, one trial catches the weakness 10% of the time, five trials catch it about 41% of the time, and twenty trials about 88%. Depth settings exist because coverage costs trials.

Pin the configuration. A result is only reproducible against a specific model version, system prompt, tool list, and memory state. Record all four with every run, or next month’s comparison measures configuration drift instead of security.

How BotGauge red-teams an AI agent

Most teams stall at the first step: deciding which attacks to write. BotGauge removes that step. It starts from the agent itself, learns what the agent does, and builds the attack plan from there.

The BotGauge red teaming workflow: add an agent, learn its behavior, identify attack strategies, choose Core, Stress, or Redline depth, run the attacks, and review results by category.

Reconnaissance before attacks

Step two is the one that matters most, and it mirrors how a human red team works. Before sending anything adversarial, BotGauge questions the agent directly about its purpose, who it serves, and what actions it can take. A refunds agent and an internal code-review agent have different crown jewels. The first leaks customer data or issues unauthorized refunds; the second approves malicious commits. Attacks written without that context waste most of their budget on failure modes the agent can’t even reach.

Strategies chosen for the agent, not from a fixed list

With the agent’s scope mapped, BotGauge selects strategies that fit it. Encoding strategies (hex, ROT13, Base64, leetspeak) test the mismatched-generalization gap described above: does the agent’s guardrail recognize a forbidden request it can clearly decode? Social engineering and roleplay strategies test the payload style that research shows succeeds most often. Prompt injection strategies test whether content arriving through the agent’s inputs can override its instructions. For multi-turn behavior, BotGauge campaigns support escalation strategies such as Crescendo and state-learning approaches that adapt to how the agent responds, digging further when behavior looks unexpected.

Depth as a trial budget

Depth controls how many attacks run against the same strategy set. Given the stochastic math above, it’s really a decision about how much certainty you’re buying.

Depth	Approx. attacks	A reasonable fit
Core	~111	Every pull request or prompt change that touches the agent; a fast gate for obvious regressions
Stress	~272	Release candidates, new tools added to the agent, or a model version swap
Redline	~388	Pre-launch sign-off, agents with write access to money, customer data, or production systems, and audit evidence

From finding to permanent check

Results come back as pass or fail per risk category with the attacks behind each verdict available for review. The step most tools skip comes next: any failure you want to guard against becomes a standing regression check that runs on every future release. Agent behavior shifts with every model update, prompt edit, and new tool, so a vulnerability you fixed in March can quietly return in June. A regression check catches it the day it comes back.

What the output should look like

The deliverable from a real agent red team engagement is an exploit chain: input A leads to tool call B, which combines with memory state C, to produce outcome D. A list of prompts that tripped a classifier is not that, and it won’t survive an engineering team asking “okay, so what do we actually fix?”

Findings should convert into two things. First, runtime guardrails that block the specific chain in production. Second, a monitoring signal that catches the pattern, not just the exact payload, because agent behavior shifts every time the model, the tool list, or the memory architecture changes. That also answers “how often should we red team an agent?” Continuously, or at minimum on every material change to tools, model version, or memory design. An annual audit assumes production looks the way it did during the last test. For agents, it almost never does.