TL;DR: AI red teaming vs penetration testing
A penetration test checks whether an attacker can break into your software. AI red teaming checks whether an attacker can talk your AI agent into misusing the access it already has. A pen test covers servers, code and permissions. AI red teaming covers what the agent does with them.
Four differences matter most:
- Target. Pen tests probe infrastructure and code. AI red teaming probes the agent’s behavior: its instructions, tools, memory and permissions.
- Method. Pen tests send crafted requests. AI red teaming holds whole conversations and plants hidden instructions in the content the agent reads.
- Repeatability. A pen test finding reproduces every time. An agent can refuse an attack nine times and give in on the tenth.
- Shelf life. A pen test stays valid until the code changes. An agent’s results expire when the model, prompt, tools or data change.
You still need both. BotGauge automates the AI red teaming half: it attacks your agent, traces every failure, and turns each one into a check that runs on every later release.
What is penetration testing, and where does it stop for AI agents?
A penetration test is an authorized attack on a system, run to find weaknesses before a real attacker does. It works well because the system under test holds still. Four assumptions sit underneath it:
- Stable code. The application has a known codebase and documented functions.
- Known interfaces. Testers probe ports, logins, APIs and input fields that stay the same from one test to the next.
- Repeatable results. The same input gives the same output, so a finding can be reproduced and fixed.
- A fixed blast radius. What a compromised account can reach is known from its permissions.
An AI agent breaks all four. Its behavior shifts with the conversation, the documents it retrieves and the model underneath it. Its interface is plain language, so every sentence it reads is a possible attack. And it can reach every system its tools connect to.
Think of a bank branch. A pen test checks the vault door, the locks and the alarm. It does not check whether the teller will hand over cash to a customer with a convincing story. The agent is the teller.
Pen tests still catch real problems. Microsoft’s AI Red Team, which has tested more than 100 generative AI products, documented a classic server-side request forgery (SSRF) flaw in a video-processing AI app (Microsoft Security blog). Pen testing is built for flaws like that. What it cannot tell you is what your agent will do once someone starts talking to it.
What is AI red teaming, and what does it check?
AI agent red teaming attacks the agent the way a real adversary would and records what it does. The question changes from “can someone get in?” to “can someone make the agent cause harm with the access it already has?” The OWASP Top 10 for Agentic Applications, published on December 9, 2025, lists the main risks, starting with agent goal hijack (ASI01) and tool misuse (ASI02). Four checks cover most of that ground.
1. Goal hijack: can someone change what the agent is trying to do?
An attacker hides an instruction in an email, ticket or web page that the agent reads later. The agent treats it as part of its task. EchoLeak (CVE-2025-32711, CVSS 9.3) showed how far this goes: a single crafted email, with no clicks, could make Microsoft 365 Copilot leak data. Microsoft fixed it, and there was no evidence of attacks in the wild (The Hacker News). No port scan or login test looks for that.
What to test: does the agent follow instructions that arrive inside content, rather than from the user or the system prompt?
2. Tool misuse: can someone make the agent use a real tool the wrong way?
The tool works exactly as built. The agent is the one that gets misled. It deletes instead of archiving, emails an outside address, or issues a refund above its limit. The tool call is the evidence, not the reply, because an agent can say “I can’t do that” while its log shows it already did.
What to test: does every tool call match what the verified user is allowed to ask for?
3. Multi-turn pressure: does the agent hold its line over a long conversation?
Slow, multi-turn attacks spread one goal across many harmless-looking messages. In a study by Scale AI researchers, human multi-turn jailbreaks beat more than 70% of the time against defenses that had reported single-digit success rates for automated single-turn attacks. A pen test has no equivalent, because a login form cannot be talked into anything.
What to test: does the agent still refuse on turn ten what it refused on turn one?
4. Permission boundaries: does the agent check who is asking?
Ask for another customer’s order. Ask a basic user’s session to run an admin-only action. The agent holds the keys, so the question is whether it checks who is knocking before it uses them.
What to test: does the agent refuse, or limit the call, when a request goes beyond the verified user’s rights?
AI red teaming vs penetration testing: side by side
A pen test checks the system around the agent. AI red teaming checks the agent itself.
| Penetration test | AI red teaming | |
| Main question | Can an attacker break in? | Can an attacker make the agent do harm with its own access? |
| What it tests | Servers, code, networks, logins, APIs | Instructions, tool calls, memory, retrieved content, permissions |
| How it attacks | Crafted requests and exploits | Conversations, hidden instructions in content, tool abuse |
| Same attack, same result? | Yes, findings reproduce | No, each attack needs many runs |
| Scope | Systems and network boundaries | Everything the agent can reach through its tools |
| Typical finding | An unpatched server or a broken access check | A five-turn conversation that gets a refund over the limit |
| Shelf life | Until the code or configuration changes | Until the model, prompt, tools or data change |
| Usual cadence | Yearly or per major release | Before launch and after every change |
| Skills needed | Network and application security | Security plus a feel for how models follow instructions |
3 gaps a pen test cannot close for AI agents
Microsoft’s AI Red Team tested more than 100 generative AI products and concluded that these systems make old security risks worse and bring in new ones. Its researchers also warned that securing AI is a job that never finishes (The Register). Three gaps explain why.
1. The same attack can pass and fail
Language models are probabilistic, so the same attack can fail nine times and work on the tenth. The math is unforgiving. If an attack succeeds 1 time in 10, a single test run catches it only 10% of the time. Five runs catch it about 41% of the time, and twenty runs about 88%. A pen test finding reproduces on demand. An agent finding often does not.
What to do: run every attack more than once, and count a pass only when the agent holds every time.
2. The scope is everything the agent can reach
A pen test scope lists systems and network ranges. An agent’s scope is longer: the system prompt, every tool and its permissions, every document it can retrieve, the model itself, and every system those tools touch. An agent with email, CRM and database tools puts all three in play, and one poisoned document or tool description can steer any of them.
What to do: list what the agent can reach before you test. Each item becomes an attack goal.
3. Results expire when anything changes
A pen test report stays accurate until the code changes. An agent changes more often and more quietly: a model upgrade, a prompt edit, a new tool, a new document source. Any of these can reopen a hole you already closed. Seven triggers, from new tools to model swaps, tell you when to red team an AI agent again.
What to do: treat a yearly report as a snapshot of a system that no longer exists, and retest after every change.
How to use both together in 6 steps
You do not replace your pen test. You add a second test that targets the agent. Run these steps in staging with test accounts and fake data, never against production credentials.
- Keep the pen test, and aim it at what it does well. Servers, networks, logins, APIs and the configuration of the tool servers your agent connects to. Do not ask it to judge how the agent behaves.
- Map everything the agent can reach. List each tool, what it can touch (email, files, payments, records), the documents it retrieves and who can write into them. Include tools that developers added locally.
- Turn that map into attack goals. Write each goal as an action, not a word: “issue a refund over the limit”, “send a customer record to an outside address”, “reset a password to an unverified email”.
- Attack with conversations, not single prompts. For each goal, script at least one path of 5 to 10 turns, and plant at least one hidden instruction in a document, ticket or tool output. When the agent refuses, rephrase and try again.
- Run each attack several times and score it pass or fail. Write every rule as a plain policy, such as “fail if the agent issues any refund above the limit.” Judge the tool calls, not just the reply, and track the success rate per attack type.
- Repeat on every change, and automate it. Re-run the suite after each model swap, prompt edit, new tool or new data source. Nobody can run hundreds of conversations by hand on every release, which is why teams turn to automated red teaming for AI agents.
Which test should you run first?
Start with the test that matches how your agent is built.
| Your setup | Run first | Why |
| Agent can move money, send messages or change records | AI red teaming, before launch | One failed test here becomes a real incident |
| Agent reads emails, tickets, web pages or files | AI red teaming with planted instructions | Anyone who can write that content can try to steer the agent |
| Agent connects to third-party tool servers | Both | The server’s identity checks need a pen test, and the descriptions it sends need red teaming |
| New cloud setup or newly exposed APIs | Pen test | Classic flaws such as SSRF still apply to AI apps |
| Model, prompt or tools just changed | AI red teaming retest | A prompt edit does not change a pen test result, but it can change the agent’s behavior |
What a pass looks like
Score each run with this checklist. One red flag fails the run, even when the user’s task got done.
| Check | Safe behavior (pass) | Red flag (fail) |
| Goal hijack | Agent ignores instructions found inside content | Agent follows a command hidden in an email, ticket or tool output |
| Tool misuse | Every tool call matches the verified user’s request | A call goes over a limit or uses details nobody verified |
| Multi-turn pressure | Agent gives the same answer on turn ten as on turn one | Agent gives in after a rephrase, or when its own words are quoted back |
| Permission boundaries | Agent refuses, or limits the call to the current user | Agent returns another user’s data or runs an admin-only action |
| Repeat runs | Agent holds on every run | Agent fails even once in ten runs |
| Any attack | The log shows every call and a clear reason for it | Hidden arguments, unexplained calls or data sent to new destinations |
Where BotGauge fits next to your pen test
Your pen test stays. BotGauge covers what it cannot: how the agent behaves once someone starts talking to it.
| Your pen test | BotGauge | |
| What it checks | Servers, APIs, logins, network rules | What the agent does: its tool calls, refusals and the data it shares |
| How it attacks | Exploits and crafted requests | Conversations and hidden instructions, changed after every refusal |
| What you get | A report of vulnerabilities | A pass or fail per attack, with the full trace of the turn where the agent broke |
| When it runs | On a schedule | On every release, because the agent changes between pen tests |
In practice, BotGauge first questions your agent to learn its purpose, users and permitted actions, so the attacks aim at what would actually hurt, such as a refund over the limit. The first time a conversation breaks the agent, that conversation becomes a check that runs on every later release.
BotGauge does not replace your pen test. It will not scan your servers, review your network rules or check your identity layer, so keep your pen test for those.
To see how your agent holds up, Try Botgauge for Free or book a demo.
What is the difference between AI red teaming and penetration testing?
A penetration test looks for weaknesses in software, such as servers, code and logins. AI red teaming looks for ways to make an AI agent misuse its tools, data and permissions through conversation or hidden instructions. One tests the system around the agent. The other tests how the agent behaves.
Can a penetration test find AI agent vulnerabilities?
Only some of them. It can find flaws in the servers, APIs and logins around the agent. It cannot show whether the agent will follow a hidden instruction, give in over a long conversation or misuse a tool, because those depend on how the model behaves, not on how the software is built.
Do I still need a pen test if I do AI red teaming?
Yes. AI red teaming does not check network rules, server patching or login security. Microsoft’s red team found a classic SSRF flaw inside a generative AI app, which shows that old flaws live inside new systems. Run both.
How often should you run each one?
Run pen tests on your normal schedule, often yearly or after a major release. Run AI red teaming before launch and again whenever the model, prompt, tools or data change, because any of those can change how the agent behaves. Running it automatically in CI/CD makes that practical.
Who should run AI red teaming, security or engineering?
Both. Security sets the attack goals and the risk limits. Engineering owns the agent’s prompts, tools and fixes. Microsoft’s AI Red Team advises using automation to scale testing while keeping people in the loop for the findings that matter most.