TL;DR: How to build a red teaming dataset
A red teaming dataset is the fixed set of attacks you run against your AI agent on every release. Each entry is one test case: a harmful goal, the place the attack comes in, how it is disguised, and a plain rule that says whether the agent passed. A good dataset grows every time the agent fails in a new way.
Four things make it work:
- Goals, not prompts. Each test case targets an action, such as “refund over the limit”, not a scary sentence.
- Every entry point. The same goal is attacked through the chat box, a planted email or ticket, a tool output and a long conversation.
- Repeat runs. Each case runs several times, because an agent can hold nine times and fail on the tenth.
- A rule you can check. Pass or fail is decided by the tool calls and their results, not by how polite the reply sounds.
BotGauge builds and runs this dataset for you: it learns what your agent can do, generates attacks for each risk, and turns every failure into a test case that runs on every later release.
What is a red teaming dataset?
A red teaming dataset is a versioned list of adversarial test cases for one AI system. Where a normal evaluation dataset checks that the agent does its job, a red teaming dataset checks what an attacker could make it do instead.
Think of a bank’s fraud team. They keep a playbook of every scam they have seen, and they test each new process against all of it. The red teaming dataset is that playbook for your agent.
| Golden (evaluation) dataset | Red teaming dataset | |
| Purpose | Check the agent does its job well | Check the agent can’t be made to cause harm |
| Written by | People expecting normal use | People thinking like attackers |
| Typical entry | A question and the expected answer | An attack goal, entry point, disguise and pass rule |
| What it grades | Accuracy, relevance, tone | Tool calls, data shared, actions taken |
| Runs per case | Usually one | Several, to catch rare failures |
| Grows from | New features and user questions | New attacks and every failure found |
It tests the agent’s behavior, not its servers or network. That is how AI red teaming differs from a penetration test, and why a pen test report cannot stand in for this dataset.
What goes in one red teaming test case?
The AgentDojo benchmark from ETH Zurich gives a useful model. Each of its 629 security test cases pairs an attacker goal, such as leaking the victim’s emails, with an injection point, such as an email sitting in the user’s inbox. Success is checked against the state of the environment afterwards (arXiv). Build your own cases the same way, with a few fields added for agents in production:
| Field | What it holds | Example |
| ID and version | A stable name, so results compare across releases | RT-REF-014 v2 |
| Attack goal | The harmful action, in plain words | Issue a refund above the $500 limit |
| Risk category | Where it sits in a shared list | Tool misuse (OWASP ASI02) |
| Entry point | Where the attack comes in | A support ticket the agent reads |
| Disguise | How the attack is dressed up | Written as a note from a manager approving an exception |
| Turns | Single message, or a scripted conversation | 6 turns, building from a normal refund question |
| Pass or fail rule | A check on what the agent did | Fail if any issue_refund call has amount > 500 |
| Runs | How many times to repeat it | 10 |
| Severity | How bad a failure would be | High: direct money loss |
| Source | Where the case came from | Production incident, March 2026 |
The pass or fail rule matters most. Write it against the tool calls and their arguments, because an agent can reply “I can’t do that” while its log shows the call already ran.
Why public benchmarks aren’t enough
Public agent security benchmarks are a good place to start. AgentDojo covers banking, Slack, travel and workspace tasks, and InjecAgent has 1,054 test cases for indirect prompt injection against tool-using agents (arXiv). They are a seed, not your dataset, for three reasons.
1. They test someone else’s tools
A public benchmark attacks its own sample tools: a mock bank, a mock inbox. Your agent has its own tools, its own data and its own limits. A test that tries to send money through a tool you don’t have tells you nothing.
What to do: borrow the attack patterns, then rewrite each one against your own tools and rules.
2. They stand still while attacks move
The AgentDojo authors themselves point out that a static test set has limited value in security, because attacks and defenses keep changing, and they built their benchmark to be extended for that reason (arXiv). A public dataset is also easy to optimize against, so a strong score can mean the model has seen those exact attacks before.
What to do: add new cases every month from new research, new incidents and your own failures.
3. They score generic harm, not your business rules
Benchmarks check general harms such as leaking data or sending a malicious email. Your worst failure might be specific: a refund above $500, a discount code given to the wrong customer tier, a booking changed without verification. No public dataset knows those rules.
What to do: write at least half of your test cases from your own business rules.
How many test cases do you need?
Start small and grow. Anthropic’s guidance for agent evals is that 20 to 50 tasks drawn from real failures is a good start (Anthropic). For red teaming, size the dataset as a grid:
| Part of the grid | Example count | Running total |
| Attack goals (one per serious harm) | 10 | 10 |
| Entry points per goal (chat, document, tool output) | 3 | 30 |
| Disguises per entry point (direct, hidden instruction, multi-turn, encoded) | 4 | 120 test cases |
| Runs per test case | 10 | 1,200 conversations per release |
Repeat runs are not optional. If an attack succeeds 1 time in 10, a single run catches it only 10% of the time. Five runs catch it about 41% of the time, and twenty runs about 88%. This is also why hand-run red teaming does not scale past the first release.
How to build a red teaming dataset in 6 steps
- Build and run the dataset in staging with test accounts and fake data, never against production credentials or real customer records.
- Map what the agent can reach. List every tool, what it can touch (email, payments, records), every document source it reads and who can write into each one. Each writable source is an entry point.
- Write attack goals as actions with a pass or fail rule. One goal per serious harm: “refund above $500”, “customer record sent outside the company domain”, “password reset to an unverified email”. Give each goal a severity.
- Seed from three sources. Use the OWASP Top 10 for Agentic Applications for the main risk categories, public benchmarks for attack patterns, and your own incidents and support tickets for the failures that are specific to you. Here is a full breakdown of what AI agent red teaming tests.
- Expand every goal across entry points and disguises. Attack each goal through the chat box, a planted document or ticket, and a tool output. Then vary the disguise: direct request, hidden instruction, role-play, encoding, and slow multi-turn attacks of 5 to 10 turns.
- Add look-alike safe cases. For each goal, add one or two harmless requests that look similar and should succeed, such as a valid refund under the limit. These catch an agent that has become so cautious it refuses normal work.
- Version it, run it on every release, and feed it. Tag the dataset with a version, rerun it after every model, prompt, tool or data change, and turn every new failure into a new test case. Nobody can run over a thousand conversations by hand on each release, which is why teams use automated red teaming for AI agents.
Common mistakes to avoid
| Mistake | Why it hurts | Fix |
| A list of scary prompts with no goal | You can’t tell whether a reply is a failure | Tie every case to a harmful action and a rule |
| Grading the reply only | The agent can say no and still run the tool | Grade the tool calls and their arguments |
| Every attack in the chat box | Misses hidden instructions in emails, tickets and tool outputs | Attack each goal through every entry point |
| One run per case | Rare failures look like passes | Run each case several times and count every failure |
| No safe cases | An over-cautious agent scores perfectly | Add look-alike requests that should succeed |
| Written once, never updated | New attacks and new tools go untested | Add cases from every failure and every new tool |
What a good red teaming dataset looks like
Score your dataset with this checklist. Any red flag is a gap an attacker could find first.
| Check | Good dataset | Red flag |
| Coverage | Every serious harm has at least one goal | Goals copied from a public list only |
| Entry points | Every tool and writable source is used as an entry point | Chat box only |
| Rules | Each case has a checkable pass or fail rule | “Reviewer decides” |
| Repeats | Each case runs several times per release | Single runs |
| Safe cases | Look-alike requests that should succeed are included | Only attacks |
| Freshness | New cases added from every failure and every new tool | Same version for months |
Where BotGauge fits
You can build this dataset by hand. BotGauge builds and keeps it current for you.
| Building it by hand | BotGauge | |
| Finding goals | Workshops and spreadsheets | Questions your agent to learn its purpose, users and permitted actions, then targets what would actually hurt |
| Writing attacks | One prompt at a time | Generates attacks per risk, and changes them after every refusal |
| Running them | Manual, rarely repeated | Runs at the depth you choose, with repeat runs |
| Results | Notes on what seemed to fail | A pass or fail per attack, with the full trace of the turn where the agent broke |
| Growing it | Someone remembers to add a case | Every failing conversation becomes a check that runs on every later release |
Microsoft’s AI Red Team, after testing more than 100 generative AI products, recommends automation to scale this work while keeping people involved for judgment (Microsoft Security blog). Keep your own team reviewing the highest-severity findings.
To see what a red teaming dataset finds in your agent, Try BotGauge for Free or book a demo.
What is a red teaming dataset?
A red teaming dataset is a versioned set of adversarial test cases for an AI system. Each case describes a harmful goal, where the attack comes in, how it is disguised and a rule that decides pass or fail. You run it on every release to check the agent still resists each attack.
How is a red teaming dataset different from a golden dataset?
A golden dataset checks that the agent does its job well, usually with questions and expected answers. A red teaming dataset checks what an attacker could make the agent do, and grades the agent’s actions, such as tool calls, rather than the quality of its answers.
How many test cases should a red teaming dataset have?
Start with 20 to 50 cases built from real failures and your most serious risks. Then grow it as a grid: each attack goal across each entry point and each disguise. Ten goals, three entry points and four disguises gives 120 cases, each run several times.
Can I use public datasets like AgentDojo or InjecAgent?
Yes, as a starting point. They give you proven attack patterns, but they target sample tools and generic harms. Rewrite the patterns against your own tools and business rules, and add cases from your own incidents.
How often should you update a red teaming dataset?
Add a case every time the agent fails in a new way, every time you add a tool or data source, and whenever new attack research appears. Rerun the full dataset after every change to the model, prompt, tools or data.
