TL;DR
Hallucination detection is the practice of automatically flagging AI outputs that are false or unsupported, so failures show up on a dashboard instead of in a customer complaint. For AI agents, it has to cover three kinds of hallucination: false statements, invented actions (such as made-up IDs or tools), and claims of work that never happened. No single method catches all three. Production systems layer cheap rule-based checks on every run, a judge model on a sample, and a comparison of what the agent said against what its trace shows it did.
What is hallucination detection?
A hallucination is an output that sounds confident but isn’t supported by facts or by the information the model was given: a fabricated citation, a wrong number, an API method that doesn’t exist (Langfuse). Hallucination detection scores outputs for those unsupported claims automatically, at production scale.
It’s different from hallucination prevention. Prevention tries to stop the model hallucinating in the first place, through better retrieval, prompts and guardrails. Detection assumes some hallucinations will slip through and catches them before, or soon after, they reach a user. It is one part of AI agent observability.
The three types of agent hallucination
| Type | What it looks like | Example | Damage |
| Content | A claim not supported by sources or reality | “Our refund window is 60 days” when the policy says 30 | Wrong information given to a user |
| Action | An invented tool, argument or value | Calling refund_order with an order ID that appears nowhere in the conversation | A real action taken on the wrong thing |
| Outcome | Saying something was done when it wasn’t | “I’ve cancelled your subscription” with no cancellation tool call in the trace | The user believes a problem is solved |
Most guides only cover the first type. In an agent, the second and third are often more dangerous, because they involve real systems. They’re also, helpfully, the easiest to catch: you don’t need a judge model to see that an order ID never appeared in any earlier step, or that no tool call matches a claimed action. Reading those traces is covered in LLM tracing for agents.
Detection methods
| Method | How it works | Catches | Cost | Where to run it |
| Rule-based checks | Validate formats, schemas, IDs and values against known rules | Invalid outputs, invented IDs, impossible values | Near zero | Every run |
| Claim vs. trace | Compare actions the agent claims with tool calls in its trace | Outcome hallucinations | Near zero | Every run |
| Argument provenance | Check that IDs and values in tool arguments appeared in earlier spans | Action hallucinations | Low | Every run with a write tool |
| Groundedness judge | A judge model checks each claim against the retrieved context | Content hallucinations in RAG answers | One model call | A sample, plus all high-risk answers |
| Consistency sampling | Generate several answers and check whether they agree | Unstable, made-up answers | Several model calls | Offline, or a small sample |
| Semantic entropy | Group sampled answers by meaning and measure how scattered they are | Confabulations, even on unseen questions | Several model calls | Offline, or high-stakes queries |
| Human review | Experts label flagged or sampled outputs | Everything, slowly | High | Flagged cases and calibration |
What the research says
Semantic entropy comes from Oxford researchers, published in Nature in 2024. The idea: ask the model the same question several times, group the answers by meaning rather than wording, and measure how spread out the meanings are. High spread suggests the model is confabulating, meaning it’s producing arbitrary, wrong answers. The method worked across tasks without task-specific data (Farquhar et al., Nature, via PMC). Because it needs several generations per question, it’s better suited to offline testing and high-stakes queries than to every live request (Maxim).
Judge models are the production workhorse because they can work without a reference answer, checking the output against the input and context recorded on the same trace (Langfuse). They’re only as good as their prompt and calibration, though, and can be lenient or inconsistent. Check every judge against a set of human-labeled examples before you trust its scores.
Layering works best. Production teams typically combine deterministic checks, semantic scoring and uncertainty signals, running cheap checks first and routing anything suspicious to a more expensive scorer (Openlayer).
A layered setup for production
Run detectors in order of cost, so the cheap ones filter traffic for the expensive ones.
1. Every run: rules, provenance and claim checks. Validate output formats, check that every ID and amount in a tool argument appeared earlier in the trace, and compare each claimed action in the final reply with the tool calls actually made. These cost almost nothing and catch most action and outcome hallucinations.
2. A steady sample: judge-model scoring. Score a fixed share of runs for groundedness against the retrieved context. Track the score over time, split by task type.
3. Every high-risk answer: judge-model scoring, always. Anything involving money, health, legal terms or account changes gets scored on every run, not just in the sample.
4. Flagged cases: human review. Route low-confidence and failed cases to people who know the domain. Their labels also keep your judge calibrated.
5. Offline: deeper methods. Use consistency sampling or semantic entropy on test sets and on the questions that produce the most flags, to find the prompts and topics where your agent is least reliable.
Tie every flag to its trace, so whoever investigates can see which retrieval, tool result or step led to the false claim. For platforms that combine tracing with judge-model scoring, see our roundup of the 12 best AI observability and evaluation tools for agents.
Turn detections into tests
Every confirmed hallucination is a ready-made test case: the input, the context the agent had, and the wrong output. Add it to your regression set and rerun the set on every model, prompt and retrieval change, so the same failure can’t return unnoticed. This is where detection meets evals; observability vs. evals vs. red teaming shows how the three connect.
How to reduce hallucinations in agents
Detection tells you how often hallucinations happen. These changes make them happen less:
• Fix retrieval first. Many content hallucinations start with missing or irrelevant context. Track empty and low-relevance retrievals, and improve them before tuning prompts.
• Let the agent say “I don’t know.” Tell it explicitly to escalate or ask when the context doesn’t contain the answer, and reward that in your evals.
• Constrain tool arguments. Use schemas, enums and lookups so the agent picks real IDs from tool results instead of typing them from memory.
• Confirm before acting. For high-impact actions, have the agent restate the target (“refunding order 4471 for $82”) and check it against the trace before the tool runs.
• Make claims match actions. Generate the final reply from tool results, not from the model’s plan, so it can only report what actually happened.
• Test the edges on purpose. Hallucinations cluster around ambiguous requests, missing data and pressure from users. Attack those edges before launch, and remember that for an agent the riskiest failures are actions, not words. Automated red teaming for AI agents shows how to run these tests on every release.
The bottom line
In a chatbot, a hallucination is a wrong sentence. In an agent, it can be a refund to the wrong order or a promise that something was fixed when nothing happened. Catch the action and outcome hallucinations with cheap checks against the trace on every run, catch content hallucinations with a calibrated judge on a sample, and turn every confirmed case into a test.
Find what’s broken in your agents before your users do. BotGauge red-teams AI agents and records the full trace behind every failure.Try Botgauge for Free or book a 30-minute demo.
How do you detect hallucinations in LLMs?
Combine methods. Run rule-based checks on every output, score a sample with a judge model that compares claims against the source context, and use consistency or uncertainty methods such as semantic entropy offline or on high-stakes questions.
What is semantic entropy?
Semantic entropy is a hallucination-detection method published in Nature in 2024. It samples several answers to the same question, groups them by meaning and measures how spread out the meanings are. A high spread suggests the model is making things up.
Can LLM-as-a-judge detect hallucinations?
Yes, and it’s the most common production method because it works without reference answers. Its reliability depends on the judge’s prompt and calibration, so check it against human-labeled examples first.
What is an action hallucination?
An action hallucination is when an AI agent invents something it then acts on, such as a non-existent tool, a made-up order ID or an amount nobody gave it. You can catch most of them by checking that every argument appeared earlier in the agent’s trace.
How do you reduce hallucinations in AI agents?
Improve retrieval, let the agent admit when it doesn’t know, constrain tool arguments with schemas, confirm high-impact actions before running them, build replies from tool results, and test the edge cases on purpose before release.