Know when your
agent falls short.
Evaluate answers, actions, and entire conversations. Define what good looks like, then see exactly where your agent falls short, with the reasoning behind every verdict.
18 built-in judges · Custom criteria · Span, trace, and session scoring
/// LOGS / EVALUATORS
Acme workspace
Logs
Observability / Logs · refund-agent
Input | Faithfulness |
|---|---|
| Where is my order 1042? | 0.91 |
| Can I return this order after 20 days? | 0.64 |
| Why was I charged twice? | 0.88 |
| Move my appointment to Thursday | 0.93 |
| Where’s my refund? | 0.90 |
| Where’s my refund? | 0.90 |
| Where’s my refund? | 0.90 |
Faithfulness (0.64)
FailedThe answer contradicts the supplied refund window, so it is not supported by the retrieved context.
Evaluator
Faithfulness judge
Threshold
0.80
Was this verdict right?
- LangChain
LangGraph- OpenAI
- Anthropic
- CrewAI
- OpenTelemetry
Score the behavior,
not just the answer.
LLM evaluation is the process of measuring whether an LLM application’s outputs are correct, safe, and useful, using automated scorers such as AI judges or human review. Because outputs vary from run to run, evaluation replaces exact-match testing with scoring against criteria you define.
18
built-in judges
3
scoring levels
0to100%
sampling control
3
result types: pass/fail, score, label
Turn “looks good” into
criteria you can check.
A fluent answer can still get the policy wrong, and your customers will notice before your dashboard does. Score the behavior that matters to your application, then read the evidence behind the result.
Define the criteria
Choose a built-in judge or write criteria for the rules specific to your business.
Test the judge
Try it on your own spans, read its reasoning, and calibrate against human labels.
Score incoming traffic
Turn it on for new data, with filters and sampling to control what gets evaluated.
Generic metrics cannot know your refund policy.
Write a custom judge for the behavior only your application needs. Choose its scope, write the criteria, and select a pass/fail, score, or label result.
- Evaluate a single step, a request, or a conversation.
- Set a visible threshold for score-based judges.
- Test the judge on a real example before you enable live scoring.
- Duplicate a built-in judge to start from a proven prompt.
- Edit a judge and old results keep the version that produced them.
/// CUSTOM JUDGE
Judge name
Refund policy accuracy
Criteria
Check that the agent states the 30-day refund window correctly and does not invent exceptions.
Evaluation scope
Result type
Score threshold
0.80
A fail should start an investigation.
Read why a judge reached its verdict, then inspect the conversation and execution behind it, so you fix the real cause instead of guessing. Calibrate against your own human labels before relying on a judge.
Explore observability/// SESSION EVALUATION
What the agent said
“You have 14 days to request a refund.”
What the policy says
“Refunds may be requested within 30 days.”
Judge reasoning
The answer contradicts the supplied refund window.
Fail · 0.64
More depth. Less guesswork.
Start with 18 judges
Cover answer quality, safety, RAG, agent behavior, and conversation quality.
Check the judge
Test on your data and calibrate with human labels to inspect agreement, precision, and recall.
Choose what gets scored
Apply filters and sampling, with an optional initial pass over past data.
Cover the behavior that matters.
Turn on built-in judges for quality, safety, RAG, and agent behavior. Duplicate any one to edit its prompt. Score judges pass at a threshold you can see.
Answer quality
Right, complete, on topic, consistent, and brief.
- Correctness
- Completeness
- Answer Relevance
- Coherence
- Conciseness
Safety and fairness
Unsafe content, leaked personal data, bias, and prompt injection.
- Toxicity & Safety
- PII Leakage
- Bias
- Prompt Injection
RAG
Whether claims are supported by retrieved documents.
- Faithfulness
- Context Relevance
- Chunk Attribution
Agent behavior
Logical steps, reasoning that matches action, and correct refusals.
- Agent Flow
- Reasoning Coherence
- Refusal Correctness
Conversation
Whether the user left satisfied, whether their goal changed, and the tone used.
- Conversation Quality
- User Intent Change
- Tone
Your own judges
Custom criteria for the rules only your business has.
- Pass or fail
- Score
- Label
See it work before it
runs on anything.
Every judge goes through the same checks before it touches live traffic, so you roll out with confidence and no cost surprises.
- 01
Preview
See which spans, traces, or sessions your filters would pick. No judge is called.
- 02
Test
Run the judge on one real example and read its reasoning before you trust it.
- 03
Estimate
See how many past items match your filters and sampling before you commit.
- 04
Enable
Score new data only, or add past data from a date you choose.
- 05
Read
Verdicts appear beside the trace, span, or session in Logs. Use Score now for one on demand.
A missing input is not
a failed answer.
When a judge lacks what it needs, BotGauge says so instead of guessing, so a gap in the data never shows up as a bad answer.
Pass
The judge ran and the behavior met your criteria.
Fail
The judge ran and it did not. Open the reasoning to see why.
Not evaluable
Required evidence was missing, redacted, or expired. Recorded as such, never as a fail.
Label only
Descriptive judges such as Tone return a label and reasoning, never a pass or fail.
Error
The check itself could not run. Kept apart from quality failures.
Check any judge against your own reviewers. Calibration reports agreement, kappa, precision, recall, F1, and a suggested threshold for score judges.
Three questions.
One loop.
Evaluation works best next to the other two. They share the same traces, so a failing eval and the evidence behind it point at the same place.
| The question | In BotGauge |
|---|---|
LLM observability What did the agent do, step by step? | Sessions, traces, and spans in Logs |
LLM evaluation Did it do the job well and safely? | Judges that score sessions, traces, and spans |
LLM red teaming How does it break under attack? | Rule-based attack runs with evidence and Verify Fix |
Secure by default.
Built for teams shipping agents to production.
SOC 2 Type II
Independently audited security practices.
Fine-grained access
Control who sees each project and resource.
SSO and SAML
Bring your identity provider.
Project-scoped keys
Keys and secrets reach one project only.
Find it. Understand it. Keep checking it.
Connect adversarial testing, execution traces, and quality checks around the same agent.
Practical details.
Before you connect.
Know what is required, what is captured, and what each result means.
Read the integration docsSpans, traces, or sessions
Score individual spans, complete traces, or whole sessions.
Pass/fail, score, or label
Use pass/fail, a score with a threshold, or a descriptive label.
Filters and sampling
Set filters and sampling rates to manage evaluation volume.
Judges are advisory
AI judge verdicts are advisory. Inspect reasoning and calibrate against human labels.
A few things to know.
Define good.
Measure it on every run.
Start with a built-in judge or write your own criteria, and find out where your agent stands today.