Skip to main content
You need traces in your project. If you have none, load the sample traces from the empty Logs page. Everything below needs the Admin or Member role. Scoring runs in a separate worker after your traces are ingested.
1

Open Judges

Choose Judges under Evaluation in the sidebar. You see every built-in judge, the level it works at, the progress of any past-data run in the Backfill column, and how much of your monthly quota is used. Tags are collapsed into a “+2” chip. The Manage Evals button on the Logs page opens this same page.
The Judges page with numbered callouts for the state toggle, search, usage, create button, level, backfill, and duplicate

The Judges page. Four judges are on, and Coherence has a past-data run in progress.

  1. State toggle. Turns the judge on or off. Turning it on opens the “start when” dialog.
  2. Search and the All filter narrow the list to built-in or custom judges.
  3. Evaluations this month. Usage against your monthly quota.
  4. Create custom evaluator opens the builder for your own judge.
  5. Level. What the judge reads: Span, Trace, or Session.
  6. Backfill. Progress of a past-data run. Here Coherence is still scoring older data.
  7. Duplicate. Copies a judge into an editable custom one.
2

Turn a judge on

Flip the toggle. A dialog asks for a sampling rate and what data to cover. New spans only is right for most people. Choose Include past spans + new spans to also score older data from a date you choose.You get the past-data choice only when you first turn the judge on. The dialog warns you, and later the setting is locked. To add past data afterwards, duplicate the judge.In the console, a judge turns on at its own level, shown in the Level column. Most judges turn on at span level, Agent Flow and Conversation Quality at session level, and the RAG judges at trace level. Other levels are available through the API.
The Enable dialog with sampling rate, New spans only, Include past spans, a warning, and the Enable button

Turning a judge on asks how much data it should cover.

  1. Sampling rate. Score every item or only a share of them.
  2. New spans only. Score what arrives from now on. This is the default.
  3. Include past spans + new spans. Also score older data from a date you choose, then keep going.
  4. One chance. Past data can only be added when you first turn a judge on.
  5. Enable creates the rule and starts scoring. Cancel leaves the judge off.
3

Send or open traces, then go to Logs

Open Logs. Each enabled judge is a column on the Sessions, Traces, and Spans tabs. A fresh trace shows when it will be scored. Choose Score now to skip the wait for one row.
The Traces tab of Logs with one column for each enabled judge and numbered callouts on the different cell states

The Traces tab of Logs, one column per enabled judge. Correctness was turned on at trace level through the API, so its cells wait to settle.

  1. Not sampled. Coherence runs on 50% of traces and this one was not picked.
  2. Settles 7:37 PM. A trace waits for quiet before it is scored. Hover for the exact time. Score now skips the wait.
  3. Failed with a cause. The judge rejected its credentials, so retrying would not help: “Needs attention”.
  4. Retrying. The judge timed out. BotGauge tries again by itself and shows the next attempt.
  5. Queued. Span judges (PII Leakage, Toxicity) score each span shortly after it arrives.
  6. Pass. A scored result. Hover for the judge’s explanation.
  7. 92%. A score judge shows a percentage. Green means it passed its threshold.
Hover any cell for the judge’s explanation or, for a waiting cell, the exact scoring time.
4

Tune it

Open a judge to change its sampling rate or how long to wait before scoring. Use the Test tab to check it on one real example first. See Rules and settings.

​
Three common setups

​
Safety checks on every response

Turn on Toxicity & Safety and PII Leakage. Both are span judges, so each model response gets its own verdict in Logs shortly after it arrives.

​
Quality of a conversation

Turn on Conversation Quality and Agent Flow. They are session judges, so results appear on the Sessions tab after the conversation has been quiet for 30 minutes.

​
Your own policy

Create a custom boolean judge such as “Refund Policy Adherence”, run Test on a few real spans, then enable it. Add a filter through the API to limit it to the agent that handles refunds. See Custom judges.

​
Judging whole traces

A judge that works at several levels, such as Correctness, turns on at span level in the console. To score whole traces, enable it through the API with subject_kind set to trace. That is how Correctness is shown in the Logs screenshots, which is why its cells wait to settle.

Enable your first judge

Go to the Judges page in BotGauge to turn on a built-in judge and see results in Logs.

Turn on a judge →

​
Next steps

Rules and settings

Sampling, wait time, past data, and filters.

Reading results

What every cell in Logs can say.