Custom judges

Choose Create custom evaluator on the Judges page. Custom judges use an AI model as the judge. You describe what to check and what counts as a pass. The page has the same Sampling rate and Apply evals to settings as any judge, and the judge starts disabled. Span judges need at least one Semantic kind. The Create eval button stays disabled until you replace the placeholder comments in the prompt. You can delete a custom judge later, even while it is on.

The Create custom evaluator page with numbered callouts for name, scope, semantic kind, state, sampling, add variable, prompt, and output type — Creating a custom evaluator.

Name. Becomes the column title in Logs. The key is generated from it.
Select scope. Span, trace, or session. Decides which fields the prompt can use.
Semantic kind. Required for span judges: pick at least one span type, such as LLM.
Eval state. A new judge starts disabled.
Sampling rate and Apply evals to work as on any judge.
Add variable. Inserts a field such as {{input}} or {{output}} into the prompt.
Judge prompt. Replace the placeholder comments with your own criteria. Create eval stays disabled until you do.
Output type. Boolean, score, or category.

Fields in your prompt

Choose Add variable to insert fields. Only some fields exist at each level.

Level	Fields you can use
Span	`input` `output` `reference` `context` `instructions` `source`
Trace	`input` `output` `conversation` `trajectory` `retrieved_context`
Session	`output` `conversation` `trajectory` `retrieved_context`

Output types

Boolean. Pass or fail.
Score. A number from 0 to 1 with a pass threshold.
Category. Labels you define. Each label carries a score and passes or fails.

The output type can’t change after you create the judge. A new judge is off until you enable it. To edit a judge that has a running rule, edit with care, because changes that would strand the rule are refused.

Check a judge before and after you rely on it

Test tab. Run the judge on real spans of your own and read the verdict and reasoning before you turn it on. See Rules and settings.
Start from a built-in. Duplicate one and edit its prompt instead of writing from scratch.
Read the explanation. Hover a Pass or Fail cell in Logs to see why the judge decided that.
Compare with human labels. The API can score a judge against labels you supply and return agreement, precision, recall, and a suggested threshold with POST /v1/eval-rules/{id}/calibrate. The labels are not stored.

Judges are AI models, and their verdicts are advisory. Span text goes into the judge’s prompt, so use Test on real examples and spot-check results.

Create a judge through the API

request

POST /v1/evaluators/custom
{
  "display_name": "Refund Policy Adherence",
  "default_scope": "span",
  "output_type": "boolean",
  "prompt_template": "Does the ANSWER state the 30-day refund window correctly?\n\nQUESTION:\n{{input}}\n\nANSWER:\n{{output}}"
}

response

{
  "key": "refund_policy_adherence",
  "spec": {
    "display_name": "Refund Policy Adherence",
    "default_scope": "span",
    "output_type": "boolean",
    "variables": { "input": "input", "output": "output" }
  }
}

See API and limits for every endpoint.

Build a custom evaluator

Go to the Judges page in BotGauge and choose Create custom evaluator to start designing your own rule.

Create a custom judge →

Get started

Judges

Results

Reference

Fields in your prompt

Output types

Check a judge before and after you rely on it

Create a judge through the API

​Fields in your prompt

​Output types

​Check a judge before and after you rely on it

​Create a judge through the API

Fields in your prompt

Output types

Check a judge before and after you rely on it

Create a judge through the API