> ## Documentation Index
> Fetch the complete documentation index at: https://www.botgauge.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom judges

> Write a BotGauge judge for your own rule: choose scope and output type, use prompt variables, test it on real spans, and create it by API.

Choose **Create custom evaluator** on the Judges page. Custom judges use an AI model as the judge. You describe what to check and what counts as a pass.

The page has the same **Sampling rate** and **Apply evals to** settings as any judge, and the judge starts disabled. Span judges need at least one **Semantic kind**. The **Create eval** button stays disabled until you replace the placeholder comments in the prompt. You can delete a custom judge later, even while it is on.

![The Create custom evaluator page with numbered callouts for name, scope, semantic kind, state, sampling, add variable, prompt, and output type](https://mintcdn.com/botguage/TsL7gBlR3XTR25B3/images/evals/create-custom-evaluator.png?fit=max\&auto=format\&n=TsL7gBlR3XTR25B3\&q=85\&s=2c23c8729c739a4055df5607ecf2e87d)

Creating a custom evaluator.

1. **Name.** Becomes the column title in Logs. The key is generated from it.
2. **Select scope.** Span, trace, or session. Decides which fields the prompt can use.
3. **Semantic kind.** Required for span judges: pick at least one span type, such as LLM.
4. **Eval state.** A new judge starts disabled.
5. **Sampling rate** and **Apply evals to** work as on any judge.
6. **Add variable.** Inserts a field such as `{{input}}` or `{{output}}` into the prompt.
7. **Judge prompt.** Replace the placeholder comments with your own criteria. Create eval stays disabled until you do.
8. **Output type.** Boolean, score, or category.

## Fields in your prompt

Choose **Add variable** to insert fields. Only some fields exist at each level.

| Level | Fields you can use |
| - | - |
| **Span** | `input` `output` `reference` `context` `instructions` `source` |
| **Trace** | `input` `output` `conversation` `trajectory` `retrieved_context` |
| **Session** | `output` `conversation` `trajectory` `retrieved_context` |

## Output types

* **Boolean.** Pass or fail.
* **Score.** A number from 0 to 1 with a pass threshold.
* **Category.** Labels you define. Each label carries a score and passes or fails.

The output type can't change after you create the judge. A new judge is off until you enable it. To edit a judge that has a running rule, edit with care, because changes that would strand the rule are refused.

## Check a judge before and after you rely on it

* **Test tab.** Run the judge on real spans of your own and read the verdict and reasoning before you turn it on. See [Rules and settings](https://www.botgauge.com/docs/evals/rules-and-settings.md#test-a-judge-on-one-real-example).
* **Start from a built-in.** Duplicate one and edit its prompt instead of writing from scratch.
* **Read the explanation.** Hover a Pass or Fail cell in Logs to see why the judge decided that.
* **Compare with human labels.** The API can score a judge against labels you supply and return agreement, precision, recall, and a suggested threshold with `POST /v1/eval-rules/{id}/calibrate`. The labels are not stored.

> [!WARNING]
>
> Judges are AI models, and their verdicts are advisory. Span text goes into the judge's prompt, so use Test on real examples and spot-check results.

## Create a judge through the API

```http request theme={null}
POST /v1/evaluators/custom
{
  "display_name": "Refund Policy Adherence",
  "default_scope": "span",
  "output_type": "boolean",
  "prompt_template": "Does the ANSWER state the 30-day refund window correctly?\n\nQUESTION:\n{{input}}\n\nANSWER:\n{{output}}"
}
```

```json response theme={null}
{
  "key": "refund_policy_adherence",
  "spec": {
    "display_name": "Refund Policy Adherence",
    "default_scope": "span",
    "output_type": "boolean",
    "variables": { "input": "input", "output": "output" }
  }
}
```

See [API and limits](https://www.botgauge.com/docs/evals/api-and-limits.md) for every endpoint.

Build a custom evaluatorGo to the Judges page in [BotGauge](https://cloud.botgauge.com) and choose Create custom evaluator to start designing your own rule.

[Create a custom judge →](https://cloud.botgauge.com)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.