API and limits

Everything in the console uses these endpoints.

Authentication

Send an API key for your project, or a token exchanged from one, as a bearer token. See API keys. The project comes from the credential, so no project ID goes in the path. Writes need the Admin or Member role. Errors look like this.

error

{ "error": { "code": "...", "message": "...", "request_id": "..." } }

Other details to know:

The since date for past data can be a date or a full date and time. The console sends the start of the local day.
Judge keys in results look like builtin:correctness. When turning a judge on, you can send the bare key (correctness).

Endpoints

Method and path	Purpose
`GET /v1/evaluators/catalog`	List judges with their enabled state, rule, and backfill status. Filters: `search`, `source=builtin\|custom`.
`POST /v1/evaluators/custom`	Create a custom judge.
`PATCH /v1/evaluators/{key}` and `DELETE`	Edit or retire a custom judge. Built-ins are read-only. Disable a judge before deleting it.
`GET /v1/evaluators/evidence-keys?scope=`	Which fields a prompt may use at each scope.
`GET /v1/evaluators/filter-options?scope=`	Fields available to a rule filter.
`POST /v1/eval-rules/enable`	Turn a judge on at a scope.
`GET` and `PATCH /v1/eval-rules/{id}`	Read a rule. Change `sampling`, `filter_text`, or `settle_seconds`.
`POST /v1/eval-rules/{id}/enable` and `/disable`	Resume or pause a rule.
`POST /v1/eval-rules/{id}/estimate-backfill`	Exact count of what a past-data run would score.
`POST` and `GET /v1/eval-rules/{id}/backfill`	Start a past-data run, or read its progress.
`POST /v1/eval-rules/test`	Run a judge once on text you supply. Nothing is stored.
`POST /v1/eval-results/query`	Read results for up to 500 traces, sessions, or spans.
`POST /v1/eval-results/score-now`	Skip the quiet period for one trace or session.
`GET /v1/eval-usage`	Evaluations used this month and the limit.

Turn a judge on, with past data

request

POST /v1/eval-rules/enable
{
  "evaluator_key": "coherence",
  "subject_kind": "trace",
  "mode": "backfill",
  "since": "2026-10-14",
  "sampling": 0.5
}

response

{
  "id": "orr_01a15e8807a97e91a9efccfc5e9449d5",
  "name": "builtin:coherence:trace",
  "subject_kind": "trace",
  "sampling": 0.5,
  "enabled": true,
  "backfill_status": "queued",
  "backfill_since": "2026-10-14T00:00:00+00:00",
  "effective_settle_seconds": 600
}

Skip the wait for one trace

request

POST /v1/eval-results/score-now
{
  "subject_kind": "trace",
  "subject_id": "2d9aafe472de92e59cf7701f28d110fd",
  "evaluator_key": "builtin:correctness"
}

response

202 Accepted
{ "request_id": "esr_d85bcacd80bb4d2da0ad", "status": "queued", "subject_kind": "trace" }

Asking twice returns the same request. Score now works for traces and sessions, needs an enabled rule at that level, and respects sampling and filters. If the worker fails, the cell shows Score now failed and the button stays.

Read results

request

POST /v1/eval-results/query
{ "subject_kind": "trace", "subject_ids": ["2d9aafe472de92e59cf7701f28d110fd"] }

response

{
  "results": {
    "2d9aafe472de92e59cf7701f28d110fd": [
      {
        "evaluator_key": "builtin:correctness",
        "display_name": "Correctness",
        "evaluation_state": "scored",
        "single_outcome": "PASS",
        "explanation": "Stub judge: looks fine.",
        "count": 1, "passed_count": 1, "failed_count": 0,
        "pending_reason": null, "can_score_now": false, "retry": null
      }
    ]
  }
}

evaluation_state is one of scored, pending, retrying, failed, not_sampled, filtered_out, or not_backfilled. scored only means a row exists. Read single_outcome (PASS, FAIL, NOT_EVALUABLE, or ERROR) for the verdict.

Quota and limits

Free plans get 5,000 evaluations per month across the whole organization. Sample data never counts. Each judge verdict counts, including “not evaluable” ones. Paid plans are not capped.
Turning a judge on never fails because of quota. If you run out, scoring pauses and past-data runs wait.
Judges read at most 20,000 characters each of a trace’s input and output.
Sampling is 0 to 1. Filters are up to 2,000 characters. A results query takes up to 500 items. A test accepts up to 50,000 characters.
Past data is limited by how long spans are retained, about 90 days, and can be added only when a judge is first turned on.
Each project can run up to 5 different judges over its sample data.

request

GET /v1/eval-usage

response

{ "month": "2026-10", "quota_state": "enforced", "used": 40, "limit": 5000 }

Keep an eye on cost

Control	How it works
Sampling rate	Score only a share of items. Items that were not picked read Not sampled.
Filters (API)	Limit a rule to matching data. The filter is applied first, then sampling.
Past-data runs	They count toward quota. The API can tell you the exact number of items before you start, and a run pauses if quota runs out.
Usage meter	The Judges page shows evaluations used this month against your limit.

Run evals from code

Open your project settings in BotGauge to copy an API key and start calling the eval endpoints.

Get an API key →

Get started

Judges

Results

Reference

Authentication

Endpoints

Turn a judge on, with past data

Skip the wait for one trace

Read results

Quota and limits

Keep an eye on cost

​Authentication

​Endpoints

​Turn a judge on, with past data

​Skip the wait for one trace

​Read results

​Quota and limits

​Keep an eye on cost

Authentication

Endpoints

Turn a judge on, with past data

Skip the wait for one trace

Read results

Quota and limits

Keep an eye on cost