See why your
AI agent failed.
BotGauge records every conversation your AI agent has, one step at a time. See what it said, what it did behind the scenes, how long it took, and what it cost, all in one place.
Works with the tools you already use · Python SDK or standard OpenTelemetry
/// OBSERVABILITY / TRACE
Acme workspace
Faithfulness failed on refund-agent · ses_1042, turn 2
- Session · ses_10423.6s · $0.007
- turn 2 of 31.8s
- agent: support1.8s
- llm: plan0.4s
- tool: lookup_order0.5s
- llm: answer0.9s
llm: answer
Input
Human: Can I return this order after 20 days? · Retrieved: “Refunds may be requested within 30 days.”
Output
You can request a refund within 14 days.
Evaluation · Faithfulness
The answer says 14 days. The retrieved policy says 30 days, so it is not supported by the context.
- LlamaIndex
- LangChain
LangGraph- Autogen
- Anthropic
- OpenAI
- CrewAI
- OpenAI Agents
Google ADK- PydanticAI
- MCP
- OpenTelemetry
Not on the list? Anything that sends standard OpenTelemetry data can send traces to BotGauge.
Follow the agent,
not just the request.
An AI agent does more than answer. It plans, looks things up, calls tools, and talks to AI models, often many times before it replies. LLM observability means recording all of that, so when an answer is wrong, slow, or expensive, you can see exactly why.Traditional monitoring tells you a request was slow or failed. LLM observability also tells you whether the answer was good, and why.
11
frameworks verified
3
views of every session
0
code changes to see your first traces (Python)
OTel
built on standard OpenTelemetry
Sessions, turns and spans.
Three simple ideas explain how BotGauge organizes everything your agent does. Think of a customer support chat: a follow-up like “and what about after 20 days?” only makes sense inside the conversation, so BotGauge keeps every turn tied to its session.
/// SESSION / SES_1042
User · turn 1
Support agent
Spans in this turn
Agent · support-agent
1.2s
Model call · plan
0.3s
Tool call · lookup_order
0.4s
Model call · answer
0.5s
Session
The whole conversation with one person, from their first message to their last.
What did this whole conversation cost, and did it go well?
One support chat about order 1042.
Shown in the Sessions tab
/// SESSION / SES_1042
User · turn 2
Support agent
Spans in this turn
Agent · support-agent
1.8s
Model call · plan
0.4s
Tool call · lookup_order
0.5s
Model call · answer
0.9s
Faithfulness · Fail
The retrieved policy says 30 days. The answer contradicts it.
Turn
One round trip inside a session: the person sends a message and the agent replies. A session can have one turn or many.
Which replies were slow, costly, or wrong?
“Can I return this order after 20 days?” and the answer.
Shown in the Traces tab
/// SESSION / SES_1042
User · turn 3
Support agent
Spans in this turn
Agent · support-agent
0.6s
Model call · answer
0.6s
Span
One timed piece of work inside a turn: the agent’s own run, a call to an AI model, a tool such as an order lookup, or a document fetch.
Which span was the bottleneck, and where did it fail?
The lookup of order 1042.
Shown in the Spans tab
/// SESSION / SES_1042
User · turn 1
Support agent
Spans in this turn
Agent · support-agent
1.2s
Model call · plan
0.3s
Tool call · lookup_order
0.4s
Model call · answer
0.5s
The answer is only the last step.
See the prompts, retrieved context, and tool calls behind a result. Find the span worth investigating fast, without piecing together disconnected logs.
Follow the conversation
Open a session to read the whole conversation, turn by turn, and see which agent handled each one.
Inspect the execution
Move between Messages, Graph view, and Timeline to see what happened inside a turn.
Put quality in context
Read quality scores next to cost and duration, then open the reason behind a failure.
A whole
conversation.
One connected
view.
A single message rarely tells the whole story. BotGauge groups everything into sessions and turns, and shows which agent handled each one.
- Read the conversation the way your customer saw it.
- Open any turn to see the model calls, tool calls, and lookups behind the reply.
- Switch between Messages, Graph view, and Timeline.
/// SESSION / SES_1042
User · turn 1
Support agent
User · turn 2
Support agent
Faithfulness · Fail
The retrieved policy says 30 days. Open turn 2 to inspect the answer span.
The numbers that matter
Every session, turn, and span comes with the figures teams ask about most. No spreadsheets needed.
$0.004
What each conversation costs
BotGauge works out cost from the tokens each model call used and the price of that model. See it for every session, turn, and span, so you can tell which turn pushed the bill up.
1.8s
How long people waited
Duration for a whole session or turn, and latency for each span. The slowest span stands out on the timeline.
1,842
How much the model read and wrote
Models handle text in small pieces called tokens. More tokens usually means a bigger bill and a slower reply.
What broke, and whether it recovered
Red: a span failed and nothing fixed it. Amber: a warning, such as a failure that a retry or a backup model then fixed. No color: all clear.
0.64
Whether the answer was right
Quality scores sit beside cost and speed, so a fast, cheap answer can’t hide a wrong one.
support-agent
Which agent handled it
See which agent handled each session, turn, and span, and filter down to just one.
Sample values from one example turn. Cost is an estimate based on model price lists.
Spot the problem
turns fast.
The Traces tab lists every turn with what the person asked, how long the reply took, and what it cost. Rows are tinted when something went wrong. Search and filter by status, latency, cost, and more, then open a turn to see exactly why.
- One row per turn, with what the person asked.
- Duration and cost side by side.
- Rows tinted red or amber when something went wrong.
/// TRACES
| Session | Input | Spans | Duration | Cost |
|---|---|---|---|---|
| ses_1042 | Where is my order 1042? | 4 | 1.2s | $0.002 |
| ses_1041 | Move my appointment to Thursday | 5 | 3.4s | $0.006 |
| ses_1040 | Where’s my refund? | 4 | 2.9s | $0.005 |
| ses_1039 | Why was I charged twice? | 4 | 2.2s | $0.004 |
- Red: failed, and nothing fixed it
- Amber: a warning, such as a failure fixed by a retry
Know where the
time and tokens went.
Find the slow spans on the timeline, then switch the view to see where the tokens and the cost went. Quality scores sit alongside, so a fast answer can’t hide a wrong one.
- Duration: how long each span took.
- Tokens: how much the model read and wrote.
- Cost: what each span added to the bill.
/// EVERY SPAN, IN CONTEXT
Not just for engineers.
Everyone who touches an AI agent gets the view they need, from one shared record, so nobody waits on an engineer to explain what happened.
See what customers saw
Read real conversations turn by turn, then see what the agent did behind each reply.
Go from symptom to cause
Jump from a wrong or slow answer to the exact prompt, tool result, or error that caused it.
Know the bill
See what each conversation, agent, and release costs, and how long users wait.
Keep data in check
Detected personal data and credentials are masked before they leave your app, and you choose how much is recorded.
You decide what gets recorded.
Choose how much detail BotGauge keeps. Personal data and credentials are masked by default at every level.
Metadata only
The model used, token counts, and timing. No message content is recorded.
Text
Prompts, responses, and tool data. Images, audio, and embeddings stay out.
Full
Everything in Text, plus images, audio, and embeddings.
// What your user typed
Please refund the order on my card 4242 4242 4242 4242.
You can email me at sam@example.com.
// What BotGauge records
Please refund the order on my card ***************.
You can email me at ***************.
Illustrative example. Masking is on by default and covers detected emails, card numbers, API keys, and passwords.
Use the SDK or your own exporter
BotGauge accepts standard OpenTelemetry, so you can start with a Python package or point an exporter you already run.
| Any OpenTelemetry exporter | BotGauge SDK |
|---|---|
What it is An exporter you already run, pointed at BotGauge | A Python package built on the standard OpenTelemetry SDK |
Sessions, turns, and agent names Not added automatically | Added to every span once your developer sets a session ID |
Masking of personal data and credentials Not applied automatically | On by default |
Start without code changes Depends on your language’s OpenTelemetry setup | Yes, for Python apps. Your developer adds a session ID in code to group conversations. |
Languages Any language with an OpenTelemetry exporter | Python |
The frameworks BotGauge has verified are listed at the top of this page. Other versions are still traced, with a warning, and CI can be set to allow verified versions only.
Safe to run in production
Built so tracing never gets in your agent’s way, or records more than you intend.
Private by default
Detected personal data and credentials are masked before anything leaves your app.
Never in your way
Tracing runs in the background. A slow or unreachable endpoint never blocks or crashes your app.
Roles for your team
Viewer, editor, and admin roles decide who can only look and who can make changes.
Works with your setup
Runs alongside your own OpenTelemetry setup, and your BotGauge key is never sent to your own collectors by default.
Find it. Understand it. Keep checking it
Connect adversarial testing, execution traces, and quality checks around the same agent.
Three steps. Then watch it work
Your developer can connect an agent with a few commands. After that, anyone on the team can open Logs and follow a conversation.
- 01
Get a key.
Sign in at cloud.botgauge.com and create an API key.
- 02
Install
Run pip install botgauge, then botgauge setup --install to add tracing for the frameworks your app uses.
- 03
Run
Start your app with botgauge instrument python app.py, then open Logs to see your first traces. To group them into conversations, your developer passes a session ID.
The Python SDK is an early 0.x release and needs Python 3.10 or later. Pin its version in production.
A few things to know.
Stop guessing.
Follow the trace.
Connect your agent and see exactly what it does in its very next conversation.