TL;DR
The chatbot evaluation metrics that matter fall into five groups: outcome (was the problem solved?), handoff (when and why did conversations leave the bot?), answer quality (were replies accurate and grounded?), experience (was it quick and painless?) and safety (did the bot stay within its rules?). Score them at three levels: each reply, each whole conversation, and the customer’s outcome. And beware the most popular chatbot metric, containment rate: a customer who gives up looks exactly like a customer who was helped.
Three levels of chatbot evaluation
A chatbot evaluation can score a single reply, a whole conversation, or the outcome for the customer, and you need all three, because each answers a different question (TestMu).

three levels of chatbot evaluation
• Reply level: was this answer accurate, relevant, grounded in the help content, and within policy?
• Conversation level: did the bot keep track of what the customer said, stay on task, and avoid loops and repetition?
• Outcome level: did the customer’s problem actually get solved, and did they stay solved?
Most failures that annoy customers live at the conversation level: forgetting what was said two turns ago, asking for the same order number twice, or circling without progress. Reply-level scores alone won’t show them.
This post covers chatbots that mostly answer questions. If your chatbot also takes actions, such as issuing refunds or changing bookings, you’ll need agent metrics as well; see AI agent evaluation metrics that matter and our guide to AI agent evaluation.
The metrics
Outcome: did the customer get what they came for?
| Metric | Definition | Level |
| Goal completion rate | Share of conversations where the user’s goal was achieved, judged from the transcript or the end state | Conversation |
| Resolution rate | Share of conversations confirmed as resolved, for example by the customer or by no follow-up | Outcome |
| Successful containment | Share of conversations resolved by the bot with a positive resolution signal, not just no handoff | Outcome |
| Recontact rate | Share of customers who come back about the same issue within a set window | Outcome |
Handoff: which conversations left the bot, and why?
| Metric | Definition | Level |
| Escalation rate | Share of conversations handed to a person | Conversation |
| Escalation quality | Whether handoffs happened when they should, with the right context passed on | Conversation |
| Abandonment rate | Share of users who leave mid-conversation without resolution | Conversation |
| Fallback rate | Share of bot turns that were “I didn’t understand” or a generic fallback | Reply |
Answer quality: was what the bot said right?
| Metric | Definition | Level |
| Groundedness | Share of claims in a reply supported by the help content or knowledge base | Reply |
| Hallucination rate | Share of replies with unsupported or false claims | Reply |
| Answer relevance | Whether the reply addresses what the user asked | Reply |
| Context retention | Whether the bot correctly uses what was said earlier in the conversation | Conversation |
Groundedness is usually scored claim by claim, as supported claims divided by all claims in a reply (TestMu).
Experience: was it quick and painless?
| Metric | Definition | Level |
| Customer satisfaction (CSAT) | Post-chat rating from the customer | Outcome |
| Turns to resolution | Number of exchanges needed to resolve the issue | Conversation |
| Response latency | Time from the user’s message to the bot’s reply | Reply |
Safety: did the bot stay within its rules?
| Metric | Definition | Level |
| Policy violation rate | Replies that promise or state things against business policy | Reply |
| Jailbreak and injection resistance | Share of manipulation attempts the bot resists | Conversation |
| Personal data exposure | Replies that reveal personal or internal information | Reply |
Two pieces of advice from practitioners are worth repeating: write the unit (per reply, per conversation or per customer) next to every metric you publish, and give each metric a precise event definition, because platforms disagree on what counts as “resolved” (TestMu).
The containment trap
Containment rate, the share of conversations that never reach a human, is the most quoted chatbot metric because it maps directly to cost savings. It’s also the easiest to misread. Some bots report high containment simply because users gave up, which is not the same as resolution (Quickchat). A customer who closes the chat in frustration and phones the call center counts as contained.
The fix is to require a positive signal. Successful containment counts only conversations with a confirmed resolution signal, divided by all conversations the bot handled, with handoffs kept in the denominator (Cekura). Good resolution signals include the customer confirming the answer helped, a completed action such as a booked return, or no contact about the same issue within a set window. If you can only add one metric alongside containment, make it resolution rate (LiveChatAI).
Other traps that make dashboards lie
• One escalation number. An escalation can mean the bot correctly recognized a complex case, or that it failed and the customer demanded a person. Track why conversations escalate, and at what point, not just how many.
• Watching levels instead of trends. A fallback rate that climbs over a few days usually means users are asking something new, such as after a product launch or a pricing change. Alert on the change (Cekura).
• Averages that hide intermittent failures. LLM chatbots are non-deterministic. A healthy average can hide a scenario that fails one time in three. Test important scenarios several times, not once.
• CSAT from a few happy users. Only some customers answer surveys, and they’re rarely the ones who gave up. Pair CSAT with recontact and abandonment.
• Mixing units. A per-reply hallucination rate and a per-conversation resolution rate can’t be compared or averaged. Label every number.
How to measure them
Before release
Build a set of test conversations, not just single questions. Include everyday requests, multi-turn conversations where key details arrive late, users who change their minds, unanswerable questions, and users trying to push the bot off course. Run each conversation several times and score it with a rubric: five or six criteria, each with a written description of what pass, partial and fail look like in a transcript, so two reviewers reach the same verdict (TestMu).
An LLM judge can apply the rubric at scale. Research on judging chatbots found that strong judge models agreed with human preferences more than 80% of the time, about as often as humans agree with each other (Zheng et al., as reported by Cekura). Still check your own judge against a sample of human-scored conversations before trusting it; LLM-as-a-judge explains how.
After release
Track outcome and handoff metrics on all conversations, and score answer quality and conversation quality on a steady sample with the same rubric you used before release. Review a handful of low-scoring conversations every week: they’re where new failure patterns show up first. Online vs offline evaluation for agents explains how the two connect.
Safety is a chatbot metric too
Customer-facing chatbots meet users who deliberately try to make them misbehave: promising refunds, revealing internal instructions, insulting customers or leaking someone else’s data. These attempts rarely look like attacks in a dashboard, and the most effective ones unfold slowly over many turns; see multi-turn attacks on AI agents. Track jailbreak and injection resistance as a metric before release, using adversarial test conversations, and alert on any policy violation in production. How to prevent prompt injection in AI agents covers the defenses.
A starter set
If you’re starting from scratch, track these seven: successful containment, resolution rate, recontact rate, escalation rate with reasons, groundedness on a sample, context retention on a sample, and policy violation rate. Add CSAT and latency once those are stable.
The bottom line
A chatbot dashboard can look excellent while customers quietly give up. Measure the outcome, not just the absence of a handoff; score replies, conversations and outcomes separately; label every metric with its unit; and treat safety as a metric, not an afterthought.
BotGauge LLM evaluations scores conversations against custom criteria, and BotGauge LLM red teaming tests how your chatbot holds up when users try to break it, which automated red teaming for AI agents explains. For tool options, see the 12 best AI observability and evaluation tools for agents; for proof, our customer stories.
Think your agent is safe? Prove it. BotGauge tests your chatbot against the users who try to break it. Try Botgauge for Free or book a 30-minute demo.
