Agent monitoringAI agent evaluationEvals

Chatbot Evaluation Metrics: What to Measure

Chatbot evaluation metrics for LLM chatbots: resolution, containment, escalation, groundedness and conversation quality, and the traps in each.
Nov 14, 20268 min read
Book a Demo
Chatbot Evaluation Metrics: What to Measure

TABLE OF CONTENT

SHARE THIS ARTICLE

TL;DR

The chatbot evaluation metrics that matter fall into five groups: outcome (was the problem solved?), handoff (when and why did conversations leave the bot?), answer quality (were replies accurate and grounded?), experience (was it quick and painless?) and safety (did the bot stay within its rules?). Score them at three levels: each reply, each whole conversation, and the customer’s outcome. And beware the most popular chatbot metric, containment rate: a customer who gives up looks exactly like a customer who was helped.

Three levels of chatbot evaluation

A chatbot evaluation can score a single reply, a whole conversation, or the outcome for the customer, and you need all three, because each answers a different question (TestMu).

three levels of chatbot evaluation

•       Reply level: was this answer accurate, relevant, grounded in the help content, and within policy?

•       Conversation level: did the bot keep track of what the customer said, stay on task, and avoid loops and repetition?

•       Outcome level: did the customer’s problem actually get solved, and did they stay solved?

Most failures that annoy customers live at the conversation level: forgetting what was said two turns ago, asking for the same order number twice, or circling without progress. Reply-level scores alone won’t show them.

This post covers chatbots that mostly answer questions. If your chatbot also takes actions, such as issuing refunds or changing bookings, you’ll need agent metrics as well; see AI agent evaluation metrics that matter and our guide to AI agent evaluation.

Score every reply, conversation and outcome

Start Free

The metrics

Outcome: did the customer get what they came for?

MetricDefinitionLevel
Goal completion rateShare of conversations where the user’s goal was achieved, judged from the transcript or the end stateConversation
Resolution rateShare of conversations confirmed as resolved, for example by the customer or by no follow-upOutcome
Successful containmentShare of conversations resolved by the bot with a positive resolution signal, not just no handoffOutcome
Recontact rateShare of customers who come back about the same issue within a set windowOutcome

Handoff: which conversations left the bot, and why?

MetricDefinitionLevel
Escalation rateShare of conversations handed to a personConversation
Escalation qualityWhether handoffs happened when they should, with the right context passed onConversation
Abandonment rateShare of users who leave mid-conversation without resolutionConversation
Fallback rateShare of bot turns that were “I didn’t understand” or a generic fallbackReply

Answer quality: was what the bot said right?

MetricDefinitionLevel
GroundednessShare of claims in a reply supported by the help content or knowledge baseReply
Hallucination rateShare of replies with unsupported or false claimsReply
Answer relevanceWhether the reply addresses what the user askedReply
Context retentionWhether the bot correctly uses what was said earlier in the conversationConversation

Groundedness is usually scored claim by claim, as supported claims divided by all claims in a reply (TestMu).

Experience: was it quick and painless?

MetricDefinitionLevel
Customer satisfaction (CSAT)Post-chat rating from the customerOutcome
Turns to resolutionNumber of exchanges needed to resolve the issueConversation
Response latencyTime from the user’s message to the bot’s replyReply

Safety: did the bot stay within its rules?

MetricDefinitionLevel
Policy violation rateReplies that promise or state things against business policyReply
Jailbreak and injection resistanceShare of manipulation attempts the bot resistsConversation
Personal data exposureReplies that reveal personal or internal informationReply

Two pieces of advice from practitioners are worth repeating: write the unit (per reply, per conversation or per customer) next to every metric you publish, and give each metric a precise event definition, because platforms disagree on what counts as “resolved” (TestMu).

The containment trap

Containment rate, the share of conversations that never reach a human, is the most quoted chatbot metric because it maps directly to cost savings. It’s also the easiest to misread. Some bots report high containment simply because users gave up, which is not the same as resolution (Quickchat). A customer who closes the chat in frustration and phones the call center counts as contained.

The fix is to require a positive signal. Successful containment counts only conversations with a confirmed resolution signal, divided by all conversations the bot handled, with handoffs kept in the denominator (Cekura). Good resolution signals include the customer confirming the answer helped, a completed action such as a booked return, or no contact about the same issue within a set window. If you can only add one metric alongside containment, make it resolution rate (LiveChatAI).

Other traps that make dashboards lie

•       One escalation number. An escalation can mean the bot correctly recognized a complex case, or that it failed and the customer demanded a person. Track why conversations escalate, and at what point, not just how many.

•       Watching levels instead of trends. A fallback rate that climbs over a few days usually means users are asking something new, such as after a product launch or a pricing change. Alert on the change (Cekura).

•       Averages that hide intermittent failures. LLM chatbots are non-deterministic. A healthy average can hide a scenario that fails one time in three. Test important scenarios several times, not once.

•       CSAT from a few happy users. Only some customers answer surveys, and they’re rarely the ones who gave up. Pair CSAT with recontact and abandonment.

•       Mixing units. A per-reply hallucination rate and a per-conversation resolution rate can’t be compared or averaged. Label every number.

How to measure them

Before release

Build a set of test conversations, not just single questions. Include everyday requests, multi-turn conversations where key details arrive late, users who change their minds, unanswerable questions, and users trying to push the bot off course. Run each conversation several times and score it with a rubric: five or six criteria, each with a written description of what pass, partial and fail look like in a transcript, so two reviewers reach the same verdict (TestMu).

An LLM judge can apply the rubric at scale. Research on judging chatbots found that strong judge models agreed with human preferences more than 80% of the time, about as often as humans agree with each other (Zheng et al., as reported by Cekura). Still check your own judge against a sample of human-scored conversations before trusting it; LLM-as-a-judge explains how.

Is your chatbot resolving issues, or are users giving up?

Start Now

After release

Track outcome and handoff metrics on all conversations, and score answer quality and conversation quality on a steady sample with the same rubric you used before release. Review a handful of low-scoring conversations every week: they’re where new failure patterns show up first. Online vs offline evaluation for agents explains how the two connect.

Safety is a chatbot metric too

Customer-facing chatbots meet users who deliberately try to make them misbehave: promising refunds, revealing internal instructions, insulting customers or leaking someone else’s data. These attempts rarely look like attacks in a dashboard, and the most effective ones unfold slowly over many turns; see multi-turn attacks on AI agents. Track jailbreak and injection resistance as a metric before release, using adversarial test conversations, and alert on any policy violation in production. How to prevent prompt injection in AI agents covers the defenses.

A starter set

If you’re starting from scratch, track these seven: successful containment, resolution rate, recontact rate, escalation rate with reasons, groundedness on a sample, context retention on a sample, and policy violation rate. Add CSAT and latency once those are stable.

The bottom line

A chatbot dashboard can look excellent while customers quietly give up. Measure the outcome, not just the absence of a handoff; score replies, conversations and outcomes separately; label every metric with its unit; and treat safety as a metric, not an afterthought.

BotGauge LLM evaluations scores conversations against custom criteria, and BotGauge LLM red teaming tests how your chatbot holds up when users try to break it, which automated red teaming for AI agents explains. For tool options, see the 12 best AI observability and evaluation tools for agents; for proof, our customer stories.

Think your agent is safe? Prove it. BotGauge tests your chatbot against the users who try to break it. Try Botgauge for Free or book a 30-minute demo.