TL;DR
LLM monitoring is the continuous tracking of how a large language model behaves in production: how fast it responds, how much it costs, how often it fails, and whether its outputs are accurate and safe. Good LLM monitoring covers four groups of metrics (performance, cost, quality and safety), alerts on changes from a baseline rather than fixed numbers, and links every alert to the request that triggered it. If the LLM powers an agent that calls tools, you also need agent-level monitoring on top.
What is LLM monitoring?
Every LLM feature, whether it’s a chatbot, a summarizer or the brain of an agent, sends prompts to a model and gets text back. LLM monitoring watches that exchange at scale. It answers three questions continuously: is the model responding reliably, is it affordable, and are its answers any good? (If your LLM powers an agent, our complete guide to AI agent observability covers the wider picture.)
The first two are familiar from any API. The third is what makes LLMs different. A model call can succeed in 800 milliseconds, cost a fraction of a cent, and still return a wrong answer, a made-up citation or a leaked piece of personal data. Traditional monitoring would call that a healthy request.
Why LLMs need their own monitoring
• Outputs vary. The same prompt can produce different answers, so a passing test today doesn’t guarantee a passing answer tomorrow.
• Models change underneath you. Providers update and retire models. A version change can shift behavior across thousands of prompts at once without a single line of your code changing.
• Costs scale with words, not requests. Longer prompts, bigger context windows and verbose answers can multiply spend while request volume stays flat.
• Failures are often semantic. Hallucinations, refusals, off-topic replies and policy violations don’t throw errors. You only catch them by looking at the content.
• Inputs can be hostile. Users and documents can carry prompt injections that try to override your instructions. Monitoring is one of your few ways to see them in production.
LLM monitoring vs. LLM observability
Monitoring tracks defined metrics against targets and tells you when something crosses a line. Observability lets you investigate a single request in depth, through its prompt, context, response and timing, to find out why (LangWatch). They share the same data. Monitoring tells you something is wrong; observability tells you what caused it. For a plain-English definition of the second, see what is LLM observability.
LLM monitoring metrics
Track metrics in four groups. Performance and cost tell you whether the service is healthy. Quality and safety tell you whether it’s doing the right thing. Most teams start with the first two because they’re easy, then discover that their real incidents live in the last two.
| Group | Metric | How to measure it | Example alert |
| Performance | Time to first token (TTFT) | Time from request to the first streamed token, p50 and p95 | p95 above your UX budget for 10 minutes |
| Performance | End-to-end latency | Full response time, p95 and p99, per model and feature | p95 up 50% after a deploy |
| Performance | Error and rate-limit rate | Failed calls and HTTP 429s ÷ all calls, per provider | 2x baseline over 10 minutes |
| Performance | Timeouts and retries | Calls that timed out or needed a retry | Retries above 5% of calls |
| Cost | Tokens in and out | Prompt and completion tokens per request, per feature | Prompt tokens per request jump 30%+ |
| Cost | Cost per request and per user | Token cost by model price, summed by request, user and feature | Daily spend above 2x the 7-day average |
| Cost | Cache hit rate | Requests served from prompt or response cache ÷ all requests | Drops sharply after a prompt change |
| Quality | Groundedness / hallucination rate | Share of answers with claims not supported by the provided context, scored by a judge model on a sample | Falls below your agreed floor |
| Quality | Format validity | Responses that fail to parse (JSON, schema, required fields) | Any spike after a model change |
| Quality | Refusal rate | Answers where the model declines a legitimate request | Rises 2x after a model update |
| Quality | User feedback | Thumbs-down, regenerations and edits per 100 responses | Two-day upward trend |
| Safety | Prompt injection signals | Detector hits on user input and retrieved documents | Any hit on retrieved content |
| Safety | Sensitive data in outputs | PII, secrets or internal data detected in responses | Any match |
| Safety | Policy violations | Toxic, off-topic or disallowed content, scored by a classifier | Any high-severity hit |
Watching for drift
LLM behavior drifts when the model, the prompts or the inputs change. Three cheap signals catch most of it: the distribution of response lengths, the refusal rate, and the trend of your sampled quality scores. Mark every model and prompt change on your charts, because the most common cause of a sudden shift is a version change you or your provider made. A model update is also a reason to re-test, not just to watch; our guide on when to red team an AI agent covers which changes should trigger a new round of testing.
How to set LLM alerts that work
• Alert on change, not absolute values. Traffic and prompt mix shift during the day. “2x the 7-day average for 10 minutes” is more reliable than any fixed number.
• Split by model, feature and provider. An overall error rate hides the one feature on the one model that’s failing.
• Use three severities. Page for safety hits and outages; investigate the same day for quality or cost jumps; review weekly for slow trends.
• Alert on any single safety hit. Leaked personal data or an injection on retrieved content shouldn’t wait for a rate to cross a threshold.
• Set minimum sample sizes. Quality scores from ten requests at 3am will swing wildly. Don’t let them page anyone.
• Link every alert to example requests. The person who gets paged should see the failing prompt and response in one click.
What to put on an LLM monitoring dashboard
Keep one dashboard per LLM feature, laid out in the order you’d ask the questions:
1. Is it up? Request volume, error and rate-limit rate, p95 latency and TTFT, split by model and provider.
2. What is it costing? Daily spend, cost per request and per user, tokens in and out, and cache hit rate.
3. Is it good? Sampled groundedness and quality scores, format-validity failures, refusal rate and user feedback.
4. Is it safe? Injection detections, sensitive-data hits and policy violations, shown as counts, because any one matters.
5. Show me examples. The latest failing or flagged requests, each linked to its full prompt, context and response.
Add release markers for every model, prompt and retrieval change to every time-series chart. Most “mysterious” shifts line up with one.
LLM monitoring tools
LLM monitoring tools fall into three groups:
• Open-source tracing and evaluation tools, such as Langfuse, Arize Phoenix and OpenLLMetry, which you can self-host and extend.
• Commercial LLM observability platforms, such as LangSmith, Braintrust and Datadog LLM Observability, which add managed storage, evaluations and team features.
• Gateways and proxies, which sit between your app and the model provider and record cost, latency and errors for every call without code changes.
When choosing, check for OpenTelemetry support (so you aren’t locked in), built-in evaluations, cost tracking per feature and user, data controls for personal information, and safety signals, not just performance charts. For a side-by-side comparison with pricing, see our roundup of the 12 best AI observability and evaluation tools for agents.
LLM monitoring vs. agent monitoring
LLM monitoring watches individual model calls. That’s enough for a chatbot or a summarizer, where one call produces one answer. It isn’t enough for an agent.
An agent strings many model calls together with tool calls, retrievals and handoffs, then takes real actions. Every individual call can look healthy while the overall task fails: the agent retrieves the wrong policy, skips a check, and approves a refund it shouldn’t. LLM metrics won’t show that, because nothing went wrong at the level of a single call.
| LLM monitoring | Agent monitoring | |
| Unit | One model call | One task, across many calls and tools |
| Main question | Is the model fast, affordable and accurate? | Did the agent complete the task correctly and safely? |
| Typical failure caught | Slow responses, hallucinations, cost spikes | Wrong actions, tool loops, policy bypasses |
| What it misses | Mistakes that span several steps | Nothing at the call level, if both are in place |
If your LLM drives an agent, run both; our guide to AI agent monitoring covers the task-level side. Testing matters too: an agent’s riskiest failures come from what it does, not what it says, which is the difference covered in LLM red teaming vs. agent red teaming.
How to set up LLM monitoring
1. List your LLM features and their owners. Each feature gets its own metrics, dashboard and on-call owner.
2. Instrument every model call. Record model and version, prompt version, tokens, latency, cost, status and a request ID, ideally with OpenTelemetry so you can change backends later.
3. Log prompts and responses safely. Store full content for sampled and flagged requests, and redact personal data before it’s written.
4. Add quality checks. Validate formats on every response, and score a sample for groundedness and relevance with a judge model you’ve checked against human labels.
5. Add safety checks. Run injection detection on inputs and retrieved content, and scan outputs for personal data and policy violations.
6. Build the dashboard and alerts. Use the five-row layout above, baseline-relative thresholds and three severities.
7. Review flagged requests weekly. Turn each new failure pattern into a test case and, where it matters, a new alert.
The bottom line
LLM monitoring starts with the easy numbers, latency, errors and cost, but the incidents that hurt usually live in the content: a hallucinated answer, a leaked record, an injected instruction. Track all four groups, alert on change, and link every alert to the request behind it.
If your model powers an agent, go one level up. Monitor whole tasks, and test the agent the way an attacker would before it reaches users.
Find what’s broken in your agents before your users do. BotGauge red-teams, traces and monitors AI agents in one place.Try Botgauge for Free or book a 30-minute demo.
What is LLM monitoring?
LLM monitoring is the continuous tracking of a large language model’s behavior in production, covering latency, errors, cost, output quality and safety, so teams can catch problems before users report them.
What metrics should you monitor for LLMs?
Track performance (time to first token, latency, error and rate-limit rates), cost (tokens and cost per request, user and feature), quality (groundedness, format validity, refusal rate, user feedback) and safety (prompt injection, sensitive data and policy violations).
What is the difference between LLM monitoring and LLM observability?
Monitoring tracks defined metrics and alerts when they cross a threshold. Observability lets you inspect individual requests in depth to find the cause. Monitoring tells you something is wrong; observability tells you why.
How do you detect LLM drift?
Watch the distribution of response lengths, refusal rates and sampled quality scores over time, and mark every model and prompt change on your charts. Rerun a fixed set of test prompts after each change and compare results.
What are the best LLM monitoring tools?
Popular options include open-source tools such as Langfuse and Arize Phoenix, commercial platforms such as LangSmith, Braintrust and Datadog LLM Observability, and LLM gateways that log every call. The right choice depends on your stack, data rules and whether you need agent-level tracing.
Is LLM monitoring enough for AI agents?
No. LLM monitoring watches single model calls. Agents chain many calls with tools and actions, so you also need task-level monitoring that checks whether the whole task succeeded and stayed within policy.