LLM Observability: Tracing, Logging, and Evals in Production

A normal monitoring dashboard will tell you your LLM feature is perfectly healthy while it quietly hands every user a wrong answer. The call returned a 200. It took 900 milliseconds. Error rate is zero. And the model confidently invented a refund policy that does not exist. Everything your APM measures is green, and everything that matters is on fire.

That gap is the whole reason LLM observability is its own discipline. Traditional application monitoring watches whether a request completed. For a language model, completing is the easy part — the model almost always returns something. The hard question is whether that something was any good, and answering it means capturing and inspecting the actual content of every call, not just its status code. This is the instrumentation layer that makes everything else — evals, guardrails, cost control — possible, because you cannot improve what you never recorded.

Why LLM apps break your normal monitoring

Two properties of these models defeat the tools you already have.

First, non-determinism. The same input can produce a different output on two runs. A metric like "p99 latency" still works, but "did it return the right thing" has no stable answer to check against. You can't write an assertion that passes every time, so you can't catch failures with the pass/fail plumbing you use for regular code.

Second, there is no single correct answer. A support reply, a summary, a generated plan — these are graded on a spectrum of quality, not a binary of correct or thrown. Your monitoring stack was built to count exceptions. It has no concept of an answer that ran flawlessly and was still unhelpful, off-tone, or subtly false.

So the job of LLM observability is to add the missing dimension: the content of what went in and what came out, recorded in enough detail that you or a judge model can evaluate quality long after the request is gone.

What to actually trace

For every LLM call, capture the whole story, not a summary:

• The full request — system prompt, all messages, and the parameters (model, temperature, max tokens). If you only log a truncated version, you can't replay it.
• The full response — the complete completion, including any refusal or partial output.
• Retrieved context — for a RAG system, the exact chunks you injected. When an answer is wrong, the first question is almost always "did we even retrieve the right context," and you can't answer it if you didn't log what you retrieved.
• Tool calls and results — which tool the model chose, the arguments it passed, and what came back. This is where agents go wrong most often.
• Tokens — input and output counts, kept separate, because they price differently and tell different stories.
• Latency and cost — per call, so you can see which step is slow and which step is expensive.
• A trace ID — one ID that ties every step of a single user request together.

The rule I hold to: log enough that you could reconstruct and replay any call months later without guessing at the inputs. That completeness is what lets a production trace become an eval case instead of a shrug.

Tracing a multi-step agent

A single LLM call is one span. An agent is a tree. The model thinks, calls a search tool, reads the result, calls another tool, and finally answers — and any one of those steps can be the thing that broke. If you only log the final output, debugging an agent is guesswork.

The pattern that works is a parent trace with a child span for every step: each model call, each tool call, each retrieval, nested under one trace ID. When the agent returns nonsense, you open the trace and walk it. Usually the failure is obvious and specific: the model picked the wrong tool, or it passed a malformed argument, or the tool returned an error the model then papered over. You are looking for the exact step where reality and the model's understanding diverged. Without step-level tracing you see only the wrong final answer, which tells you nothing about why. This is a core part of building agents that actually work — you cannot debug a loop you can't see inside.

Logging for evals, not just for debugging

Most people instrument LLM apps to debug the fire in front of them. That's necessary, but the higher-value reason to log is to build your eval set over time.

Every complete trace you capture is a candidate eval case. The full request is the input; the output is there to judge; and once you decide what the right answer should have been, you have a graded case. If you log properly from day one, you are quietly accumulating the most valuable asset in the whole system — a set of real inputs drawn from real usage — instead of having to invent test cases from your imagination later. Structured, replayable logs are the raw material that evals are built from.

Online vs offline evaluation

These are two different jobs, and you want both.

Offline evaluation runs a fixed eval set against your app before you ship. You change one variable — a prompt edit, a model swap — and watch the score move. It's a controlled experiment, and it's how you catch a regression before it reaches a user.

Online evaluation scores real production traffic as it flows. You can't hand-grade every call, so you lean on cheap automated signals: the rate of schema-valid outputs, a judge model scoring a random sample, guardrail trip rates, and explicit user feedback like thumbs-up or a correction. Online eval is how you learn what your offline set never imagined — the weird inputs real people actually send.

Offline tells you a change is safe. Online tells you what's happening now.

Catching regressions and drift

A regression is something you changed: a prompt tweak or model swap that drops your offline score. Your eval suite catches it before release, if you run it as a gate.

Drift is more insidious because nothing in your code changed. Your online metrics slide anyway — maybe your real inputs shifted with a new customer segment, or a provider updated a model version under a name you thought was stable. You catch drift only by tracking a quality metric continuously and alerting when it moves. When the alarm fires, you sample the traces around the dip and read the actual failing outputs. The number tells you something broke; only the traces tell you what.

Close the loop

Here is where it all connects into a system that gets better on its own.

A production trace shows a bad output — flagged by a thumbs-down, a low judge score, or your own review. You take that exact request, write down what the correct behavior should have been, and add it to your eval set as a new case. Now that specific failure is a permanent test. The next prompt or model change that would reintroduce it fails offline before it ships.

That's the loop: production traces → flagged failures → new eval cases → offline gate → safer releases → more traces. Every real-world surprise becomes a guardrail against its own recurrence. An eval set built this way stops being a static snapshot someone wrote once and becomes a living record of everything that has ever gone wrong, which is exactly what you want standing between a model change and your users.

None of this is a product you buy in an afternoon — the tracing category, the eval-runner category, and your own logging all have to agree on one trace ID and one place the data lands. But the payoff is a feature you can change without fear, because the instrumentation catches the silent failures your dashboards were never built to see. For what to do with the failures you catch, see guardrails in production; for the upstream habits that make outputs traceable and gradeable in the first place, prompt engineering for production and the building with LLMs field guide.

FAQ

How is LLM observability different from normal APM? Normal APM measures whether a request succeeded — status code, latency, error rate. An LLM call can return a 200 in 800ms and still be completely wrong, because the output is non-deterministic and there's no single correct answer to diff against. LLM observability adds the content dimension: you capture the full prompt, the full response, the retrieved context, the tool calls, and tokens and cost, so you can judge quality after the fact, not just uptime.

What should I log for every LLM call? The full request and response (system prompt, messages, and completion), the model and parameters, any retrieved context or tool calls and their results, token counts split into input and output, latency, cost, and a trace ID that ties the whole request together. Log enough that you could replay the call and turn it into an eval case months later without guessing what the inputs were.

What's the difference between online and offline evaluation? Offline evaluation runs a fixed eval set against your app before you ship a change — a controlled experiment where you change one variable and watch the score. Online evaluation scores real production traffic as it happens, using cheap automated signals like schema-valid rate, judge scores on a sample, and user feedback. Offline catches regressions before release; online catches drift and edge cases the offline set never imagined.

How do I catch a regression or drift in an LLM app? Track a quality metric over time, not just at release. Regressions show up when a prompt or model change drops your offline eval score; drift shows up when your online metrics slide even though nothing in your code changed — a model provider updated a version, or your real inputs shifted. Alert on the metric, sample the traces around the drop, and read the actual failing outputs.

How do production traces feed back into my eval set? That's the whole point of logging the full request. When a trace shows a bad output — flagged by a user, a low judge score, or your own review — you turn that exact request into a new eval case with the correct expected behavior written down. Your eval set stops being a static snapshot and becomes a growing record of every real failure, so that specific failure can never silently return.