AI agent observability is the ability to answer, after the fact, exactly what an agent did and why: what prompt it received, which tools it called with which arguments, what came back, and what it cost. Most teams discover they need it the first time an agent does something wrong in production and the logs show nothing but a successful HTTP 200. This post describes the observability layer actually running inside my site's agent engine — a self-hosted ring buffer trace store — and when it stops being enough.
Why agents fail silently
Traditional monitoring answers "is the service up?" Agent failures live in a different gap: the gap between "the API call succeeded" and "the agent did the right thing". An agent can return a fluent, confident, wrong answer with every HTTP status green. It can skip a tool call it should have made, call the right tool with the wrong arguments, or reason correctly over context that was silently truncated. None of that shows up in uptime dashboards or error rates. If you only log errors, a misbehaving agent is indistinguishable from a healthy one — which is why the first production incident usually arrives as a confused customer, not an alert.
What to actually trace
A useful trace captures six things per request, tied together by a request ID. The exact prompt and context sent — not the template, the rendered final text, because context assembly bugs are invisible otherwise. Every tool call: name, arguments, result, and whether it was retried. Token counts in and out, because cost regressions are behaviour regressions. Latency per step, so you know whether the model or a tool is the bottleneck. The final output as delivered to the user. And the version of the prompt that ran, because "which prompt produced this?" is the first question in every postmortem. Miss any one of these and some class of bug becomes unreconstructable.
The ring buffer pattern
The trace store in my agent engine is an in-memory ring buffer: a fixed-size circular array where each completed request appends its trace and the oldest trace is overwritten when the buffer is full. No database, no disk writes, no external service, no per-event billing. Writing to it is an in-process memory operation, so it adds effectively zero latency to the request path — which matters, because an observability layer that slows the agent down gets turned off.
Bounded memory is the right shape for this, and not just as a cost dodge. An unbounded log file grows until it becomes a disk-space incident of its own, and 99% of traces are never read. Debugging almost always concerns the recent past — the last few hundred requests — and a ring buffer keeps exactly that window hot and queryable. The honest tradeoff: restart the process and history is gone. For a small production system that tradeoff is fine; you are trading durable history you rarely needed for zero cost and zero operational surface.
ONE TACTIC A WEEK
Tracing tool calls specifically
The tool-call layer is where the expensive bugs hide, because tools are where agents touch the world. The three failure shapes worth designing the trace view around: wrong arguments (the agent called the CRM update with the wrong record ID), skipped calls (it answered from its own head instead of checking the data source), and silent retries (a flaky tool succeeded on attempt three, masking a reliability problem that will eventually not recover). A trace view that renders tool calls as a sequence — name, arguments, result, duration, retry count — turns each of these from an hour of guesswork into a ten-second read. When an agent gives a wrong answer, my first look is never the model output; it is the tool-call sequence that fed it.
Connecting traces to prompt versions
Every trace records the identifier of the prompt version that produced it. This sounds like bookkeeping until the first bad-output report arrives after a week of prompt iteration, and the difference between answering "which version caused this" instantly versus reconstructing it from git log and deploy timestamps is the difference between a fix and an argument. Prompt versioning is its own discipline — I cover it separately — but the observability half of it is simple: the version ID travels with the request, into the trace, and out to any eval or alert that fires on it.
When to graduate to a hosted platform
A ring buffer stops being enough at three thresholds. Team size: once several people debug the same system, traces need to be shareable and durable, not trapped in one process's memory. Compliance: if you must prove what an agent did weeks later, in-memory history is disqualifying by design. And cross-service tracing: when a request spans multiple services and queues, you want distributed tracing infrastructure rather than a hand-rolled correlation scheme. Below those thresholds, hosted platforms mostly sell you a nicer viewer for data you could keep yourself; above them, they earn their price. The mistake is adopting one on day one because a vendor blog said observability requires it.
Turning traces into alerts
Logging for debugging and logging that pages someone are different disciplines. A trace store answers questions you ask; an alert asks the question for you. The promotion path: pick the few trace-derived signals that indicate real user harm — tool-call failure rate, latency past a hard threshold, token cost per request drifting above budget, eval-score drift on sampled outputs — and page on those only. Alert on everything and the channel gets muted within a month, which is worse than no alerts because it feels like coverage. The ring buffer plus three or four well-chosen alerts covers a single-operator production system honestly; everything beyond that should be pulled by a threshold you actually hit, not pushed by a vendor's feature list.