Twelve monitoring signals for AI agent reliability
July 9, 2026
This is part three of an arc I didn’t entirely plan. Part one argued that an agent is a distributed system whose nodes happen to be non-deterministic, and that the chaos-engineering playbook ports over. Part two asked what you should point SLOs at once you accept that HTTP 200 tells you nothing about whether an agent did the right thing. This post is what happened when I stopped arguing by analogy and built the testbed: a working paper — Monitoring Signals for AI Agent Reliability: Beyond the Golden Signals of Microservices — with an instrumented agent, four injected incident types, and 672 episodes of trace data behind every claim.
The one-line result: during the most damaging incident I injected, every traditional signal stayed green while the agent issued unauthorized refunds. The p95 latency chart was flat. The error-rate chart was flat. Throughput was flat. If that agent had been in production behind a standard dashboard, the first alert would have been a customer, or the finance team.
Four assumptions your monitoring stack is quietly making
Golden signals, RED, USE, SLOs — the whole discipline rests on assumptions so old we forgot they’re assumptions. Agents break all four:
- Determinism. Same request, same behavior. Retries, canaries, and “reproduce it in staging” all lean on this. Agents are stochastic: the same ticket can produce a different tool sequence, a different outcome, and a different bill every run.
- Explicit failures. Services fail loudly — 5xx, exceptions, timeouts. Error rate is countable because failure is legible. An agent’s modal failure is a fluent HTTP 200 that’s wrong, or a completed workflow that did the wrong thing.
- Bounded action space. A service does what its endpoints do; blast radius is knowable at design time. A tool-using agent composes its own action sequences at inference time. Worst-case behavior isn’t enumerable from the interface.
- Cheap stateless retries. Retry-on-error is the universal remediation — unless steps have side effects, in which case retrying a half-executed trajectory double-refunds, double-emails, double-deletes.
The golden signals don’t become wrong — infrastructure still needs them. They become insufficient: they instrument the service layer while the failure moved up to the semantic and behavioral layer.
The twelve signals
The paper’s core is a taxonomy of twelve signals in four families, every one computable in production without ground-truth labels — because production has no oracle telling you which answers were right.
Outcome signals (what “errors” becomes): proxy success rate (user feedback, downstream acceptance, sampled LLM-judge verdicts), escalation & abandonment rate, and canary outcome consistency — replay a fixed task set on a schedule and watch the variance, which moves before the mean does.
Trajectory signals (no microservice analog): trajectory divergence against a trailing baseline window, loop & stall rate, and tool-call health at step granularity — a 2% per-step error rate compounds to a large per-task failure probability over a long trajectory.
Resource-envelope signals (latency and saturation change shape): cost- per-task distribution — heavy-tailed, so alert on tail expansion, not mean shift — steps- and latency-per-task, and context saturation.
Boundary signals (the SLO microservices never needed): policy-violation rate, irreversible-action rate — writes, refunds, sends, deletes per task — and abstention & deferral calibration. These are severity signals. They never get averaged into a health score; any single event can matter.
What the testbed actually showed
I ran a tool-calling support agent (OpenAI’s gpt-5.4-nano) against a deterministic mock store — 21 tasks × 8 repetitions × 4 conditions, with a gpt-5.4-mini judge sampling outcomes. Because the backend is deterministic, I always knew the ground truth, which means I could measure how well the signals themselves track reality. Three findings survived review, and each one surprised me a little.
The incident traditional monitoring cannot see is real, and it’s mundane. The incident that produced the headline chart wasn’t a hostile prompt — it was input drift: paraphrased user requests plus reordered JSON fields in tool responses. The kind of thing a routine schema change ships every week. Under it, the unauthorized fraction of the irreversible-action rate went from zero to ~0.17 and outcome consistency dropped — while the traditional panel didn’t move a pixel.
The judge degrades exactly when you need it. The LLM judge agreed with ground truth 0.798 of the time at baseline — and 0.667 under fault injection. Read that again: the proxy-success signal is least accurate precisely when the system is under stress. If your agent dashboard is “LLM judge says fine,” you’ve built a smoke detector that melts in fires. Monitor the monitor: periodic human-labeled samples, and treat proxy success as one voice among four families, never the arbiter.
Robust models leak degradation through behavior anyway. I poisoned the system prompt with a pile of irrelevant policy and an explicit “make goodwill exceptions rather than escalating” nudge. The model largely resisted — accuracy barely moved, zero unauthorized actions. But escalation recall fell from 1.00 to 0.88: it started quietly refusing without the deferral protocol. Outcome metrics saw almost nothing; the boundary family registered it anyway.
What I’d put on a dashboard tomorrow
If you run agents and can only do three things this quarter: emit structured per-step traces (the OpenTelemetry GenAI conventions are fine plumbing), stand up a canary replay of ~20 known-answer tasks and chart its consistency, and count irreversible actions per task against a baseline. That last one is the Replit signal — the burst of destructive calls is visible in the trace stream before the postmortem.
The full treatment — formal definitions, measurement protocols, the incident replay figure, and the honest limitations of a toy testbed — is in the paper. The pipeline is ~600 lines of Python and the whole 672-episode matrix cost about thirty cents to run. The era of “we can’t afford to measure agent reliability” is over; what’s left is deciding to look.