Statistical anomaly detection beats LLM-judged evals for production monitoring
Everyone reaches for an LLM to judge LLM outputs. For catching production regressions, a rolling Z-score over sixteen cheap numeric signals caught more real incidents, faster, for a fraction of the cost.
Sentinel started with the fashionable design: sample production traffic, have a judge model grade responses, alert on grade drops. It worked in the demo and fell apart in production — judge costs scaled with traffic, judgments drifted with the judge's own model updates, and the latency between an incident starting and enough graded samples accumulating was measured in hours.
The replacement is embarrassingly classical: sixteen numeric signals per request (latency percentiles, token counts, refusal-phrase hits, format-validity checks, embedding drift against a golden set) with rolling Z-scores per signal. An incident is a sustained excursion across correlated signals, not one bad sample.
The interesting part is what each approach catches. The judge was better at subtle quality decay — answers that got blander. The Z-scores were better at everything that actually pages someone: silent truncations after a provider update, a prompt-injection wave shifting refusal rates, a tokenizer change inflating costs 3x overnight.
The hybrid that shipped: statistics decide WHEN something is wrong (cheap, fast, deterministic), and a judge model runs a diagnostic pass on the anomalous window to describe WHAT went wrong — Gemini writes the root-cause narrative that lands in the Datadog incident.
Open question: golden-set maintenance. Embedding drift against a fixed reference set slowly goes stale as legitimate usage evolves; refreshing the reference set without laundering real regressions into the new baseline is an unsolved chore.