SYSTEMS · LLM OBSERVABILITY
Sentinel wraps LLM calls as FastAPI middleware, tracks 16+ metrics per request, and turns Z-score anomalies into Datadog incidents with Gemini-written root cause analysis.
↓ SCROLL
Teams shipping LLM features run mostly blind. When costs spike, latency creeps up, or answer quality drops, there is no straightforward way to explain why or to catch it before users do. Token usage, spend, latency, and quality signals sit scattered across logs, so cost surges and regressions surface late and get diagnosed by hand. Sentinel targets that visibility gap for language-model applications: seeing what a model is actually doing, request by request, and being warned the moment its behavior drifts.
Sentinel sits in front of the model as a FastAPI middleware layer that records every interaction. Each request produces 16+ metrics covering tokens, cost, latency, and quality indicators, which stream to Datadog as telemetry. A detector compares each metric against a rolling baseline using Z-scores, and anything past the threshold becomes a Datadog incident with a severity level. Google Gemini then writes a plain-language root cause explanation for that incident, so the alert arrives with a first diagnosis attached instead of just a number to chase.
The /chat endpoint handles the LLM call as middleware, recording 16+ metrics per request (tokens, cost, latency, quality indicators) before the response returns downstream.
The telemetry layer pushes collected metrics to Datadog, building a running record of model behavior across requests.
The anomaly detector maintains a rolling baseline over the last 100 samples and flags any metric whose Z-score exceeds the threshold (default 3.0).
Flagged anomalies become Datadog incidents with a severity classification, so issues escalate automatically instead of waiting to be noticed.
Google Gemini analyzes the flagged incident and generates a plain-language root cause explanation, attaching a first diagnosis to every alert.
Tracks 16+ metrics on every call: token counts, cost, latency, and quality indicators.
Uses Z-score analysis against rolling baselines to catch cost surges, latency spikes, and quality drops as they happen.
Turns detected anomalies into Datadog incidents with severity levels, removing the manual triage step.
Google Gemini writes a readable explanation of why each incident fired, so responders start from a hypothesis.
Ships as a Docker container that runs locally or deploys to Google Cloud Run with Datadog and Google AI credentials.