AI Monitoring Alerts That Actually Page: The Signal-to-Noise Problem in LLM Observability

The Two Alert Failure Modes (and Why Both Are Deadly)

Most teams running LLM systems add some monitoring. That is good. But alert quality breaks in one of two ways. Teams rarely notice which failure mode they are in until a user complaint arrives instead of a page.

Alert fatigue. The on-call queue fills with pages: every latency spike, every token overage, every vendor 429. Ninety-five percent are noise. The engineer paged at 2 AM six nights in a row starts ignoring every page. When a real incident arrives, it looks like the rest. It gets dismissed. The team hears about it from users the next morning.

Silent failure. The team got burned by noisy alerts, so they set very high thresholds. Nothing pages. Latency climbs. Hallucination rate rises. Cost per resolved task doubles. The monitors show green. The vendor dashboard shows green. Users report wrong answers and slow responses. The team has no record of when things started breaking or how long it has been going on.

LLM observability is harder than normal service monitoring for one reason: outputs are not predictable. A 500 error is clear. A wrong answer is not. You cannot detect a bad answer the way you detect an error code. That forces you to be very careful about which metrics really tell you something. If your setup is paging on noise or missing real problems, a production AI engineer can audit your monitoring layer and map it against the controls that fix both failure modes.

Why the Signal-to-Noise Frame Applies to LLM Alerting

Signal-to-noise ratio is a core idea in electrical engineering. It is the ratio of a real signal's power to background noise. A communication system works when that ratio is high enough to detect the signal. When the ratio falls too low, the signal gets lost.

This is not just a metaphor. Seven years of electrical engineering work in Luanda and a BSEE from the University of South Florida put SNR and matched-filter design at the center of how I think about detection problems. Applied to LLM observability, the match is exact. The alert system is the receiver. Production metrics are the channel. Real degradation is the signal. The noise floor is everything else: per-request latency jitter, token-count changes from variable input lengths, and transient vendor throttling that retry logic fixes. These carry no information about real system health. Alerting on them hides the real signal.

Real degradation has a clear shape. It lasts. It shows up across several related metrics at once. It trends rather than spikes. SLO breach means latency stays above the agreed level for a full rolling window, not just one slow request. Error budget burn rate is a trend: the budget is being used up faster than agreed. Cost-per-task increase is a lasting economic shift, not one expensive call.

The design question for your alert system is the same one an EE asks when building a receiver: what is the smallest real signal you can detect above the noise, and what does that signal look like? Every section below answers part of that question.

The Three Signals That Always Deserve a Page

1. SLO Breach: Latency Exceeds the Agreed Threshold for the Critical Path

The most common SLO alerting mistake is picking the wrong metric. Alerting on mean latency looks smooth because fast requests hide slow ones. A sustained slowdown on your critical path can raise mean latency by 40% and still look fine. Alerting on a single slow request just creates noise. Every system has one-off slow calls from cold starts, model provider variation, or network jitter.

The right alert fires on sustained SLO breach. That means p95 or p99 latency over a rolling window (5 minutes or 15 minutes, depending on traffic) compared against the agreed SLO for each critical path. "Per critical path" matters. A global latency SLO hides problems. Your most important user path can be breaking while low-priority tasks run fast, and the global average looks fine.

This connects to the 12-Control AI Incident Readiness Audit: Control 1 says monitor every critical path with a named SLO. A well-run system names SLOs per path, not one global metric. If you have no per-path SLOs, you cannot calibrate alerts. You cannot set a threshold before you know what you are protecting.

See the SLO design and error budgets guide for the full framework for naming and measuring SLOs before you set alert thresholds.

2. Error Budget Burn Rate: Consuming the Budget Faster Than the Agreed Rate

Raw error-count alerts are better than single-request alerts, but they still have a problem. A spike during high traffic looks the same as slow degradation during low traffic, even when the effects are opposite. Error budget burn rate fixes this. It does not just tell you if errors are happening. It tells you whether the error budget will run out before the next SLO window closes at the current rate.

The burn-rate model comes from Google's SRE practices and DORA research. It uses two thresholds. A fast-burn alert fires when the error budget will be gone in less than one hour. Page someone now. A slow-burn alert fires when the budget will be gone in less than 24 hours. Create a ticket for same-day investigation. Fast burn catches sudden failures. Slow burn catches gradual problems that raw error counts would miss until the budget was already empty.

Burn rate alerts fire earlier and more accurately than raw error counts because they measure the rate of change. One failed request in a 30-day window is negligible. Ten failures in 10 minutes on a system with a 99.5% SLO is a fast-burn alert. The distinction is real. The raw error count (10) is the same in both cases.

3. Cost-Per-Task Spike: The Economics of the Pipeline Are Breaking

Cost per API call is noise. It changes with input length, model tier, and batch size. A single long input that costs 20% more than average is not an incident. It is normal variance in a system that handles inputs of different lengths. Alerting on every call above average cost creates high-frequency noise with no useful meaning.

Cost per resolved task is signal. It measures how much it costs to produce one complete, correct output. This metric stays stable under normal conditions because input variance averages out across many requests. When cost per resolved task rises and stays up, something structural is breaking. Maybe the model got more verbose without resolving tasks better. Maybe a retry storm is piling up. Maybe a caching failure is forcing repeated work.

Here is a concrete example. A model update makes the model more verbose. Per-call token cost rises. But the model is also more accurate. Resolution rate improves and retries drop. Cost per resolved task stays flat or gets better. That is not an incident. Now consider the same scenario but resolution rate does not improve. Cost per resolved task rises with flat resolution. That is an incident. The pipeline is spending more without getting better results. This maps to the AI Cost Reality Check's Criterion 1: cost per resolved task, the spend-audit lens that separates real cost signals from per-call noise.

The Three Noise Sources to Filter, Not Page On

4. Per-Request Latency Jitter

Every networked LLM call has a natural latency noise floor. Provider response time varies with server load, routing, and prompt complexity. A single request that takes 3x the mean is not an incident. It is just one sample from the tail of a distribution that always had a tail. Paging on individual slow requests means you are paging on noise, not on a real problem.

The right approach is percentile tracking over a rolling window. The p95 or p99 latency for the last 5 minutes tells you if the tail is growing in a real, lasting way. A single slow request does not. Stop alerting on single-request latency. Alert on rolling-window percentile latency compared to your per-path SLO.

5. Minor Token Count Overage

Token counts change with input structure. A longer document, a complex query, or a long conversation history uses more tokens. A single request that runs 15% over the token budget is not an incident. It is just natural variance in the inputs.

Token count overage becomes a signal only as a trend. If average tokens per request climb week over week with no improvement in quality or resolution rate, that is worth a ticket. It is not worth a real-time page. Track token count as a weekly trend in your calibration loop, not as a real-time alert on individual requests.

6. Temporary Vendor Throttling or Rate-Limit Events

Retry logic exists to handle transient throttling. A 429 response that resolves after exponential backoff and a successful retry is the system working as designed. Paging on every 429 creates noise that hides the real signal: sustained throttling that has used up the retry budget and is causing real failures for users.

The right alert fires on one of two conditions. First: rate limiting that outlasts the retry window, meaning it is structural, not transient. Second: the retry budget is used up and real request failures are now piling up against the error budget. The first is a ticket. The second feeds the burn-rate alert described above.

The Matched-Filter Approach: Alert on Patterns, Not Points

In EE receiver design, a matched filter is a detector tuned to the expected shape of a real signal, not just any crossing of a generic threshold. It maximizes signal-to-noise ratio at the output by comparing the incoming signal against a template. Random noise does not match the template, so it stays small at the output. A real signal matches strongly, so it rises above the noise floor even when buried at the input.

Applied to LLM alerting, the matched-filter equivalent is a multi-condition alert rule. Fire when latency is trending up AND cost per task is rising AND your accuracy proxy (eval coverage or resolution rate) is falling, all at the same time over the same rolling window. The key is correlation across independent metrics. Random noise rarely pushes all three metrics the same way at once. Real degradation (a structural pipeline problem) does. A model behavior change raises latency, raises per-task cost, and drops resolution rate together. That correlated triple-signal fires the alert. A single metric spiking alone does not.

Building this requires multi-condition alert rules in your observability stack. Single-metric threshold rules are easier to set up, which is why most teams use them. But they cause the alert-fatigue problem described at the start of this article. The matched-filter approach takes more work to configure and needs a calibration loop to tune window sizes and thresholds. It reduces false positives because noise cannot easily satisfy all three conditions at once.

The control-theory framing behind this calibration approach is in the control-theory framing that underlies threshold calibration in production AI. The matched-filter model here is the alerting-layer version of those same stability and feedback principles.

You can also check your pipeline's failure-mode visibility before tuning thresholds using the free stability auditor tool. If the auditor finds gaps in failure-mode coverage, those gaps are where to start the calibration work in the next section.

What Good Alert Calibration Looks Like in Practice

The framework above turns into a concrete set of alert rules and a calibration loop. Table 1 classifies each metric. Table 2 is the alert calibration reference you can use directly.

Table 1: Signal vs. Noise Classification for LLM Monitoring

Table 2: Alert Calibration Reference

The calibration loop that keeps this table accurate has three steps. First, measure the false-positive rate every month. Count pages that did not lead to a confirmed incident, divided by total pages. A false-positive rate above 20% means thresholds need tightening. Second, review and adjust thresholds every quarter based on traffic patterns and any model or pipeline changes. Third, run a retrospective after every paged incident. Ask two questions: did the alert fire at the right time, and was the severity right?

This discipline underlies sincllm's own production benchmark of 99% pipeline reliability across 500 or more transcripts on sr-demo-ai.com. That level of reliability needs calibrated alerting, not just more monitors. Uncalibrated alerting produces either fatigue (more monitors, more noise, less trust) or silence (high thresholds, false security). The calibration loop is what makes monitoring actually useful.

NIST AI RMF 1.0's MANAGE function covers ongoing monitoring, incident response, and alerting for deployed AI systems. ISO/IEC 42001:2023's performance monitoring and corrective action sections apply to teams using a formal AI management system. OWASP LLM Top 10 (2025) identifies LLM04 (Model Denial of Service) as the attack class a well-calibrated cost-per-task alert detects early. It also identifies LLM08 (Vector and Embedding Weaknesses) as a drift case the matched-filter correlated alert can catch before it becomes a full incident.

Before calibrating thresholds, confirm every critical path is instrumented. That prerequisite work is in the companion checklist for instrumenting every critical path before calibrating alert thresholds. Calibration without full instrumentation gives you accurate thresholds on incomplete coverage. You can have excellent alert quality on the paths you watch while other paths fail silently.

The companion LLM observability metrics guide maps those paths into joined service, model, retrieval, tool, quality, safety, and product-outcome telemetry.

Does Your Current Alert Setup Have These Problems?

These four questions are a quick check of your current monitoring. Answer each one honestly. "Unsure" is a valid answer and is often the most useful one.

Two or more "Unsure" answers means your monitoring has not been calibrated against the signal types in this article. That is not a criticism of the team that built it. Most LLM monitoring starts from "instrument everything and alert on everything," which is the right posture for a new system. Calibration is the next step. It requires applying the SNR discipline described here.

Conclusion

Alert quality in LLM production systems is an engineering discipline, not a configuration task. The difference matters. Configuration is one-time: pick thresholds, enable rules, done. Engineering is a loop: define the signal you want to detect, measure the noise floor you are working against, tune the detector, measure the false-positive rate, and recalibrate on evidence.

The signal-to-noise problem in LLM observability is solvable. The three signals that always deserve a page (SLO breach, burn-rate exhaustion, cost-per-task spike) are measurable and have clear detection conditions. The three noise sources that should never page alone (per-request jitter, single-token overage, transient throttling) have clear filtering rules. The matched-filter correlated alert cuts false positives by making the alert condition structurally hard for noise to satisfy. The calibration loop keeps the system accurate over time as the model, traffic, and pipeline change.

The same tools electrical engineers use to detect signals in noisy channels apply directly to LLM observability. The discipline is portable. Applying it to your alert setup is the difference between a monitoring layer that pages on real incidents and one that is either noisy or silent.