Azure Monitor: validate an SLO burn-rate alert before paging on-call

An alert on an instantaneous error rate handles two common situations poorly. It can wake on-call for a two-minute spike with no lasting impact, or remain quiet during a moderate degradation that consumes a week’s error budget in a few hours. An SLO burn-rate alert supports a more useful decision: is the service spending its error budget fast enough to require action now?

The use case is a user-facing Azure API instrumented with Application Insights and Log Analytics. Its availability SLO is defined over eligible requests. The team wants fast degradation routed to on-call while slower consumption enters a service backlog. This runbook validates the calculation, data, windows, routing and return path before a production notification is enabled.

Write the SLO contract before the query

A burn rate is not a threshold selected in a dashboard. It is the observed error rate divided by the error rate allowed by the SLO. With a 99.9% objective, the error budget is 0.1%. A burn rate of 1 spends that budget at the planned rate; a burn rate of 14.4 spends it 14.4 times faster.

Start by fixing what belongs in the calculation. Health checks, synthetic requests, expected client errors and maintenance traffic must not move in and out of scope during an incident.

service: orders-api
environment: production
objective: 99.9
period: 30d
eligible_events:
table: AppRequests
filter:
  - AppRoleName == orders-api
  - SyntheticSource == empty
good_event:
- Success == true
- ResultCode < 500
excluded:
- route == /health
- planned_maintenance == true
owners:
service: team-orders
telemetry: team-platform
alert_routes:
fast_burn: on-call
slow_burn: service-backlog
rollback:
restore_rule_version: slo-orders-v1
keep_dashboard_query: true

The contract must also name the SLI source. Mixing a platform metric, application logs and a synthetic check in the same denominator creates a number that is hard to explain and nearly impossible to replay.

Calculate the SLI and burn rate on a stable basis

The following query calculates eligible volume, bad events and burn rate. Column names must match the actual schema; the shape of the calculation should stay identical across the dashboard, alert and replay.

let Objective = 0.999;
let AllowedErrorRate = 1.0 - Objective;
let EvaluationWindow = 1h;
AppRequests
| where TimeGenerated > ago(EvaluationWindow)
| where AppRoleName == "orders-api"
| where isempty(SyntheticSource)
| where Name != "GET /health"
| extend IsBad = iff(Success == false or toint(ResultCode) >= 500, 1, 0)
| summarize Eligible=count(), Bad=sum(IsBad)
| extend ErrorRate = todouble(Bad) / todouble(Eligible)
| extend BurnRate = ErrorRate / AllowedErrorRate
| project Eligible, Bad, ErrorRate, BurnRate

Handle low traffic explicitly. One failure out of two requests creates a dramatic burn rate without providing the same evidence as one thousand failures. Add a minimum volume or a second condition on bad-event count. Document the threshold against the service’s real traffic profile instead of copying it from a generic example.

Combine a short and a long window

An actionable burn-rate alert normally needs two windows. The short window confirms that degradation is still active. The long window prevents paging on a spike that has already ended. Both conditions must be true for the urgent route.

let Objective = 0.999;
let AllowedErrorRate = 1.0 - Objective;
let MinimumEligible = 100;
let BurnRate = (Window:timespan) {
AppRequests
| where TimeGenerated > ago(Window)
| where AppRoleName == "orders-api"
| where isempty(SyntheticSource)
| where Name != "GET /health"
| extend IsBad = iff(Success == false or toint(ResultCode) >= 500, 1, 0)
| summarize Eligible=count(), Bad=sum(IsBad)
| extend Rate=iff(Eligible >= MinimumEligible,
    (todouble(Bad) / todouble(Eligible)) / AllowedErrorRate,
    real(null))
| project Rate, Eligible, Bad
};
let Short = toscalar(BurnRate(5m) | project Rate);
let Long = toscalar(BurnRate(1h) | project Rate);
print ShortBurnRate=Short, LongBurnRate=Long
| extend ShouldPage = ShortBurnRate >= 14.4 and LongBurnRate >= 14.4

Window and threshold values are a starting point to validate against the budget, traffic and expected response time. A second rule can use longer windows for slower consumption and create a ticket without calling on-call. The split keeps the semantics clear: immediate operational urgency on one side, a drift to correct on the other.

Prove telemetry quality before enabling paging

An SLO alert is only as reliable as its denominator. Before activation, check ingestion delay, collection gaps, sampling and route-name changes. A sudden drop in eligible traffic can hide a real degradation or amplify a few failures.

let Window = 24h;
AppRequests
| where TimeGenerated > ago(Window)
| where AppRoleName == "orders-api"
| summarize
  Eligible=count(),
  Failed=countif(Success == false or toint(ResultCode) >= 500),
  DistinctOperations=dcount(Name),
  P95IngestionDelay=percentile(ingestion_time() - TimeGenerated, 95)
by bin(TimeGenerated, 15m)
| order by TimeGenerated asc

Compare this series with an independent signal: gateway request volume, an App Service request metric, a synthetic test or a business counter. The purpose is not to merge every signal into the SLI. It is to detect when the chosen source becomes incomplete.

Replay known incidents before the first page

A rule should not move directly from a notebook to on-call. Replay it over at least three windows: an incident that should have paged, a short spike that should not have paged, and a healthy period with typical traffic. Keep the volumes, burn rates and expected decision for each window.

Case A - sustained production failure
Expected: short and long windows breach
Route: on-call
Evidence: user impact and eligible volume are sufficient

Case B - two-minute deployment spike
Expected: short window may breach, long window stays below threshold
Route: no page, deployment event retained

Case C - slow weekly budget consumption
Expected: fast-burn rule stays quiet, slow-burn rule opens follow-up
Route: service backlog

Case D - telemetry gap
Expected: SLO page is blocked or marked unreliable
Route: observability incident, not application rollback

Replay must use exactly the same filters as the deployed rule. A simplified copy in a workbook can provide false confidence if the Scheduled Query Rule applies another scope or aggregation.

Deploy the rule without losing control

Deploy the rule disabled or connected to a non-urgent action group first. Capture its configuration, identity, scope, frequency and actions before changing notification routing.

RG="rg-observability-prod"
RULE="slo-orders-fast-burn"

az monitor scheduled-query show --resource-group "$RG" --name "$RULE" --query "{enabled:enabled,scopes:scopes,evaluationFrequency:evaluationFrequency,windowSize:windowSize,severity:severity,criteria:criteria,actions:actions}" --output json

# Validation gate before paging:
# - query replayed on known windows
# - minimum traffic condition present
# - ingestion latency below the shortest window
# - on-call action group tested independently
# - previous rule definition retained for rollback

Enable notification in stages. A shadow period lets the team compare decisions from the new rule with incidents and existing alerts. Moving to paging must be a dated decision with an owner and an initial review window.

Decide validation, correction or rollback

Closure is not the observation that the rule is green. It must produce an explicit decision.

validate:
when:
  - known_incident_pages
  - transient_spike_does_not_page
  - eligible_volume_is_stable
  - action_group_delivery_is_proven
action:
  - enable_on_call_route
  - review_after_first_budget_event

correct:
when:
  - filters_or_denominator_are_wrong
  - low_traffic_creates_false_burn
  - ingestion_delay_exceeds_short_window
action:
  - keep_shadow_mode
  - fix_sli_or_collection
  - replay_validation_matrix

rollback:
when:
  - paging_is_noisy_or_unexplainable
  - rule_misses_known_sustained_failure
  - telemetry_change_breaks_denominator
action:
  - detach_on_call_action_group
  - restore_previous_rule_version
  - keep_dashboard_query_for_evidence
  - open_follow_up_with_failed_case

The priority rollback is to remove the on-call route, not delete the query and its evidence. The team keeps the signal in observation, corrects the calculation and replays the same cases before another activation.

Conclusion

An SLO burn-rate alert is useful only when it turns an error budget into an operational decision. The calculation must start from a stable SLI, apply windows consistent with traffic, verify data quality and separate fast consumption from slow drift.

The runbook ends with evidence: the sustained incident pages, the transient spike does not, low traffic is bounded, and the action group actually delivers. If any condition fails, the rule remains in shadow mode or returns to its previous version. Paging is enabled only when the behavior is explainable and replayable.