Infrastructure
Azure Monitor Action Groups: diagnose missing notifications before changing alerts
A production runbook for qualifying an Azure Monitor alert that fired but did not notify anyone, with action groups, receivers, processing rules, webhooks, evidence, validation and rollback.
An Azure Monitor alert can fire correctly and still fail the operation. The condition is met, the alert instance exists, the resource is unhealthy, but no one receives the email, webhook, SMS, voice call, ITSM ticket or automation trigger that should start the response. Under pressure, the usual reflex is to edit the alert rule, lower the threshold or add another receiver. That can create duplicate notifications without proving why the original path failed.
The use case is a production alert that should notify an on-call team through an Action Group. The runbook goal is to decide whether the alert rule is healthy, the Action Group is misconfigured, a receiver path failed, an alert processing rule suppressed delivery, an integration rejected the payload, or rollback is needed to the last known working notification path.
Freeze the expected notification contract
Start with the contract, not the portal screenshot. An alert notification path is an operational interface: condition, scope, action group, receiver, integration, escalation and acknowledgement.
Incident
Alert rule: prod-api-high-error-rate
Resource scope: rg-prod-app / app-prod-api
Fired time: 2026-07-04T05:42:00Z
Expected action group: ag-platform-oncall-prod
Expected receivers: email, webhook, ITSM or automation
Expected owner: platform-oncall
Expected escalation window: 5 minutes
Last known good notification
Questions before changing the alert
Did the alert rule really fire?
Which action group version was attached at fire time?
Was any alert processing rule active?
Which receiver should have delivered first?
Is the integration endpoint healthy and authenticated?
Is rollback available to a known working receiver?If the team cannot name the expected receiver and escalation path, editing the rule will only add uncertainty.
Prove the alert fired once
Separate the detection signal from the delivery signal. First prove the alert instance, its state, severity, target and action group reference.
SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
ALERT_RULE="prod-api-high-error-rate"
RG="rg-monitoring-prod"
az account set --subscription "$SUBSCRIPTION"
az monitor metrics alert show --resource-group "$RG" --name "$ALERT_RULE" --query "{enabled:enabled,severity:severity,scopes:scopes,actions:actions,criteria:criteria}" --output json
az monitor activity-log alert list --resource-group "$RG" --query "[?name=='$ALERT_RULE'].{enabled:enabled,scopes:scopes,actions:actions}" --output jsonUse the command that matches the alert type. Metric alerts, scheduled query rules, activity log alerts and smart detection alerts do not expose the same fields. The goal is the same: prove that the firing rule referenced the expected Action Group when it fired.
Check Action Group membership and drift
Action Groups drift like any other production configuration. A receiver can be disabled, renamed, moved to another group, changed from common alert schema to a custom payload, or updated without anyone testing the full path.
RG="rg-monitoring-prod"
ACTION_GROUP="ag-platform-oncall-prod"
az monitor action-group show --resource-group "$RG" --name "$ACTION_GROUP" --query "{enabled:enabled,groupShortName:groupShortName,emailReceivers:emailReceivers,webhookReceivers:webhookReceivers,azureFunctionReceivers:azureFunctionReceivers,logicAppReceivers:logicAppReceivers,armRoleReceivers:armRoleReceivers}" --output json
az monitor action-group list --resource-group "$RG" --query "[].{name:name,enabled:enabled,receivers:length(emailReceivers)+length(webhookReceivers)+length(azureFunctionReceivers)+length(logicAppReceivers)}" --output tableBlock blind edits when the Action Group was changed near the incident window. Restore or test the last known working receiver before adding more destinations.
Look for alert processing rules and suppression
A missing notification is not always a broken Action Group. Alert processing rules can suppress or route notifications during maintenance, deployments, incidents or noisy periods. Those rules are useful, but they must be visible in the decision.
Check suppression and routing
Alert processing rules active during the incident
Scope overlap with the affected resource
Severity filters
Monitor service filters
Schedule timezone and recurrence
Action group override rules
Maintenance window or deployment freeze
Block changes when
A suppression rule matches but has no owner
The schedule timezone is ambiguous
The rule routes to a different action group without evidence
The maintenance window ended but suppression stayed enabled
The alert rule is edited before suppression is understoodSuppression is not a failure when it is intentional, scoped and documented. It becomes an incident when nobody can explain why the notification was muted.
Correlate alert, action group and receiver evidence
The useful evidence chain is: rule fired, action group selected, receiver called, integration accepted or rejected, on-call workflow created or failed. Do not stop at the first green status.
let StartTime = datetime(2026-07-04T05:35:00Z);
let EndTime = datetime(2026-07-04T06:10:00Z);
AzureActivity
| where TimeGenerated between (StartTime .. EndTime)
| where OperationNameValue has_any ("alert", "actionGroups", "metricAlerts", "scheduledQueryRules")
| project TimeGenerated,
OperationNameValue,
ActivityStatusValue,
Caller,
ResourceGroup,
ResourceId,
CorrelationId,
Properties
| order by TimeGenerated ascIf the receiver is a webhook, Logic App, Function, ITSM connector or internal API, correlate the backend logs with the same window. A 200 from Azure Monitor is not enough if the downstream system rejects the payload or cannot authenticate the request.
Validate receiver-specific failure modes
Each receiver fails differently. Treat the receiver as a production dependency instead of a passive notification address.
Email receiver
Address still owned by the on-call group
Mailbox, distribution list and spam policy are healthy
Azure Monitor confirmation state is valid when required
Webhook receiver
Endpoint reachable from Azure Monitor
Authentication secret or header still valid
Common alert schema expected by the receiver
Payload size and rate accepted
Backend logs include correlation ID
Logic App or Function receiver
Trigger enabled
Managed identity or connection still valid
Downstream ticketing or chat connector healthy
Run history shows success or failure
ITSM or incident tool
Integration token valid
Routing key still maps to the right service
Deduplication did not merge into a closed incident
Escalation policy activeThe right fix may be outside Azure Monitor. If the webhook secret expired, lowering the alert threshold does not improve detection or response.
Run a controlled notification test
Test the notification path with a controlled signal, not by weakening the production alert. Use a temporary test rule or a documented test payload, and keep the test tied to the same Action Group and receiver path.
test_plan:
purpose: prove_action_group_delivery
target_action_group: ag-platform-oncall-prod
receiver_under_test: webhook_primary_incident_tool
method:
- create_temporary_test_alert_or_use_safe_test_payload
- include correlation_id in payload or incident note
- keep severity and receiver path close to production
- verify downstream incident creation and acknowledgement
success:
- alert instance created
- action group invoked
- receiver backend accepted payload
- on-call workflow visible
- test artifact closed with evidence
cleanup:
- remove temporary test rule
- revert test routing if changed
- attach result to incident recordA test that only sends an email to an engineer does not validate a webhook-based on-call path. Match the path that failed.
Decide fix, rollback or rule change
Keep the decision explicit. The failure may require an Action Group correction, receiver repair, suppression rollback, integration rollback or, only after evidence, an alert rule change.
Fix Action Group
The alert fired and referenced the expected group
Receiver is missing, disabled or drifted
Last known working receiver can be restored
Controlled test validates delivery
Repair receiver or integration
Azure Monitor attempted delivery
Backend rejected authentication, schema or routing key
Integration logs prove the rejection
Test payload passes after repair
Rollback suppression or routing
Alert processing rule matched unexpectedly
Maintenance schedule or filter is wrong
Owner confirms notification should have been sent
Rollback restores the expected action group
Change the alert rule only when
The condition did not fire for the real symptom
The scope or dimension is wrong
Severity or action group attachment is incorrect
Receiver path has already been proven healthy
Block changes
No one can prove which receiver should have fired
Downstream logs are unavailable
Multiple action groups were edited during the incident
A test would notify production without approvalThe order matters. Fixing delivery before changing detection keeps the signal trustworthy.
Keep a rollbackable notification path
Notification changes need a rollback path just like deployment changes. A good rollback is not a second noisy Action Group forever. It is a known working receiver that can be restored while the primary integration is repaired.
Rollback path
Restore last known working Action Group version
Re-enable previous receiver only for affected severity and scope
Keep temporary fallback owner and expiry time
Validate one controlled notification
Record why the primary receiver failed
Remove fallback after primary path passes test
Rollback is incomplete when
The fallback stays permanent without owner
Duplicate notifications are expected as normal
Suppression remains unexplained
The incident tool receives alerts but escalation does not happen
The alert rule was weakened to compensate for delivery failureA fallback that never expires becomes another source of drift. Give it an owner and a removal condition.
Conclusion
A missing Azure Monitor notification is not solved by adding more noise. It is solved by proving the chain: alert fired, Action Group selected, suppression evaluated, receiver invoked, integration accepted, on-call workflow visible.
The safe decision is to repair the failing link or rollback to a known working notification path before changing the alert rule. Detection and delivery are separate responsibilities; keeping them separate is what makes the next incident diagnosable.