Infrastructure
Azure Monitor: diagnose an Action Group before silencing alerts
A production runbook for qualifying an Azure Monitor Action Group by separating rules, receivers, webhooks, escalation paths, KQL evidence, validation and rollback before reducing notifications.
When a team receives too many alerts, the fastest reflex is often to disable the Action Group, remove a receiver or cut the webhook that wakes up on-call. That may feel reasonable during an incident, but it can also remove the only signal the team still knows how to use.
The use case is an Azure platform where several Azure Monitor rules share one Action Group: email, Teams, ITSM webhook, Logic App, automation or incident connector. A wave of alerts starts. Some notifications stop leaving Azure, others duplicate the same symptom, and the team can no longer tell whether the fault is the rule, the channel, the escalation route or the monitored service. The runbook goal is to decide what to change, what to preserve and how to roll back without making the incident invisible.
Freeze the notification contract
Before touching the Action Group, write down what it is supposed to do. An Action Group is not only a list of email addresses. It is an operations contract: which rules call it, which channels are critical, who receives what, which webhooks create incidents, and which notifications may be slowed down temporarily.
action_group:
name: ag-prod-critical
scope: production platform alerts
owner: platform-operations
critical_channels:
- type: webhook
target: incident-management
purpose: create_or_update_incident
- type: sms_or_voice
target: on-call-primary
purpose: wake_up_for_sev1
secondary_channels:
- type: email
target: platform-watch
purpose: context_and_followup
- type: teams
target: operations-channel
purpose: shared_visibility
change_rules:
require_ticket: true
maximum_silence: 30m
preserve_one_user_impact_signal: true
rollback_owner: incident-commanderIf this contract does not exist, rebuild it from the actual configuration first. Otherwise, the change will be judged from pressure in the moment: “too noisy”, “not enough alerts”, “the webhook seems broken”.
Map the rules that call the group
A noisy Action Group is not always the problem. It may simply relay a rule that is too broad, a poorly chosen dimension or a threshold that became obsolete after a traffic change. List the rules using it before modifying the group itself.
ACTION_GROUP_ID="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-monitoring/providers/microsoft.insights/actionGroups/ag-prod-critical"
az monitor metrics alert list --query "[?contains(to_string(actions), '$ACTION_GROUP_ID')].{name:name,resourceGroup:resourceGroup,enabled:enabled,severity:severity,scopes:scopes}" --output table
az monitor scheduled-query list --query "[?contains(to_string(actions), '$ACTION_GROUP_ID')].{name:name,resourceGroup:resourceGroup,enabled:enabled,severity:severity,scopes:scopes}" --output tableThe output should answer three questions: which rules are actually connected, which ones are critical to user impact, and which ones can be adjusted without hiding a production incident.
Separate channel failure from noisy rules
Do not confuse a broken channel with a noisy rule. If the ITSM webhook no longer creates tickets, silencing the rule does not fix the integration. If the rule emits a thousand valid but unusable events, changing the webhook does not reduce noise.
Symptom: many notifications, coherent tickets
Main hypothesis: broad rule, noisy dimension or obsolete threshold
Candidate action: adjust the rule, not the whole Action Group
Symptom: alerts visible in Azure Monitor, no external notification
Main hypothesis: channel, webhook, disabled receiver or integration error
Candidate action: test the receiver and keep the rule active
Symptom: duplicate notifications across channels
Main hypothesis: several rules or several receivers for the same symptom
Candidate action: deduplicate escalation, do not delete the signal
Symptom: notification arrives late after recovery
Main hypothesis: evaluation window, frequency, webhook latency or ITSM backlog
Candidate action: qualify latency before changing severityThis matrix prevents a common mistake: disabling the entire group when only one rule or one receiver is failing.
Read alert evidence before silence
Azure Monitor should show which rules fired, when, on which resources and with which severity. Keep that evidence before any silence, even a temporary one.
let StartTime = datetime(2026-07-17T08:00:00Z);
let EndTime = datetime(2026-07-17T09:00:00Z);
AlertsManagementResources
| where todatetime(properties.essentials.startDateTime) between (StartTime .. EndTime)
| extend alertRule = tostring(properties.essentials.alertRule)
| extend severity = tostring(properties.essentials.severity)
| extend monitorCondition = tostring(properties.essentials.monitorCondition)
| extend target = tostring(properties.essentials.targetResource)
| extend targetType = tostring(properties.essentials.targetResourceType)
| summarize fired=count(),
firstSeen=min(todatetime(properties.essentials.startDateTime)),
lastSeen=max(todatetime(properties.essentials.startDateTime)),
targets=dcount(target),
sampleTargets=make_set(target, 5)
by alertRule, severity, monitorCondition, targetType
| order by fired descThe query does not prove that the Action Group notified anyone, but it proves the pressure created by the rules. That is the baseline for deciding whether the reduction belongs on the rule, the receiver, the escalation channel or the monitored service.
Test each receiver as a production dependency
A webhook or Logic App receiver is a dependency. It can have its own secret, expected IP range, throttling behavior, payload format and failure modes. Treat it as a production integration, not as a configuration field.
Check each receiver
Receiver is enabled in the Action Group
Name and destination match the expected owner
Last configuration change is known
Secret, token or endpoint is still valid
Webhook or incident connector response is observable
Azure alert correlates with the external ticket
Duplicate and resolved alerts behave as expected
Block global disablement when
Only one receiver is failing
Critical alerts remain correct
The external integration throttles but eventually accepts
No return test has been preparedFor a webhook, replay with a test alert or a non-critical rule. For a human channel, confirm real reception with the on-call person, not only the “enabled” state in Azure.
Reduce noise without muting the incident
A good reduction is bounded, documented and reversible. It keeps at least one user-impact signal and avoids changing several layers at once.
silence_plan:
incident: INC-20260717-014
action_group: ag-prod-critical
reason: notification storm from one non-user-impact rule
start: 2026-07-17T08:40:00Z
end: 2026-07-17T09:10:00Z
changes:
- target: rule
name: queue-depth-warning
action: disable_notification_only_or_snooze
keep_metrics: true
- target: action_group
name: ag-prod-critical
action: keep_critical_webhook_and_oncall_enabled
preserved_signals:
- api_5xx_rate_sev2
- customer_login_failure_sev1
rollback:
owner: incident-commander
condition: end_time_reached_or_user_impact_confirmed
expected_state: all_critical_receivers_enabledIf the tooling cannot cleanly snooze a receiver or a rule, keep the previous state in the incident ticket and prepare the return command before applying the change.
Validate signal return
The end of a silence window is a production test. Do not only re-enable the configuration. Verify that the signal reaches the right destination, at the right level, without duplication.
Validation after return
The critical rule still fires in Azure Monitor
The Action Group calls the expected receiver
The webhook creates or updates the right incident
The human channel receives one actionable notification
Resolved notifications do not reopen a closed incident
The ticket contains evidence before, change, evidence after and rollbackRun this validation with a controlled signal or a non-destructive alert. The goal is not to create an outage. The goal is to prove that the notification chain is operable again.
Decide: fix the rule, repair the receiver or roll back
The final decision should name the responsible layer.
Fix the rule
Too many alerts come from a poorly calibrated query, dimension or severity
Receivers behave correctly
A user-impact signal remains available
Repair the receiver
Azure Monitor fires correctly
An external channel does not receive or transforms the alert incorrectly
Other receivers remain useful
Roll back the Action Group change
The reduction hides a user-impact symptom
The critical channel no longer receives a test signal
The silence window expires
The team can no longer explain who is notified and whyAn Action Group should remain an operating interface, not a panic switch. The useful end state is an explainable alerting chain: a rule adjusted if it was noisy, a receiver repaired if it was broken, or a quick rollback if the reduction made production less observable.