Azure Monitor: diagnose an alert storm after deployment

An alert storm after deployment puts the team in a poor operating position: on-call channels fill up, dashboards turn red, and the immediate temptation is to disable the action group or raise thresholds. That may sometimes be needed to regain control, but it is not yet a diagnosis.

The use case is an Azure service that has just been deployed or reconfigured: App Service, Functions, APIM, Application Gateway, an automation job, a networking component or an internal workload. A few minutes later, Azure Monitor starts firing at scale. The runbook goal is to separate real incident, monitoring noise, warm-up effect, poorly calibrated threshold, broad dimension or a change that should be rolled back.

Read the storm as a signal chain

An alert is not a raw fact. It is the result of a chain: metric or log, query, dimension, aggregation window, threshold, rule, action group, notification and runbook. When many alerts fire at once, follow that chain before deciding.

Source signal
Azure Monitor metric, Application Insights log, resource diagnostic or custom table
Timestamp, resource, region, environment, dimension

Rule
Query alert or metric alert
Evaluation window
Frequency
Static or dynamic threshold
Dimension or split by

Notification
Action group called
Channel affected: email, Teams, webhook, ITSM, automation
Event count and deduplication

Change context
Application deployment
Infrastructure or network change
Alert rule modification
Traffic or dependency change

Decision
Real incident
Temporary noise
Wrong rule
Application or infrastructure rollback
Controlled notification suspension

This prevents treating every alert as one block of noise. A rule may be noisy because it correctly reports a regression, or because it observes normal startup behavior without enough context.

Stabilize without losing evidence

If the on-call channel becomes unusable, reduce the noise in a controlled way. The trap is cutting the only evidence that will explain the incident.

Immediate goal
Keep one channel readable
Preserve traces
Do not hide user impact

Acceptable actions
Group notifications when the channel supports it
Snooze one rule briefly with a ticket and end time
Move a non-critical notification to a secondary channel
Keep logs and metrics active
Preserve at least one user-symptom alert

Risky actions
Disable the whole action group without a return time
Change thresholds without keeping the previous value
Delete the rule before export or capture
Treat noise as proof that there is no incident

Stabilization must be visible in the incident ticket: touched rule, duration, reason, owner and return condition.

Group alerts by symptom

Start by grouping alerts by symptom rather than by channel. One wave may mix latency, 5xx errors, exceptions, dependency saturation and failed jobs. This grouping shows whether the problem is central or peripheral.

let Window = 2h;
let DeploymentTime = datetime(2026-06-18T07:30:00Z);
AlertsManagementResources
| where todatetime(properties.essentials.startDateTime) > ago(Window)
| extend firedAt = todatetime(properties.essentials.startDateTime)
| extend alertName = tostring(properties.essentials.alertRule)
| extend severity = tostring(properties.essentials.severity)
| extend monitorCondition = tostring(properties.essentials.monitorCondition)
| extend target = tostring(properties.essentials.targetResource)
| extend targetType = tostring(properties.essentials.targetResourceType)
| extend minutesAfterDeployment = datetime_diff('minute', firedAt, DeploymentTime)
| summarize alerts=count(),
          firstSeen=min(firedAt),
          lastSeen=max(firedAt),
          targets=dcount(target),
          sampleTargets=make_set(target, 5)
by alertName, severity, monitorCondition, targetType
| order by alerts desc

The exact table depends on Azure Monitor and Log Analytics configuration. If your alerts are exported differently, keep the same principle: rule name, target, severity, first occurrence, last occurrence and number of affected resources.

Compare with the deployment

An alert after deployment is not always caused by the deployment. It can reveal a slow dependency, a baseline that was too low, or a rule that had never seen that traffic pattern. Compare time, component and change together.

Correlation questions
Did the first alerts start after deployment?
Does the alerting component match the deployed component?
Did a nearby dependency alert before the workload?
Did traffic change at the same time?
Had the threshold already been reached without notification?
Was an alert rule or action group changed?
Does the metric measure a user symptom or an internal signal?

An application rollback is defensible when the signal touches the delivered component, starts inside the change window and is confirmed by logs or user-facing metrics. It is weaker when only internal noisy rules changed state without visible impact.

Check whether the rule watches the right dimension

Alert storms often come from rules that are too broad: split by instance, region, status code, operation, backend or queue. A rule may be correct on one dimension and unmanageable when it notifies separately for every value.

For each noisy rule
Target scope: service, group, subscription or workspace
Dimension: instance, operation, status code, hostname, backend, queue
Split by enabled or not
Number of fired series
Threshold value
Aggregation window
Evaluation frequency
Auto-mitigation or automatic resolution
Action group called

The correction is not always raising the threshold. It may be changing the dimension, reducing scope, grouping notifications or creating two rules: one user-symptom rule and one lower-urgency diagnostic rule.

Read service logs before thresholds

Before changing an alert, verify whether the service really reports a regression. The example below starts from Application Insights, but the same logic applies to APIM, Application Gateway, Functions, Container Apps or an internal component.

let Window = 2h;
let DeploymentTime = datetime(2026-06-18T07:30:00Z);
requests
| where timestamp > ago(Window)
| summarize total=count(),
          failed=countif(success == false),
          p95=percentile(duration, 95)
by bin(timestamp, 5m), cloud_RoleName
| extend failureRate = todouble(failed) / todouble(total)
| extend period = iff(timestamp < DeploymentTime, "before", "after")
| order by timestamp asc

More alerts without more errors, latency or business impact points to the rule. A rise aligned with user errors points to the service. A rise only on a dependency points to the call path, not necessarily to the deployed code.

Decide between fix, temporary silence and rollback

Keep the decision explicit. Changing an alert rule is not a service rollback. Rolling back the service is not a monitoring correction. Both may be needed, but they answer different evidence.

Proven finding
User impact confirmed after deployment
  Decision: application or infrastructure rollback depending on the change
  Validation: errors, latency or user symptom return to expected level

Rule too sensitive without service impact
  Decision: adjust threshold, window or dimension while keeping history
  Validation: the rule can still detect the useful symptom

Action group or deduplication misconfigured
  Decision: fix notification routing or grouping
  Validation: a test produces one readable notification

Expected warm-up or migration
  Decision: bounded temporary silence or documented transitional threshold
  Validation: automatic return and post-deployment review

Unknown cause but channel saturated
  Decision: stabilize notifications without stopping collection
  Validation: evidence remains available and one user-symptom alert is kept

The right decision is the one the team can read the next day without guessing why it was taken.

Keep monitoring rollbackable

An alert modification must be rollbackable like a production change. Otherwise the platform keeps modified thresholds for weeks and the next incident arrives without reliable detection.

Before change
Export or capture the previous rule
Record threshold, window, dimensions and action group
Link the change to the incident ticket
Set a review time

After change
Test a controlled notification when possible
Verify logs are still arriving
Confirm the rule still detects the target symptom
Restore the old threshold if the change was only transitional
Add the scenario to the deployment runbook

This monitoring rollback matters as much as the application rollback. Without it, the system may look calm because it no longer listens correctly.

Conclusion

An alert storm after deployment is both an observability incident and a production incident. Stabilize notifications, preserve evidence, group symptoms, compare with the change, review dimensions and read service logs before touching thresholds.

The final decision must be clear: roll back the service when impact is proven, correct the rule when the signal is misconfigured, use temporary silence only when bounded and traced, or fix the action group when notification behavior is the problem. The goal is not silence. It is getting back to alerts that help operate the system.