Azure Monitor and KQL: decide a deployment rollback without silencing alerts

After a deployment, the operational question often arrives too early: should the team roll back or hold the line? Azure Monitor alerts start firing, application logs get noisy, a dashboard turns red, and everyone looks for proof inside a very short window. Silencing the action group may feel useful, but it often destroys the signal needed to decide.

The use case is concrete: an Azure application has just been deployed behind a private path, a gateway, APIM or a managed service. The first alerts arrive within ten minutes. The team must decide whether the change introduced a real regression, whether monitoring is observing normal warm-up, or whether the alert rule is poorly calibrated. The runbook goal is to produce an operational decision: continue, watch, fix forward, or roll back with validation.

Name the decision before querying

KQL should not become an open-ended hunt for anomalies. Before opening the logs, write down the decision you need and the criteria that make it defensible. A rollback is not justified because one metric moved. It is justified when user impact, technical regression and timing around the change line up strongly enough.

Decision to produce
Continue the deployment
Watch with a short observation window
Fix forward with a bounded change
Roll back the version or configuration

Minimum evidence
Timestamped deployment window
User symptom or SLO degradation
Plausible technical change
Comparable before/after signal
Affected scope: service, region, revision, route, identity or dependency
Validation test after action
Documented return path

Stop conditions
Critical error increasing after the change
Confirmed user degradation
Dependencies healthy but deployed service unstable
Not enough evidence for a safe fix forward
Rollback available and less risky than waiting

This step prevents two opposite mistakes: rolling back on the first noise, or staying exposed because the evidence is not perfectly packaged yet.

Set a clean change window

Most investigations fail because the time window is vague. Isolate three periods: baseline before deployment, change window and observation period after deployment. Keep those boundaries in the queries, even if the team later adjusts scope.

let deploymentStart = datetime(2026-06-21T09:30:00Z);
let deploymentEnd = datetime(2026-06-21T09:42:00Z);
let beforeWindow = 45m;
let afterWindow = 45m;
let serviceName = "orders-api";
AppRequests
| where TimeGenerated between ((deploymentStart - beforeWindow) .. (deploymentEnd + afterWindow))
| where AppRoleName == serviceName
| extend phase = case(
  TimeGenerated < deploymentStart, "before",
  TimeGenerated <= deploymentEnd, "deploying",
  "after")
| summarize
  requests = count(),
  failed = countif(Success == false),
  p95_duration_ms = percentile(DurationMs, 95)
by phase, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

The exact query is not the main point. The discipline is: compare the same service, same granularity, same log source and same definition of failure.

Separate user impact from internal noise

A service can produce more exceptions without real user impact, for example during warm-up, client reconnection or cache refresh. The opposite is also true: a small number of visible exceptions can hide client-side timeouts. Start the rollback decision with impact.

let deploymentStart = datetime(2026-06-21T09:30:00Z);
let deploymentEnd = datetime(2026-06-21T09:42:00Z);
let afterWindow = 45m;
AppRequests
| where TimeGenerated between ((deploymentStart - 45m) .. (deploymentEnd + afterWindow))
| where AppRoleName == "orders-api"
| summarize
  total = count(),
  failed = countif(Success == false),
  server_errors = countif(ResultCode startswith "5"),
  client_errors = countif(ResultCode startswith "4"),
  p95_duration_ms = percentile(DurationMs, 95)
by bin(TimeGenerated, 5m), OperationName
| extend failure_rate = todouble(failed) / todouble(total)
| where total > 20
| order by TimeGenerated asc, failure_rate desc

If the failure rate rises on a core user operation, the investigation can move toward rollback. If only health endpoints or internal jobs are noisy, the decision may be to watch or adjust the alert.

Correlate regression with version or configuration

Timing alone is not enough. A spike after deployment may come from unusual traffic, an external dependency, a network incident or an expired identity. Tie the symptom to an artifact: version, revision, slot, image, rule set, route, feature flag or configuration.

let deploymentStart = datetime(2026-06-21T09:30:00Z);
AppTraces
| where TimeGenerated > deploymentStart - 30m
| where AppRoleName == "orders-api"
| extend version = tostring(Properties["appVersion"])
| extend revision = tostring(Properties["revision"])
| summarize
  traces = count(),
  errors = countif(SeverityLevel >= 3),
  sample_error = anyif(Message, SeverityLevel >= 3)
by version, revision, bin(TimeGenerated, 10m)
| order by TimeGenerated asc, errors desc

When the faulty version is visible, the decision gets simpler. When it is not visible, that is an operations gap to fix: an application that does not log its version makes rollbacks slower and harder to defend.

Check dependencies before blaming the deployment

An application rollback will not fix an unavailable dependency, broken DNS resolution, an overbroad WAF rule or an identity without access. Before going back, verify whether the deployed service is failing because it is wrong or because its environment changed.

let deploymentStart = datetime(2026-06-21T09:30:00Z);
AppDependencies
| where TimeGenerated between ((deploymentStart - 45m) .. (deploymentStart + 60m))
| where AppRoleName == "orders-api"
| summarize
  calls = count(),
  failed = countif(Success == false),
  p95_duration_ms = percentile(DurationMs, 95),
  result_codes = make_set(ResultCode, 10)
by Target, DependencyType, bin(TimeGenerated, 5m)
| extend failure_rate = todouble(failed) / todouble(calls)
| where calls > 10
| order by TimeGenerated asc, failure_rate desc

If one dependency fails for several services at once, rolling back the application may not be the first action. If dependencies stay stable while the new version fails, rollback becomes more credible.

Produce a stop, go or rollback matrix

The decision should fit in a few lines, not in a scattered chat thread. A simple matrix helps the team choose without inventing new criteria during the incident.

Continue
No measurable user degradation
Errors limited to warm-up or internal endpoints
Dependencies stable
Alerts noisy but explained

Watch
Weak or intermittent signal
User impact not confirmed
Expected stabilization window
Additional validation in progress

Fix forward
Cause identified and bounded fix available
Smaller blast radius than a full rollback
Immediate validation available
Fix can itself be reverted

Roll back
Confirmed user degradation
Regression tied to version or configuration
No bounded fix available inside the window
Rollback tested or known procedure
Success criterion after rollback defined

This matrix should be prepared before the incident. During the incident, the team should only fill in the evidence and choose the matching line.

Validate rollback as a production change

A rollback is not magic undo. It is a production change that can fail, hide another cause or reintroduce an older limitation. Validate it with the same signals that justified the decision.

Before rollback
Target version or configuration identified
Command or procedure reviewed
Expected impact written down
Notification channel preserved
Start time recorded

During rollback
Watch user errors
Watch p95 latency or main SLO
Watch critical dependencies
Keep logs active

After rollback
Compare 15 minutes before/after
Confirm user signal recovery
Check for new critical errors
Capture stabilization evidence
Open durable fix or post-incident follow-up

The validation should also say what to do if rollback does not reduce impact. In that case, go back to dependency, routing, identity or platform diagnosis instead of chaining blind rollbacks.

Automate collection, not the sensitive decision

Automation can prepare the decision without taking it alone. An AWX job, script or internal agent can collect the same KQL queries, produce a summary and attach evidence to the ticket. The rollback decision should remain human until the organization has proven that scope, thresholds and recovery are safe.

Acceptable automated collection
Deployment window
Before/after error rate
Before/after latency
Most affected operations
Failed dependencies
Observed version or revision
Links to dashboards and logs

Mandatory human validation
Confirm user impact
Choose rollback or fix forward
Accept recovery risk
Trigger the sensitive change
Confirm stabilization

This separation keeps tooling useful without turning the incident into an automatic red button.

Conclusion

Deciding a rollback after deployment is not about instinct or bravery. It is a chain of evidence: change window, user impact, before/after comparison, version correlation, dependency state, stop/go criteria and validation after action.

Azure Monitor and KQL provide the material, but the runbook provides the decision. When criteria are ready before the incident, the team can act quickly without silencing alerts, without confusing noise and regression, and without rolling back more broadly than needed.