Azure Service Health: qualify a regional signal before failover

An Azure Service Health or Resource Health signal rarely arrives in a calm moment. A region reports degradation, a resource turns unavailable, a dashboard starts moving, and the team has to decide quickly: wait, fail over, roll back, stop an automation path, or communicate impact.

The trap is treating the Azure signal as if the decision had already been made. Service Health describes a platform event. Resource Health brings the signal closer to a specific resource. Neither one proves application impact, the right failover scope, or the safety of rollback on its own. This runbook connects provider signal, user symptoms, dependencies, routing, DNS, traces and return path before a regional failover is executed.

Define the decision scope

Start by writing the incident contract. Regional failover is not a single technical command. It touches routing, caches, data, queues, identities, private endpoints, observability and the ability to return.

incident:
signal: Azure Service Health degradation
region: westeurope
first_seen_utc: 2026-07-19T06:42:00Z
impacted_services:
  - App Service
  - Azure SQL
  - Storage
workload: customer-api-prod
primary_region: westeurope
secondary_region: northeurope

decision_needed:
- wait_and_monitor
- fail_over_traffic
- rollback_last_release
- disable_write_path
- pause_automation

evidence_required:
- provider_event_scope
- user_visible_symptom
- resource_health_state
- dependency_errors
- routing_and_dns_state
- data_consistency_risk
- rollback_path

Without that scope, the team can confuse an Azure regional issue, an application regression, a DNS mistake, a dependency limit or an observability gap.

Separate Service Health, Resource Health and application signal

Service Health is useful provider context. Resource Health ties the signal to a concrete resource. Application logs show whether users are actually affected. Read the three layers together.

Service Health
Provider event
Declared region and services
Status, latest message, start time
Possible impact across subscriptions

Resource Health
State of a concrete resource
Available, Degraded, Unavailable or Unknown
Transition history
Platform or user annotation

Application signal
Error rate, latency, saturation, timeouts
User journeys affected
Dependencies in error
Difference between primary and secondary region

Decision
Failover is justified only when application signal and risk of staying exceed failover risk

A Service Health event without application symptoms may justify tighter monitoring. Application symptoms without a Service Health event should remain a product or infrastructure incident until evidence says otherwise.

List affected resources without extrapolating

Before failover, verify which resources are actually degraded. If you query Azure Resource Graph, the goal is to connect critical resources, their region and their health status.

resourcehealthresources
| where type =~ "microsoft.resourcehealth/availabilitystatuses"
| extend resourceId = tostring(properties.targetResourceId)
| extend availabilityState = tostring(properties.availabilityState)
| extend reasonType = tostring(properties.reasonType)
| extend summary = tostring(properties.summary)
| where resourceId has "/resourceGroups/rg-prod-"
| project resourceId, availabilityState, reasonType, summary, location
| order by availabilityState asc, resourceId asc

Do not use this view alone to decide. Use it to reduce the scope: which critical resources are degraded, which remain available, and which dependencies are not covered by the signal.

Confirm user impact before failover

Regional failover can make an incident worse when the secondary region is not warm, data is not synchronous, private paths have not been validated, or allowlists do not cover the recovery region. Confirm user impact before taking the heavy action.

let Window = 2h;
let PrimaryRegion = "westeurope";
requests
| where timestamp > ago(Window)
| extend region = tostring(customDimensions["region"])
| summarize total=count(),
          failed=countif(success == false),
          p95_ms=percentile(duration, 95)
by bin(timestamp, 5m), region, cloud_RoleName
| extend failure_rate = todouble(failed) / todouble(total)
| where region in (PrimaryRegion, "northeurope")
| order by timestamp asc

If errors increase only on a non-user internal endpoint, the decision may be to degrade a feature or suspend automation. If the primary journey fails, failover becomes a stronger candidate.

Verify routing, DNS and the secondary path

Failover often fails on operational details: DNS TTL too long, Front Door or Traffic Manager weights not ready, cold secondary origin, invalid certificate, missing Private DNS Zone link, firewall or NAT not prepared, probe watching the wrong path.

User path
Front Door, Traffic Manager, Application Gateway or public DNS
Health probe checking the real application endpoint
Certificate and hostname valid on secondary
Known TTL and acceptable propagation time

Private path
Private DNS Zone linked to secondary VNets
Hybrid resolver or forwarder ready
UDR, NSG, firewall and NAT checked
PaaS dependencies reachable from the recovery region

Data path
Replication mode known
RPO and RTO reread
Queues, jobs and consumers idempotent
Writes blocked when consistency is not guaranteed

Private Endpoint may be part of the path, but it should not become the whole diagnosis. Failover also depends on global routing, DNS, application state, replication and security controls.

Decide with an operable matrix

The runbook must produce a decision that can be read later. An on-call team should not have to guess why traffic moved at 08:00 or why the team chose to wait.

Wait and monitor
Service Health degraded but no confirmed user impact
Resource Health available on critical resources
Synthetic probes and application logs stable
Review scheduled inside a short window

Fail over traffic
User impact confirmed on primary region
Secondary path validated by application probe
Data and dependencies compatible with failover
Rollback or primary return documented

Roll back the last change
Symptom started after deployment or infrastructure change
Service Health does not cover the affected component
Secondary would carry the same application risk
Previous healthy state known

Degrade one feature
Main journey stable
Secondary dependencies or asynchronous jobs affected
Writes or automations could duplicate state
Degraded mode testable and reversible

Block failover
Secondary state unknown
Replication or consistency not proven
DNS, certificates or probes not valid
Return path cannot be executed cleanly

The decision must include a reevaluation time. Waiting without a horizon is not a strategy.

Execute failover with guardrails

If failover is selected, reduce simultaneous changes. Change one routing surface, observe, then continue. Failover automation should stay bounded: no global purge, no alert threshold edits, no broad network opening just to make traffic pass.

change:
action: shift_traffic_to_secondary
owner: incident-commander
start_utc: 2026-07-19T07:10:00Z
traffic_surface: front-door-origin-priority
primary_origin: app-westeurope
secondary_origin: app-northeurope

guards:
max_change_per_step: one_routing_surface
keep_logs_enabled: true
keep_user_symptom_alert: true
freeze_nonessential_deployments: true
pause_write_automation_if_consistency_unknown: true

validation:
- synthetic_probe_secondary_success
- user_error_rate_decreases
- dependency_errors_do_not_move_to_secondary
- no_duplicate_jobs_or_messages
- rollback_instruction_confirmed

The best failover leaves enough evidence to understand later whether it fixed the incident or only moved the symptom.

Validate after failover or rollback

After action, validate both service behavior and control. A green page is not enough if asynchronous jobs duplicate work, traces are no longer correlated, or returning to primary has become unsafe.

Service validation
User error rate returned to expected level
Latency stable on the secondary path
Critical dependencies without a new dominant error
Queues, jobs and webhooks without duplication
Data consistent with the replication mode

Control validation
Decision, time, evidence and owner recorded in the incident
User-symptom alerts still active
Logs and traces visible on the active region
Primary return or routing rollback tested read-only
Temporary actions have an end time

If failover reduces impact but return is unclear, the incident stays open. The target state may be stable, but operations have not returned to a controlled mode yet.

Conclusion

An Azure Service Health or Resource Health signal should trigger diagnosis, not automatic failover. The useful question is not only “is Azure down?” but “what impact do we see, which path is affected, which secondary is ready, and which return path remains possible?”.

By separating provider signal, resource state, user symptom, routing, DNS, data and observability, the team can choose between monitored waiting, failover, rollback, degraded mode or blocking the change. The decision becomes operable: it carries evidence, validation and a return path instead of a rushed reaction to a regional alert.