Azure Chaos Studio: bound a resilience test before disrupting production

A resilience test can cause the incident it was meant to prevent. A scenario behaves correctly in preproduction, then reaches too many production resources, runs through an overprivileged identity, or continues after the service has already exhausted its error budget. The problem is not fault injection by itself. It is the lack of a verifiable execution contract.

Consider an Azure API deployed across availability zones, protected by a frontend and monitored in Azure Monitor. The team wants to use Azure Chaos Studio to simulate the loss of a zone or the unavailability of one component. The goal is not to claim that “production is resilient.” It is to test one precise hypothesis, within a bounded scope, with a prepared abort decision and a recovery path.

State the hypothesis before configuring the scenario

Start with what the test must prove. A list of faults is not a hypothesis. The disruption must map to an expected system behavior and then to an observable signal.

test:
id: chaos-api-zone-loss-2026-08-09
service: orders-api
environment: production
hypothesis: >
  The API remains available when one application zone becomes unavailable
  and traffic is served by healthy instances in the remaining zones.
workspace_scope: rg-orders-prod
scenario: compute-zone-down
target_allowlist:
  - app-zone-1
explicitly_excluded:
  - database-primary
  - shared-dns
  - central-firewall
observation_window_minutes: 15
abort_if:
  availability_below_percent: 99.0
  p95_latency_above_ms: 900
  error_rate_above_percent: 2.0
recovery_owner: platform-oncall
change_ticket: CHG-2026-0819

This contract makes exclusions as important as targets. A Chaos Studio Workspace discovers resources within its scope and runs Scenarios through its managed identity. The Workspace scope, that identity’s roles and the Scenario configuration jointly define the blast radius. Classic experiments follow the same principle, but identity and permissions are attached to each experiment.

Verify the effective blast radius

Do not approve a test from the resource group name shown in the portal. Review the complete Azure resource IDs, the identity that will actually execute the actions and its RBAC assignments. Contributor at subscription scope defeats containment even when the Scenario UI shows a single VM.

WORKSPACE_ID="/subscriptions/<sub>/resourceGroups/rg-orders-prod/providers/Microsoft.Chaos/workspaces/chaos-orders-prod"
PRINCIPAL_ID="<workspace-managed-identity-object-id>"

az resource show --ids "$WORKSPACE_ID" --query "{id:id, identity:identity, properties:properties}" -o json

az role assignment list --assignee-object-id "$PRINCIPAL_ID" --all --query "[].{role:roleDefinitionName, scope:scope}" -o table

az resource list --resource-group rg-orders-prod --query "[].{name:name, type:type, id:id}" -o table

The review must produce an explicit inventory: resources reachable by the identity, resources selected by the Scenario, and authorized actions. With classic experiments, also verify that every resource is onboarded as a target and that only the required capabilities are enabled. A missing capability should fail the action; it should not be added under pressure during the production window.

Separate permission to edit a Scenario from permission to run it. The operator executing the test does not also need the ability to widen scope, add another fault and grant roles in the same window.

Capture an actionable baseline

You cannot assess a test without knowing normal behavior. Capture a baseline immediately before the run using the same signals as the abort criteria: availability, failures, latency and dependencies.

let TestStart = datetime(2026-08-09T08:30:00Z);
let BaselineStart = TestStart - 30m;
let TestEnd = TestStart + 15m;
AppRequests
| where TimeGenerated between (BaselineStart .. TestEnd)
| where AppRoleName == "orders-api"
| extend Window = iff(TimeGenerated < TestStart, "baseline", "chaos")
| summarize
  Requests = count(),
  Errors = countif(Success == false),
  Availability = 100.0 * countif(Success == true) / count(),
  P95DurationMs = percentile(DurationMs, 95)
by Window, bin(TimeGenerated, 1m)
| extend ErrorRate = 100.0 * Errors / Requests
| order by TimeGenerated asc

Add infrastructure signals specific to the service: healthy frontend backends, saturation on surviving instances, messaging backlog, dependency failures and probe status. The Scenario does not pass because one request still succeeds. It passes only if the system remains within its thresholds for the whole observation window and recovers without hidden operational debt.

Freeze concurrent changes as well. A deployment, secret rotation or network change during the run makes causality hard to establish and can turn a controlled abort into an ambiguous incident.

Run through progressive gates

A Scenario that passed in preproduction should not immediately run across the full production scope. Use gates that increase traffic realism, duration and target count separately.

Gate 1 - configuration review
Exact scenario revision recorded
Target allowlist and exclusions reviewed
Workspace identity roles reviewed
Stop permission tested by the operator

Gate 2 - preproduction
Same fault family and observation queries
Recovery action executed successfully
No unexplained telemetry gap

Gate 3 - production canary
One target or one bounded zone
On-call, service owner and observer present
Deployments and infrastructure changes frozen

Gate 4 - decision
Continue only while every abort signal is healthy
Stop on missing telemetry, unexpected target or threshold breach
Never widen scope during the active run

The run must produce an execution ID correlated with the change ticket, dashboards and operator log. For Workspaces and Scenarios, retain the Scenario report. For classic experiments, retain execution details and the status of each action. Without that correlation, the team may know that a fault was injected but not exactly when or where.

Treat stop as the start of recovery

Stopping or canceling a run prevents the remaining Scenario actions, but it does not prove that the service has returned to its initial state. An action that already took effect can require compensation: restart a resource, restore routing, re-enable a rule, wait for reconnection or drain accumulated backlog.

recovery:
immediate:
  - stop the active scenario run
  - record the stop timestamp and last completed action
  - verify that no unexpected target was affected
compensate:
  - restore the target configuration from the approved baseline
  - confirm traffic returns to healthy instances
  - drain or replay backlog only after consumers are stable
validate:
  - availability and error rate healthy for 15 minutes
  - p95 latency back inside baseline tolerance
  - dependencies and health probes healthy
  - no remaining action or retry in progress
rollback_the_test:
  - disable the scenario revision
  - remove temporary target capabilities if they are no longer needed
  - reduce identity assignments added only for this test
  - attach evidence and decision to the change ticket

Exercise this path before the production window. In particular, verify that the operator can stop the run, that the owner of every compensating action is available and that restoration commands do not depend on the component being deliberately disrupted.

Close with an evidence-based decision

The runbook does not end with “test completed.” It ends with one of three decisions: hypothesis validated, inconclusive result, or insufficient resilience.

The hypothesis is validated only when the expected targets were affected, thresholds remained healthy, telemetry is complete and recovery is proven. The result is inconclusive when logs are missing, a concurrent change contaminates the observation, or the fault did not execute as designed. Resilience is insufficient when an abort threshold is breached or recovery requires an undocumented intervention.

For the last two outcomes, do not immediately rerun with a wider scope. Correct the Scenario, the system or the runbook, then return to the previous gate. A useful chaos test does not maximize disruption. It reduces uncertainty without surrendering control of production.