Azure Site Recovery: test a recovery plan without touching production

An Azure Site Recovery plan can be fully configured and still fail when an incident occurs. Replication may look healthy while the startup order is wrong, the recovery network cannot resolve required names, or an identity has lost access to a dependency. The useful proof is a test failover that brings the application back and produces verifiable evidence.

The use case is a three-tier application protected by Azure Site Recovery: database, application services and web frontend. The team needs to exercise its recovery plan without interrupting production or accidentally connecting test machines to live flows. The runbook must end with a decision: plan ready, targeted repairs followed by another test, or a cleanly aborted and removed exercise.

Define what the exercise must prove

A test failover should not stop at seeing VMs boot. Write the success criteria before execution: acceptable recovery point, startup order, available dependencies, application checks and maximum duration.

exercise:
recovery_plan: rp-orders-prod
owner: platform-operations
protected_scope:
  - sql-orders-01
  - api-orders-01
  - web-orders-01
recovery_point: latest_app_consistent
test_vnet: vnet-drill-weu
production_connectivity: forbidden

success_criteria:
- all_recovery_plan_groups_complete
- dns_resolves_test_dependencies
- database_accepts_read_only_validation
- api_health_returns_expected_build
- synthetic_order_uses_test_data_only
- no_production_queue_or_webhook_is_reached
- cleanup_completes_and_replication_remains_healthy

decision:
ready: all_critical_checks_pass
repair: bounded_gap_with_owner_and_deadline
abort: isolation_or_data_safety_not_proven

This contract separates infrastructure recovery from business recovery. A running VM proves neither data consistency nor the service’s ability to process a request.

Build a genuinely isolated test network

The drill network should reproduce useful subnets and address ranges without a route to production. Do not add peering or a route merely to make troubleshooting easier. If Active Directory or DNS is required, provide a dedicated test instance or a controlled copy inside that boundary.

Before test failover
Dedicated and clearly named test VNet
Subnet names consistent with Site Recovery mappings
No peering to production VNets
No route to production on-premises ranges
NSGs restrict ingress to approved administration paths
Test DNS configured and critical resolutions documented
Internet egress absent or limited to required destinations
Webhooks, queues, SMTP and partner APIs replaced with test targets

Block the exercise
Selected VNet is the production recovery-site network
A dependency can work only by writing to production
The plan assumes a fixed IP that is unavailable in the test subnet
The team cannot distinguish drill data from live data

Site Recovery attempts to reuse the expected subnet name and IP address. If the subnet is missing or the address is unavailable, placement can differ. Inspect the created NICs instead of assuming the mapping was preserved.

Read the recovery plan as a dependency graph

Recovery plan groups should represent application order. In a typical architecture, the database starts before the API, followed by the frontend. Components in one group start in parallel, so two dependent services should not share a group just to make the plan shorter.

Group 1 - foundations
Test DNS or directory services when required
Database
Storage mount and consistency check

Group 2 - services
APIs and workers
Secrets and identities available
Validation pause before frontend exposure

Group 3 - test exposure
Web frontend
Internal load balancer or test endpoint
Synthetic probes

Actions to review
Pre-actions do not depend on production endpoints
Azure Automation runbooks are idempotent and bounded to the test resource group
Manual actions have an owner and resume criterion
Post-actions do not enable public DNS or client traffic

Recovery automation should receive an explicit test context. A runbook that infers its environment from a VM name can execute the wrong action after a rename or naming-convention change.

Choose the recovery point for a reason

The latest processed point usually minimizes recovery time, while an application-consistent point prioritizes application state. The right choice depends on the scenario. For a transactional application restart drill, an application-consistent point is often the stronger candidate; for measuring minimum RTO, the latest processed point may be relevant.

Record the selected timestamp and its distance from live production. If one VM in the plan does not have the expected recovery-point type, stop before silently combining incompatible states.

Execute and collect evidence as the plan progresses

Start test failover from the recovery plan, select the isolated VNet and follow the Site Recovery jobs. Capture each transition: prerequisite checks, VM creation, group startup, automated actions and manual pauses.

{
"exerciseId": "drill-2026-08-08-orders",
"recoveryPlan": "rp-orders-prod",
"recoveryPointUtc": "2026-08-08T05:42:00Z",
"testVnet": "vnet-drill-weu",
"jobs": [
  {"group": "database", "status": "completed", "durationSeconds": 420},
  {"group": "services", "status": "completed", "durationSeconds": 265},
  {"group": "frontend", "status": "completed", "durationSeconds": 118}
],
"checks": {
  "replicationHealth": "healthy",
  "dns": "pass",
  "database": "pass",
  "apiHealth": "pass",
  "syntheticJourney": "pass",
  "productionIsolation": "pass"
},
"observations": []
}

The example timings illustrate an evidence format, not a universal target. Measure your own milestones: job start, VM availability, service readiness and validated business journey.

Validate from the recovery path

Checks should originate inside the test network and use the names that the recovered application actually uses. Connecting directly to a VM IP can hide broken DNS, an invalid certificate or a misconfigured load balancer.

set -euo pipefail

TEST_HOST="orders.drill.internal"
EXPECTED_BUILD="2026.08.08"

getent hosts "$TEST_HOST"

curl --fail --silent --show-error --connect-timeout 5 --max-time 15 "https://$TEST_HOST/health"

actual_build=$(curl --fail --silent "https://$TEST_HOST/version")
test "$actual_build" = "$EXPECTED_BUILD"

# This journey must use a test tenant, queue and webhooks.
curl --fail --silent --show-error -H "X-Drill-Id: drill-2026-08-08-orders" -H "Content-Type: application/json" --data '{"customer":"drill-only","amount":1}' "https://$TEST_HOST/api/orders/validate"

Add a negative control: the service must not publish to a production queue, call a live webhook or update a public DNS record. Isolation is a test result, not merely an assumed VNet property.

Classify gaps without improvising during the drill

When a step fails, preserve the state and classify the gap before changing anything. A boot problem is not handled like a DNS problem; an identity failure does not justify opening an NSG.

VM missing or job blocked
Check replication, quota, SKU, disk, zone and the Site Recovery job

VM running, service stopped
Check plan order, mounts, local configuration and startup dependency

Name unresolved or wrong IP
Check test VNet DNS, subnet, records and caches

401 or 403 from a dependency
Check identity, audience, role and test-resource scope

Application healthy, business journey failing
Check queue, webhook, certificate, load balancer and test data

Traffic observed toward production
Stop the drill, cut the network path, retain evidence and clean up

Material corrections belong in the recovery plan, IaC or source runbook and should be proven in a new exercise. A manual repair on a temporary VM does not make the recovery plan reliable.

Clean up and confirm the nominal state

Finish with Cleanup test failover and record observations. Confirm that temporary VMs and NICs are gone, no lock or auxiliary resource remains orphaned, and replication health for protected items is still good.

Plan ready
All critical criteria pass from the test network
Measured recovery time meets the internal objective
Production isolation is proven
Cleanup is complete and replication remains healthy

Plan needs repair
Bounded gap has an owner, source correction and retest date
No critical validation is marked successful through a manual workaround
The test is cleaned up before another run

Exercise aborted
Isolation, data consistency or a critical dependency is not controlled
Test flows are cut immediately
Evidence is retained, cleanup runs and an incident is opened when needed

Conclusion

A successful Azure Site Recovery test is more than three green VMs in the portal. It proves a recovery point, startup order, isolated network, available dependencies, controlled application journey and complete cleanup.

The final decision should stay binary for critical controls: the plan is ready, or it must be repaired and tested again. That discipline turns Site Recovery from reassuring configuration into a recovery system the operations team can actually use.