Infrastructure

Azure IaC: validate an infrastructure plan before production apply

A production runbook for qualifying a Terraform, Bicep or ARM plan before apply with drift, CI identity, Azure Policy, destructive changes, validation and rollback.

08 Jul 2026 azureiacterraformbicepwhat-ifazure-policyci-cdidentityobservabilityrunbookrollbackproduction

An IaC plan that passes CI is not automatically safe for production. It can hide manual drift, replace a critical resource, modify an identity, recreate an endpoint, remove a diagnostic setting or bypass an Azure Policy constraint. Under pressure, the team may approve the apply because the pipeline is green, then discover that the plan was technically valid but operationally unsafe.

The use case is an Azure platform managed with Terraform, Bicep or ARM from a CI/CD pipeline. The change touches networking, a managed identity, Key Vault, App Service, diagnostic settings or policy assignments. The runbook goal is to decide whether the apply can proceed, whether the plan must be fixed, whether drift must be handled first, or whether the change needs a bounded rollback path.

Freeze the change scope

Start by turning the plan into an operational contract. An IaC diff is not only a list of resources. It is a promise about what will be created, modified, replaced, deleted and validated afterward.

text iac-change-contract.txt

Change to qualify
Environment: production
Azure scope: subscription, resource group or management group
Tool: Terraform, Bicep or ARM
Pipeline: platform-iac-prod
Execution identity: workload identity federation or managed identity
Sensitive resources: network, identity, secrets, diagnostic settings, policies
Apply window: approved and recoverable

Evidence required before apply
Plan or what-if attached to the change
Deletes and replacements listed
Manual drift known or excluded
CI identity and permissions verified
Azure Policy impact understood
Application validation prepared
Rollback or mitigation path documented

If the plan does not clearly state what can be destroyed or replaced, the apply should wait. The first guardrail is a decision that can be read by someone else.

Separate plan, drift and intent

A plan can contain three different things: the intended change, existing drift, and side effects from a provider or module. Mixing them creates applies that are too broad.

text plan-classification.txt

Classify each diff
Intended
  Resource or property requested by the PR
  Expected impact described in the change note
  Post-apply validation defined

Drift to qualify
  Value changed outside IaC
  Tag, diagnostic setting, role assignment or network setting differs
  Change unrelated to the PR

Risk to block
  Replacement of a stateful resource
  Deletion of private DNS link, diagnostic setting, lock or role assignment
  Change to execution identity
  Azure Policy or exemption widened too far
  Difference not explained by the PR

The right move is not to absorb all drift into the same apply. Fix or accept drift explicitly before delivering the functional change.

Verify the pipeline identity

Before trusting the plan, prove which identity produced it. An OIDC federation, service connection or Azure role issue can produce an incomplete plan, or succeed with broader rights than expected.

bash 01-ci-identity-scope.sh

SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
PRINCIPAL_ID="11111111-1111-1111-1111-111111111111"

az account set --subscription "$SUBSCRIPTION"

az role assignment list --assignee "$PRINCIPAL_ID" --all --query "[].{role:roleDefinitionName,scope:scope,condition:condition}" --output table

az ad sp show --id "$PRINCIPAL_ID" --query "{appId:appId,displayName:displayName,servicePrincipalType:servicePrincipalType}" --output json

The question is not only whether the pipeline has permission. It is also whether it has only the permissions needed for the expected scope. A production apply executed with broad rights is harder to reason about during rollback.

Read the plan as a risk list

For Terraform, extract plan actions instead of rereading the full console output. For Bicep or ARM, use what-if and isolate deletes, replacements and sensitive resource changes.

bash 02-terraform-plan-risk-summary.sh

terraform plan -out=tfplan
terraform show -json tfplan > tfplan.json

jq -r '
.resource_changes[]
| {
    address: .address,
    type: .type,
    name: .name,
    actions: .change.actions
  }
| select(.actions | index("delete") or index("replace"))
' tfplan.json

bash 03-bicep-what-if-risk-summary.sh

RESOURCE_GROUP="rg-prod-platform"
TEMPLATE="main.bicep"
PARAMETERS="main.prod.bicepparam"

az deployment group what-if --resource-group "$RESOURCE_GROUP" --template-file "$TEMPLATE" --parameters "$PARAMETERS" --result-format FullResourcePayloads --output json > what-if.json

jq -r '
.changes[]
| select(.changeType == "Delete" or .changeType == "Modify")
| {resourceId: .resourceId, changeType: .changeType}
' what-if.json

Replacing an App Service Plan, subnet, Private DNS Zone link, Key Vault access model or diagnostic setting does not carry the same risk as changing a tag. Put those risks at the top of the review.

Check Azure Policy before apply

Azure Policy can block the apply, modify resources through deployIfNotExists, or hide drift until remediation has run. The plan must include that constraint.

bash 04-policy-state-before-apply.sh

SCOPE="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod-platform"

az policy state list --resource "$SCOPE" --query "[].{policy:policyDefinitionName,assignment:policyAssignmentName,compliance:complianceState,resourceId:resourceId}" --output table

az policy assignment list --scope "$SCOPE" --query "[].{name:name,displayName:displayName,enforcementMode:enforcementMode}" --output table

Block the apply if the change requires disabling a policy without an owner, widening an exemption without an expiry date, or ignoring non-compliance on the resource being modified.

Prepare useful post-apply validation

A successful apply is not validation. It only proves that the IaC engine completed. Validation must start from the real consumption paths: application request, DNS resolution, identity, logs, metrics and security controls.

yaml post-apply-validation.yml

validation:
infrastructure:
  - resource_exists_with_expected_sku_and_location
  - no_unplanned_delete_or_replace
  - diagnostic_settings_still_enabled
  - locks_and_tags_preserved_when_required

network:
  - private_dns_resolution_from_workload_network
  - effective_routes_match_expected_path
  - nsg_or_firewall_logs_show_expected_flow

identity:
  - managed_identity_or_federated_principal_unchanged
  - key_vault_or_api_access_validated_with_real_identity
  - no_broad_role_added_for_apply_convenience

application:
  - synthetic_probe_passes
  - logs_visible_with_correlation_id
  - user_path_or_health_endpoint_validated

decision:
  - promote_change
  - hold_and_fix
  - rollback_or_mitigate

Write the validation before the apply. Otherwise the team may discover too late what it should have proven.

Decide apply, fix or rollback

Keep the decision short and defensible. A plan can be technically valid and still too risky for a production window.

text iac-apply-decision.txt

Allow apply
Deletes and replacements are expected or absent
Drift is understood and separated from the change
CI identity is bounded to the expected scope
Azure Policy is compliant or the exception is approved
Post-apply validation is ready

Fix before apply
The plan contains drift unrelated to the PR
A stateful resource would be replaced
An identity or role assignment changes without justification
A policy would be bypassed to make the deployment pass
Probes or logs for validation are unavailable

Rollback or mitigate
The apply removed a diagnostic or access path
Application validation fails after the change
A resource was replaced instead of modified
The previous plan can be reapplied with a reduced scope
A temporary mitigation has an owner and expiry date

IaC rollback must also stay bounded. Reverting to an old commit without reviewing the plan can reintroduce other changes. Produce a rollback plan, review its deletes, then validate it as a normal change.

Keep evidence in the PR

The PR should contain enough evidence for operations to understand the decision later: plan, risk summary, identity, policy, validation and rollback.

markdown pull-request-evidence.md

## Infrastructure change evidence

- Plan artifact: attached to CI run
- Risk summary: no unplanned delete or replace
- Drift: tag drift accepted separately, no network drift included
- CI identity: workload identity federation, scoped to rg-prod-platform
- Azure Policy: no new exemption, current non-compliance not touched
- Validation: private probe, Key Vault read, diagnostic logs
- Rollback: previous module version with targeted plan reviewed
- Decision: apply approved for production window

That trace prevents the next incident from reopening the whole investigation. It turns IaC into an operational interface, not only a deployment mechanism.

Conclusion

A production IaC plan should be reviewed as a change runbook. The safe decision does not come from a green CI status. It comes from evidence: intent separated from drift, deletes visible, identity bounded, Azure Policy understood, validation ready and rollback reviewed.

The apply can proceed when the plan is explainable and the team knows what to check afterward. It should wait when the diff mixes change, drift and side effects. That discipline is what makes automated infrastructure genuinely operable.