Azure Resource Graph: reconstruct configuration drift before rollback

An Azure API slows down immediately after a deployment window. The code did not change, probes are still green and no platform incident is reported. A route, network rule, service setting or automation-managed tag may still have drifted. Reapplying the full template or restoring the last application release would produce activity, but not evidence.

This runbook covers a production environment made of several Azure resources changed through IaC, the portal and automation. It uses Azure Resource Graph Change Analysis to isolate the changed property, then an Activity Log export in Log Analytics to recover the operation and identity. The final decision is deliberately narrow: keep the change, repair one property, return to the previous IaC version or suspend the automation that introduced it.

Freeze the symptom and change window

Start with an observable impact, not a list of suspected resources. Record a repeatable test, its UTC timestamp, affected components and one unaffected control component. That comparison keeps the team from blaming the latest deployment for an older or broader failure.

Incident: inc-20260809-014
Symptom: latency and timeouts on orders-api
First observed: 2026-08-09T15:18:00Z
Last healthy test: 2026-08-09T15:04:00Z
Search window: 14:45Z to 15:30Z

Primary resource
/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-orders-prod/providers/Microsoft.Web/sites/orders-api

Dependencies to compare
VNet Integration and subnet
route table and NSG
App Configuration and Key Vault
App Service plan

Expected evidence
property before and after
change timestamp
originating identity or application
correlation with symptom onset
validation test and rollback path

Change Analysis retains a limited data window. Collect evidence while the incident is active and export the useful result into the incident record. It is designed to find recent changes, not replace durable configuration history.

Query changes at the useful scope

The resourcechanges table can search creates, updates and deletes across subscriptions. Start with time and resource group boundaries, then project actor attributes and the changed-properties bag. Records usually become available within a few minutes, so an immediate empty result is not yet proof that no change occurred.

resourcechanges
| extend changeTime = todatetime(properties.changeAttributes.timestamp),
       targetResourceId = tolower(tostring(properties.targetResourceId)),
       targetResourceType = tostring(properties.targetResourceType),
       changeType = tostring(properties.changeType),
       changedBy = tostring(properties.changeAttributes.changedBy),
       changedByType = tostring(properties.changeAttributes.changedByType),
       clientType = tostring(properties.changeAttributes.clientType),
       operation = tostring(properties.changeAttributes.operation),
       correlationId = tostring(properties.changeAttributes.correlationId),
       changeCount = toint(properties.changeAttributes.changesCount),
       changedProperties = properties.changes
| where changeTime between (datetime(2026-08-09T14:45:00Z) .. datetime(2026-08-09T15:30:00Z))
| where targetResourceId contains "/resourcegroups/rg-orders-prod/"
| project changeTime, targetResourceId, targetResourceType, changeType,
        changedBy, changedByType, clientType, operation, correlationId,
        changeCount, changedProperties
| order by changeTime desc

Do not filter down to the App Service too early. A route table, NSG, subnet or identity change can create the symptom without touching the application resource. Conversely, a long list of tag updates does not explain a timeout. Rank deltas by their ability to change the execution path.

Read the delta, not only the actor

The changedProperties field includes available previous and new values. Export the JSON result and inspect property paths that affect runtime behavior: network integration, rules, references, identity, sizing, TLS options or application settings. A different value is not automatically causal. It becomes a candidate when it precedes the symptom and changes a real path.

Property snapshots are not always available for deletes. Creates do not expose every changed property because that would generate excessive noise. In those cases, complete the record with the IaC template, current state and Activity Log.

For each candidate delta
Does the change precede the first failure?
Can the property alter the request path?
Does the same delta exist on the control component?
Does current state match the approved IaC repository?
Did automation rewrite the value after deployment?
Can a bounded test confirm the effect without another change?

Classify
confirmed causal change
plausible cause to test
noise with no operational effect
platform or unattributed change
incomplete evidence

changedBy or clientType can be empty, unspecified or reported as System. That does not prove the platform caused the incident. It only means Change Analysis did not attribute the event to an actionable actor, so other logs are required.

Connect the change to Activity Log

When Activity Log is exported to Log Analytics, use correlationId, resource ID and the UTC window to recover the ARM operation, caller and status. The Change Analysis operation field represents an authorization action such as Microsoft.Web/sites/write. It is useful for searching, but it does not always describe the exact property modification.

let StartTime = datetime(2026-08-09T14:45:00Z);
let EndTime = datetime(2026-08-09T15:30:00Z);
let TargetResource = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-orders-prod/providers/Microsoft.Web/sites/orders-api";
let ChangeCorrelationId = "00000000-0000-0000-0000-000000000000";
AzureActivity
| where TimeGenerated between (StartTime .. EndTime)
| where tolower(ResourceId) == tolower(TargetResource)
  or tostring(CorrelationId) == ChangeCorrelationId
| project TimeGenerated,
        OperationNameValue,
        ActivityStatusValue,
        Caller,
        CallerIpAddress,
        CorrelationId,
        ResourceId,
        Properties
| order by TimeGenerated asc

Next, find the CI run, automation job or change request at the same timestamp. The objective is not merely to name a person. Identify the authority that can write the value again. A portal repair will last only until the next Terraform or policy execution if either owns that configuration.

Compare current state with declared intent

Once the delta is isolated, compare three states: pre-incident value, current value and value declared in IaC. This triangulation separates manual drift, an expected deployment with an unexpected effect and automatic convergence toward an incorrect source value.

Do not use a full apply as a diagnostic tool. First generate a plan using the same commit, backend and production variables. If the plan wants to modify dozens of resources, it is not a bounded rollback. Prepare a targeted repair or return to the known commit, then review the plan before execution.

Choose repair, rollback or keep

Chronology alone is insufficient. The selected change must explain the symptom, and a controlled test must confirm recovery. Record the decision before running the reversal.

incident: inc-20260809-014
candidate_change:
resource: orders-api
property: properties.siteConfig.vnetRouteAllEnabled
previous: false
current: true
changed_at: 2026-08-09T15:11:42Z
authority: terraform-pipeline-prod
evidence:
- change precedes first timeout by 6 minutes
- dependency path follows the forced route
- control application without the change remains healthy
decision: rollback_iac_commit
validation:
- same synthetic request succeeds 5 times
- dependency latency returns to baseline range
- Resource Graph shows the expected reverse change
- no unrelated resource changes are introduced
stop_conditions:
- plan touches resources outside orders-api scope
- route behavior remains unchanged
- error rate increases during canary
rollback_of_rollback:
- restore the incident commit
- stop the pipeline
- escalate to network and application owners

The property above is an example incident record, not a configuration recommendation. Correct state depends on the architecture. Keep the change when tests prove it is healthy. Roll it back only when a precise delta explains the impact and the reversal is itself observable.

Validate the reversal and close the loop

After the repair, rerun the exact initial test. Check the symptom, application telemetry, dependency and the new resourcechanges record. A green HTTP response is insufficient if routing, identity or error volume remains degraded.

Then remove the source of drift: restrict the manual path, repair the IaC module, narrow the automation identity or alert on a critical property. For history beyond Change Analysis retention, export useful changes regularly to Log Analytics or governed storage.

Conclusion

Azure configuration drift is better handled as an evidence chain than as a deployment hunch. Resource Graph locates the delta, Activity Log recovers the operation, IaC exposes intent and a canary validates the effect.

The outcome is an operable decision: retain a healthy change, repair one faulty property or roll back the authority that introduced it. If causality remains incomplete, suspend the affected automation and preserve current state instead of launching a broad reversal without evidence.