Azure Policy remediation: fix drift without breaking production

An Azure Policy initiative can report drift without fixing it immediately. Diagnostic settings are missing, required tags were never applied, DeployIfNotExists configurations did not land, or a new scope inherited stricter rules than expected. The risk starts when remediation is treated as a global cleanup button: the identity lacks the right permissions, the target resources are not the expected ones, exemptions are ignored, and the team discovers the impact only after the first failures.

The use case is an Azure platform team that wants to correct policy drift without turning governance into an opaque production change. The runbook goal is to decide what to remediate, with which identity, on which scope, in which order, with which evidence and with which rollback if the correction creates side effects.

Treat remediation as a production change

Azure Policy remediation is not just a compliance report. Depending on the policy effect, it can deploy resources, modify configuration or trigger operations across many objects. It should be handled as production automation.

Before launch
Policy or initiative involved
Exact scope: management group, subscription, resource group or resource
Effect: DeployIfNotExists, Modify, AuditIfNotExists or upstream Deny
Non-compliant resources by type and environment
Managed identity used by the assignment
RBAC permissions required for correction
Existing exemptions and justification
Change window and application owner

Expected evidence
List of targeted resources
Sample of excluded or exempt resources
Role assignments for the remediation identity
Limited first batch and result
Activity, Policy and ARM error logs
Decision: continue, limit, fix the policy or roll back

The useful question is not “how many resources are non-compliant?”. It is “which resources will be changed if this remediation runs now?”.

Identify the real policy effect

Policies do not remediate in the same way. Audit or Deny can report or block, but they do not correct an existing resource. Remediation mostly applies to policies that can apply or deploy something, such as DeployIfNotExists or Modify.

ASSIGNMENT_NAME=deploy-diagnostics-to-law
SCOPE=/subscriptions/00000000-0000-0000-0000-000000000000

az policy assignment show --name "$ASSIGNMENT_NAME" --scope "$SCOPE" --query '{
  name:name,
  displayName:displayName,
  scope:scope,
  policyDefinitionId:policyDefinitionId,
  identity:identity,
  enforcementMode:enforcementMode,
  parameters:parameters
}' -o jsonc

az policy state summarize --policy-assignment "$ASSIGNMENT_NAME" --scope "$SCOPE" -o table

If the assignment has no managed identity while the policy needs to deploy or modify resources, remediation will fail or cannot run. If enforcementMode was intentionally disabled to observe before enforcing, the runbook should record that decision before correcting anything.

Build the target list before execution

Remediation should start from a readable target list. A global compliance summary is not enough. You need to see resources, types, groups, affected environments and current exemptions.

ASSIGNMENT_ID=$(az policy assignment show --name "$ASSIGNMENT_NAME" --scope "$SCOPE" --query id -o tsv)

az policy state list --filter "PolicyAssignmentId eq '$ASSIGNMENT_ID' and ComplianceState eq 'NonCompliant'" --query '[].{
  resourceId:resourceId,
  resourceType:resourceType,
  complianceState:complianceState,
  policyDefinitionAction:policyDefinitionAction,
  policyDefinitionReferenceId:policyDefinitionReferenceId
}' -o table

This list also exposes anomalies: a sandbox subscription inside the production scope, an unexpected resource type, an old assignment still active, or a missing exemption for a temporarily accepted case.

Verify the remediation identity

Remediations often fail because the assignment’s managed identity does not have the required roles on the right scope. Granting Owner by default is rarely defensible. The role should match the real action: create a diagnostic setting, add a tag, deploy an extension, enable a configuration or link a resource.

PRINCIPAL_ID=$(az policy assignment show --name "$ASSIGNMENT_NAME" --scope "$SCOPE" --query identity.principalId -o tsv)

az role assignment list --assignee "$PRINCIPAL_ID" --all --query '[].{role:roleDefinitionName,scope:scope}' -o table

For a policy that deploys diagnostic settings to a Log Analytics Workspace, the identity may need permissions on the target resources and on the destination workspace. For a Modify policy, it must be allowed to write the affected field. Test that permission before widening the batch.

Start with a limited batch

The first batch should be intentionally small. It verifies permissions, policy behavior, logging and the absence of unexpected effects. If the platform uses environment tags, domain-specific resource groups or subscriptions per product line, the first batch should follow that boundary.

REMEDIATION_NAME=remediate-diagnostics-prod-batch-01
RESOURCE_GROUP=rg-prod-orders

az policy remediation create --name "$REMEDIATION_NAME" --policy-assignment "$ASSIGNMENT_ID" --resource-group "$RESOURCE_GROUP" --resource-discovery-mode ReEvaluateCompliance --parallel-deployments 5 --failure-threshold 0.1

az policy remediation show --name "$REMEDIATION_NAME" --resource-group "$RESOURCE_GROUP" --query '{provisioningState:provisioningState,createdOn:createdOn,lastUpdatedOn:lastUpdatedOn}' -o jsonc

The point is not to move slowly for its own sake. The point is to make the first correction interpretable. If the batch fails, the team can tell whether the issue sits in the policy, identity, scope or one specific resource.

Watch errors through Activity Log and Policy states

A remediation that fails clearly is still usable. A remediation that fails without readable logs should not be expanded. Read compliance states after reevaluation and ARM operations generated by the identity.

let Window = 4h;
AzureActivity
| where TimeGenerated > ago(Window)
| where OperationNameValue has_any ("Microsoft.PolicyInsights/remediations", "Microsoft.Authorization/policyAssignments")
 or Caller has "Policy"
| project TimeGenerated,
        OperationNameValue,
        ActivityStatusValue,
        ResourceGroup,
        ResourceProviderValue,
        ResourceId,
        Caller,
        CorrelationId,
        Properties
| order by TimeGenerated desc

Available Log Analytics tables vary by configuration. The intent is stable: find the operation, target resource, caller identity, status and error message. AuthorizationFailed points to RBAC. An invalid template points to the policy. An unsupported resource points to an exclusion or a more precise condition.

Treat exemptions as decisions, not omissions

Exemptions should not become a grey zone. Some are legitimate: a temporary application constraint, incompatible service, migration in progress, or a security-approved exception. Others hide a policy that is too broad. Before a wider batch, review exemptions and their expiry dates.

az policy exemption list --scope "$SCOPE" --query '[].{
  name:name,
  displayName:displayName,
  exemptionCategory:exemptionCategory,
  expiresOn:expiresOn,
  policyAssignmentId:policyAssignmentId
}' -o table

Production remediation should respect decided exemptions. If an exemption blocks a correction that is now required, make the decision explicit: remove the exemption, replace it with a narrower one, or adjust the policy so it targets better.

Decide expansion, pause or rollback

After the first batch, write the decision down. Continuing without reading the result is just blind automation.

Decision: expand
The limited batch is complete
Target resources match the expected scope
The identity has sufficient permissions
Errors are absent or understood
Logs can reconstruct the action

Decision: pause
Unexpected resources appear
Exemptions are missing or ambiguous
The failure rate exceeds the accepted threshold
An application team must validate a side effect

Decision: rollback or fix
The policy modifies an unintended field
The DeployIfNotExists template is incorrect
The identity was granted overly broad permissions
A critical resource is degraded after correction

Rollback depends on the effect. For a tag or diagnostic setting, it may mean restoring the previous configuration. For a DeployIfNotExists deployment, you need to know which resources were created and whether they can be removed without losing logs or dependencies. For an overly broad policy, the real rollback is often a definition or parameter fix, followed by a fresh evaluation.

Conclusion

Azure Policy remediation is useful when it remains explainable. The runbook should turn global non-compliance into a bounded change: real effect, exact scope, remediation identity, target list, first batch, logs, exemptions and decision.

The operational decision then becomes simple: expand when the first batch is clean, pause when scope or exemptions are unclear, fix the policy when the effect is not the expected one, and roll back when a modification creates production impact. Governance stays automated, but it stays explainable.