Infrastructure

Azure Key Vault: migrate access policies to Azure RBAC without breaking workloads

A production runbook for inventorying Key Vault access, staging data-plane roles, cutting over to Azure RBAC with real probes, validating workloads and rolling back without broadening permissions.

19 Sept 2026 azurekey-vaultrbacidentitysecurityobservabilityautomationrunbookrollbackproduction

An Azure Key Vault has relied on access policies for years. Security wants Azure RBAC so that vault administration and access to secrets are governed separately. The team creates role assignments, changes the permission model, and several applications immediately return 403. Granting Key Vault Administrator to every failing identity may restore service, but it hides missed consumers and leaves excessive data-plane access behind.

The running case is a vault used by a Function App that reads secrets, a pipeline that renews a certificate, and an on-call group that inspects metadata. This runbook prepares the migration, performs a bounded cutover, observes access from the real runtimes, and reaches one decision: keep Azure RBAC, correct one precise role, or temporarily return to access policies.

Freeze the authorization contract

Capture the vault state before changing it. The evidence pack needs the active authorization model, access policies, network settings, diagnostic settings, direct and inherited role assignments, and the identities expected to use the vault. Networking stays out of scope during this change: a 403 after cutover is not a reason to open the firewall or public access.

bash 01-snapshot-key-vault-access.sh
set -euo pipefail

RG="rg-platform-prod"
VAULT="kv-orders-prod"
VAULT_ID=$(az keyvault show -g "$RG" -n "$VAULT" --query id -o tsv)

az keyvault show -g "$RG" -n "$VAULT" --query '{id:id,rbac:properties.enableRbacAuthorization,tenant:properties.tenantId,accessPolicies:properties.accessPolicies,networkAcls:properties.networkAcls}' -o json > key-vault-before.json

az role assignment list --scope "$VAULT_ID" --include-inherited --all -o json > role-assignments-before.json

az monitor diagnostic-settings list --resource "$VAULT_ID" -o json > diagnostic-settings-before.json

Tie the snapshot to a change ID and UTC timestamp. Also prove that the operator separately holds Microsoft.Authorization/roleAssignments/write and Microsoft.KeyVault/vaults/write. Permission to update the vault does not prove permission to create the required role assignments.

Inventory the consumers, not only the policies

An access policy identifies a principal and permissions, but not the service using them or its criticality. Resolve every objectId to a user, group, service principal, or managed identity. Also find identities absent from the policies that already inherit RBAC access from the resource group or subscription.

yaml key-vault-consumers.yml
vault: kv-orders-prod
consumers:
- name: orders-function-prod
  principal_id: 11111111-1111-1111-1111-111111111111
  principal_type: managed_identity
  operations: [secrets/get]
  probe: read-secret-version
  criticality: critical
- name: certificate-renewal-pipeline
  principal_id: 22222222-2222-2222-2222-222222222222
  principal_type: service_principal
  operations: [certificates/get, certificates/import]
  probe: import-test-certificate
  criticality: scheduled
- name: platform-oncall
  principal_id: 33333333-3333-3333-3333-333333333333
  principal_type: group
  operations: [secrets/list-metadata]
  probe: list-secret-metadata
  criticality: interactive

unknown_policies: []
unowned_principals: []
validation_owner: platform-team

Do not migrate an unknown principal by granting a broad role “to be safe.” Treat it as a stop condition, identify the owner, and inspect historical activity. An unused policy can be removed later through a separate change.

Translate operations into data-plane roles

Map required operations, not job titles. An application reading a secret value will usually need Key Vault Secrets User; a pipeline managing secrets may need Key Vault Secrets Officer; keys and certificates use their own role families. Key Vault Reader exposes metadata, not secret values. Key Vault Contributor manages the vault on the control plane but does not automatically grant data-plane access.

Built-in roles do not reproduce every fine-grained access policy. When none covers the requirement without excess permission, use a reviewed and versioned custom role. Prefer vault scope for an application’s vault. Object-level scope can be useful, but it creates more assignments to operate and should not compensate for a vault that mixes unrelated applications.

bash 02-stage-key-vault-rbac.sh
VAULT_ID=$(az keyvault show -g "$RG" -n "$VAULT" --query id -o tsv)

az role assignment create --assignee-object-id "11111111-1111-1111-1111-111111111111" --assignee-principal-type ServicePrincipal --role "Key Vault Secrets User" --scope "$VAULT_ID"

az role assignment create --assignee-object-id "22222222-2222-2222-2222-222222222222" --assignee-principal-type ServicePrincipal --role "Key Vault Certificates Officer" --scope "$VAULT_ID"

Create assignments before cutover and allow time for propagation. They do not authorize the data plane while the vault still uses access policies, but they must exist, resolve to the intended principals, and survive review before the change window.

Build a validation matrix

The permission model applies to the whole vault. There is no per-identity canary on that vault: once enableRbacAuthorization becomes true, every workload depends on RBAC. Test the mapping on a representative non-production vault first, then use a short production window with probes executed by the real identities.

text rbac-cutover-matrix.txt
Principal                    Positive test                 Negative test
orders-function-prod         read canary secret            write a new version
certificate-renewal-pipeline import test certificate       read application secret
platform-oncall              list secret metadata          read a secret value

Required evidence
- functional result through the normal network path
- expected objectId in the trace
- expected operation and scope
- no emergency administrator role
- negative test denied
- reviewed rollback command and available operator

A test under an operator’s personal identity does not validate a managed identity. Run each probe from its normal execution surface: Function App, pipeline runner, or controlled administration workstation.

Cut over with explicit stop conditions

Publish the exact cutover time, pause deployments that change identities or secrets, and recheck the snapshot and role assignments. The toggle is simple. The controls surrounding it determine whether the change is safe.

bash 03-enable-rbac-and-probe.sh
az keyvault update --resource-group "$RG" --name "$VAULT" --enable-rbac-authorization true

az keyvault show -g "$RG" -n "$VAULT" --query properties.enableRbacAuthorization -o tsv

# Run probes with each real identity next.
# The operator account is not application evidence.

Stop when a critical workload fails, the observed identity differs from inventory, audit evidence is missing, or the proposed fix requires a broader role than the contract. Propagation delay can justify a short wait with bounded retries. It does not justify a string of improvised assignments.

Read denials without confusing identity and network

Key Vault can send the AuditEvent category to Log Analytics. Depending on the destination mode, events may land in a resource-specific table or in AzureDiagnostics. Adapt fields to the schema actually enabled, and inspect operation, status, principal, caller address, and cutover time together.

kusto 04-key-vault-rbac-cutover.kql
let Cutover = datetime(2026-09-19T16:15:00Z);
union isfuzzy=true AZKVAuditLogs, AzureDiagnostics
| where TimeGenerated between (Cutover - 15m .. Cutover + 45m)
| extend Vault = tostring(column_ifexists("Resource", Resource)),
       Operation = tostring(column_ifexists("OperationName", OperationName)),
       Status = tostring(column_ifexists("HttpStatusCode", ResultSignature)),
       Principal = tostring(column_ifexists("Identity", identity_claim_oid_g)),
       CallerIp = tostring(column_ifexists("CallerIpAddress", CallerIPAddress))
| where Vault has "kv-orders-prod" or _ResourceId has "/vaults/kv-orders-prod"
| summarize Calls=count(), FirstSeen=min(TimeGenerated), LastSeen=max(TimeGenerated)
by Principal, Operation, Status, CallerIp
| order by Status desc, Calls desc

A 403 for the expected principal and operation points toward role, scope, or propagation. No event at all points back toward DNS, network, the wrong vault name, or missing diagnostics. Portal success beside application failure often means that two different identities were tested.

Decide: validate, correct, or roll back

Keep Azure RBAC when every positive test passes, every negative test remains denied, critical workloads are stable, and no broad temporary role was added. Correct in place when one identified principal lacks an expected operation and a minimal role covers it. Continue observing after the window because some scheduled jobs run infrequently.

Roll back when several critical consumers are missing, the runtime principal is unknown, inherited assignments make effective access unclear, or audit evidence is unavailable. The return to the previous model must have been rehearsed outside production. Before executing it, confirm that the snapshotted access policies are still present and no concurrent change altered them.

bash 05-rollback-to-access-policies.sh
az keyvault update --resource-group "$RG" --name "$VAULT" --enable-rbac-authorization false

az keyvault show -g "$RG" -n "$VAULT" --query '{rbac:properties.enableRbacAuthorization,accessPolicies:properties.accessPolicies}' -o json

# Replay the complete matrix. A successful toggle is not service recovery.

Do not immediately delete the staged role assignments. They preserve evidence and support a later retry. Mark them as inactive for this vault, repair the inventory, and schedule a new migration instead of keeping an unexplained hybrid state.

Conclusion

Moving Key Vault to Azure RBAC is not a settings change. It migrates the data-plane authorization contract. Success depends on an identity inventory, minimal roles derived from operations, probes executed by actual runtimes, and usable audit evidence during cutover.

The final decision must be binary and defensible: keep Azure RBAC because expected access and expected denials were both validated, or temporarily restore access policies with the snapshot intact. Granting an administrator role until the 403 disappears is neither validation nor rollback.