Infrastructure
Azure Key Vault: diagnose secret references before rotation
A production runbook for qualifying a Key Vault reference failure with managed identity, RBAC, secret version, diagnostics, KQL, validation and rollback before rotating or broadening access.
An application can fail in production even when the secret still exists. A Key Vault reference may point to a disabled version, a managed identity may have changed, an RBAC assignment may not have propagated yet, a network rule may block the vault, or a rotation may have replaced the value before the runtime refreshed its configuration. The visible symptom is often a 403, an empty value, a stale secret or a restart loop. The unsafe reflex is to rotate the secret again or grant broader access.
The use case is an Azure App Service or Azure Functions API that reads secrets from Key Vault through application references, an SDK call or an automation configuration. After a deployment, a rotation or an identity change, the application no longer gets the expected value. The runbook goal is to decide whether to fix the reference, restore a version, repair identity, wait for propagation, roll back configuration or stop the rotation.
Freeze the secret-runtime contract
Start with the expected contract. A production secret is not only a value. It is a name, version, consumer, runtime identity, read method, rotation window and rollback path.
Secret-runtime contract
Application: api-billing-prod
Vault: kv-prod-platform
Secret: billing-db-password
Expected version: latest or explicit version
Consumer: App Service, Function, worker, pipeline or runbook
Read method: Key Vault reference, SDK, injected variable or automation
Runtime identity: system-assigned or user-assigned managed identity
Last known rotation
Last valid version
Rollback: previous version, previous slot, previous configuration or secondary secret
Questions before action
Which identity really reads the secret?
Does the reference target latest or a pinned version?
Is the current version enabled and not expired?
Did the vault receive a request from the runtime?
Are permissions, network and runtime cache separated?If the team cannot name the real identity and the expected version, another rotation only adds drift instead of resolving the incident.
Separate reference, identity and network
Three failure families can look identical from the application. A malformed reference never requests the secret. An identity without permission receives a denial. A blocked network path leaves different evidence in Key Vault. Avoid fixing every layer at once.
APP_RG="rg-prod-app"
APP_NAME="app-billing-prod"
VAULT_NAME="kv-prod-platform"
SECRET_NAME="billing-db-password"
az webapp config appsettings list --resource-group "$APP_RG" --name "$APP_NAME" --query "[?contains(value, '@Microsoft.KeyVault')].{name:name,value:value}" --output table
az keyvault secret show --vault-name "$VAULT_NAME" --name "$SECRET_NAME" --query "{id:id,enabled:attributes.enabled,expires:attributes.expires,created:attributes.created,updated:attributes.updated}" --output json
az webapp identity show --resource-group "$APP_RG" --name "$APP_NAME" --output jsonFor Functions, Container Apps or a worker, adapt the command to the runtime. The important part is to capture the reference, secret and identity at the same point in time.
Check the version before rotating
A Key Vault reference can target SecretUri without a version or an explicit version. Both patterns can be valid, but they fail differently. With a pinned version, rotation will not be consumed until the reference changes. With latest, a disabled, expired or invalid version can break the runtime.
VAULT_NAME="kv-prod-platform"
SECRET_NAME="billing-db-password"
az keyvault secret list-versions --vault-name "$VAULT_NAME" --name "$SECRET_NAME" --query "[].{id:id,enabled:attributes.enabled,expires:attributes.expires,created:attributes.created,updated:attributes.updated}" --output table
az keyvault secret show --vault-name "$VAULT_NAME" --name "$SECRET_NAME" --query "{id:id,valuePresent:value != null,enabled:attributes.enabled,expires:attributes.expires}" --output jsonDo not read the clear-text value to prove the incident. Useful evidence is the version state, identifier, date and accessibility from the production identity.
Correlate Key Vault and runtime logs
The vault should tell whether the request arrived, who sent it and why it failed. The application logs should tell whether configuration was resolved, cached, expired or missing.
let StartTime = datetime(2026-07-05T08:00:00Z);
let EndTime = datetime(2026-07-05T09:00:00Z);
let VaultName = "kv-prod-platform";
let SecretName = "billing-db-password";
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource has VaultName
| where OperationName has_any ("SecretGet", "SecretList")
| extend secretName = tostring(parse_url(tostring(id_s)).Path)
| where id_s has SecretName or requestUri_s has SecretName
| project TimeGenerated,
OperationName,
ResultType,
ResultSignature,
CallerIPAddress,
identity_claim_appid_g,
identity_claim_oid_g,
requestUri_s,
CorrelationId
| order by TimeGenerated ascIf no Key Vault event matches the runtime, inspect the reference, cache, network path or environment variable first. If the vault returns Forbidden, focus on identity, RBAC or access policy. If the vault returns NotFound, check the name, version and environment.
Test with the real identity
A test from an administrator workstation does not validate the production path. The read must be reproduced with the identity that runs the application, or with equivalent evidence in the logs.
Identity checks
Expected system-assigned or user-assigned identity is enabled
Configured client ID matches the identity used by the runtime
Key Vault Secrets User role or access policy covers the right vault
RBAC propagation has had enough time
No recent change replaced the slot, Function or worker identity
Key Vault logs show the same object ID as the runtime
Block broader permissions when
The real identity is unknown
The test was done with a human account
Several identities are attached but none is explicitly selected
The vault receives no request from the runtime
The requested role goes beyond reading the required secretThe goal is to avoid the broad role that happens to work. The correct fix grants only the read path needed by the expected consumer.
Validate cache, restart and slot behavior
Even when Key Vault is healthy, the application may keep the old value. App Service references, injected variables, slots and some workers do not all reload secrets at the same time. The decision must account for runtime behavior instead of multiplying rotations.
validation:
target: api-billing-prod
secret: billing-db-password
expected_version: latest_valid_version
checks:
- key_vault_logs_show_secret_get_from_runtime_identity
- runtime_logs_show_configuration_refresh_or_restart
- application_health_check_passes_with_correlation_id
- no_new_forbidden_or_notfound_errors
- previous_secret_version_still_available_for_rollback
controlled_actions:
- refresh_configuration_if_supported
- restart_one_instance_or_slot_if_required
- swap_back_only_if validation_fails
stop_when:
- new version breaks dependency authentication
- runtime still reads old version after documented refresh
- Key Vault denies production identity
- rollback version is disabled or expiredFor a blue-green slot flow, validate the slot that will receive traffic. A correct reference in staging does not prove production will use the same identity or version after swap.
Decide fix, rollback or delayed rotation
Keep the decision limited to the proven defect. A reference failure is not always a secret problem.
Fix the reference
Secret name, vault or target version is wrong
Reference points to a disabled or obsolete version
Production slot does not carry the same configuration as the validated slot
Repair identity or RBAC
Key Vault receives the runtime request
ResultType shows Forbidden or equivalent
Object ID matches an expected identity without enough permission
Target role stays limited to secret read
Wait or force a controlled refresh
Secret is readable by the production identity
Reference is correct
Runtime keeps a cached value
Restart or refresh is tested on a bounded scope
Roll back
New version breaks the downstream dependency
Rotation removed the last working version too early
Identity or slot change broke several consumers
Previous configuration restores service and remains auditable
Delay rotation
Real identity is not proven
Key Vault logs are unavailable
Secret rollback is not valid
Downstream consumer cannot confirm acceptance of the new valueA successful rotation is not the creation of a new value. It is proof that expected consumers read the right version and that the old one can be retired without breaking the service.
Conclusion
Diagnosing a Key Vault reference means treating the secret as a full production dependency: reference, version, identity, permissions, network, runtime cache, logs and rollback. Until those pieces are separated, rotating the secret or broadening access hides the cause.
The reliable decision is to fix the proven layer, validate with the real identity, keep a return version and remove temporary permissions. Key Vault then remains a security control, not an opaque dependency that breaks deployments at the worst time.