Azure Key Vault: recover a deleted secret before recreating it

An application starts returning SecretNotFound. The expected secret has disappeared from Key Vault even though the deployment worked minutes ago. Recreating the same name from a CI variable looks fast, but it can propagate an unverified value, break version-pinned references and hide the deletion that started the incident.

The production case is a secret shared by three consumers: an application uses an unversioned URI, a job pins a version, and an APIM policy resolves the same object. The runbook must prove deletion, recover the soft-deleted object through a dedicated identity, and validate each contract. Purge and recreation stay outside the emergency path.

Freeze the incident before writing

Record the logical name, referenced versions and consumer scope. Never copy the secret value into the incident.

incident:
detected_at_utc: 2026-08-18T14:12:00Z
vault: kv-platform-prod
secret: payment-api-token
symptom: SecretNotFound
last_known_good_utc: 2026-08-18T13:55:00Z

consumers:
- name: checkout-api
  reference: unversioned
- name: settlement-job
  reference: version-pinned
- name: apim-policy
  reference: named-value

guardrails:
expose_secret_value: false
purge_allowed: false
recreate_allowed: false
recovery_requires_change_record: true

A pinned consumer and an unversioned consumer do not validate the same contract. The former expects an immutable identifier; the latter resumes from the latest active version.

Prove the soft-deleted state

Use a diagnostic identity without purge rights. Check the vault, the active object and then the deleted object.

set -euo pipefail

SUBSCRIPTION="<subscription-id>"
VAULT="kv-platform-prod"
SECRET="payment-api-token"

az account set --subscription "$SUBSCRIPTION"
az keyvault show --name "$VAULT" --query "{id:id,softDelete:properties.enableSoftDelete,purgeProtection:properties.enablePurgeProtection,retentionDays:properties.softDeleteRetentionInDays,rbac:properties.enableRbacAuthorization}" --output json

az keyvault secret show --vault-name "$VAULT" --name "$SECRET" --query "{id:id,enabled:attributes.enabled,updated:attributes.updated}" --output json || true

az keyvault secret show-deleted --vault-name "$VAULT" --name "$SECRET" --query "{name:name,recoveryId:recoveryId,deletedDate:deletedDate,scheduledPurgeDate:scheduledPurgeDate}" --output json

If the object is active, investigate its name, version, caller identity or access path. If it is deleted, recovery remains possible during retention. If it exists in neither state, stop: creating a value without an authoritative source is a security rotation, not a recovery.

Purge protection does not recover the object. It blocks permanent deletion before retention expires and prevents a privileged identity from turning a recoverable incident into irreversible loss.

Reconstruct the deletion event

Correlate Key Vault diagnostic logs with the suspected change, pipeline or runbook.

let Vault = "kv-platform-prod";
let Secret = "payment-api-token";
let WindowStart = datetime(2026-08-18T13:30:00Z);
let WindowEnd = datetime(2026-08-18T15:30:00Z);
AzureDiagnostics
| where TimeGenerated between (WindowStart .. WindowEnd)
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where Resource has Vault
| where OperationName has_any ("SecretDelete", "SecretRecover", "SecretGet")
| where requestUri_s has Secret
| project TimeGenerated, OperationName, ResultType, ResultSignature,
        identity_claim_appid_g, identity_claim_oid_g,
        CallerIPAddress, requestUri_s, correlationId_g
| order by TimeGenerated asc

Columns vary by collection mode. Preserve the evidence chain: UTC time, operation, result, identity, source address, URI and correlation ID. Missing logs become a corrective action; they do not justify an invented root cause.

Prepare a bounded recovery

The emergency identity receives the relevant recover permission without purge rights. The vault authorization model determines whether Azure RBAC or a legacy access policy must be checked.

Before execution, confirm:

• the vault, object name, deletion window and scheduled purge date;
• every consumer and its expected version;
• no concurrent rotation or recreation;
• a second-person review of the operation;
• written stop conditions.

Do not combine recovery with a new value, broader network access or a redeployment. One action, one hypothesis, one validation.

Recover without reading the value

set -euo pipefail

VAULT="kv-platform-prod"
SECRET="payment-api-token"

az keyvault secret recover --vault-name "$VAULT" --name "$SECRET" --output none

for attempt in 1 2 3 4 5 6; do
if az keyvault secret show --vault-name "$VAULT" --name "$SECRET" --query "{id:id,enabled:attributes.enabled,created:attributes.created,updated:attributes.updated}" --output json; then
  exit 0
fi
sleep 5
done

echo "recovery_not_visible_after_bounded_wait" >&2
exit 1

The wait is bounded. Never fall through to automatic creation: recovery can be asynchronous, a permission may be missing, or the selected vault may be wrong.

Validate consumers

Seeing an active secret does not prove production is healthy. Test with real runtime identities and paths without logging the value.

Vault
Object active with enabled=true
Expected versions visible in metadata
No new Delete or Purge event

Consumers
Unversioned URI readable with runtime identity
Pinned version still present and accessible
APIM reference resolves without a policy change
SecretNotFound returns to normal
No new 401, 403 or retry storm

Stop
Expected version missing or disabled
Deleting identity still active
Value may be compromised
Business side effect uses an unverified credential

Prefer a health endpoint that confirms access without revealing the secret. Otherwise, the probe should log only status, requested version and a correlation ID.

Decide whether to keep, rotate or roll back

Keep the recovery when pinned and unversioned references work, errors disappear, and the deletion source is contained. Then correct the identity or automation that caused the event.

Start a separate rotation when the value may have been exposed, its integrity is no longer defensible, or the target provider revoked it. That rotation needs its own dual-value, consumer cutover and revocation plan.

Roll back when the wrong version becomes active or a consumer creates an unexpected side effect: isolate the consumer first and restore its previous reference. Never use purge as rollback. If the object must be removed again, keep it in the soft-delete lifecycle so it remains recoverable during retention.

Conclusion

A deleted secret is a lifecycle incident, not an invitation to recreate a value as fast as possible. The team must distinguish the active object, soft-deleted object, consumed version, deleting identity and actual impact.

The healthy outcome is an evidence-backed decision: recover the existing object, validate each reference, keep the restoration or begin an independent rotation. Purge stays outside emergency handling, and rollback protects consumers without erasing evidence.