Infrastructure
Azure Key Vault: diagnose latency and throttling before rotating secrets
A production runbook for qualifying Key Vault degradation by separating latency, throttling, identity, network path, application cache, logs and rollback before starting a secret rotation.
When an application fails to read a Key Vault secret, the risky reflex is immediate: rotate the secret, redeploy the application, widen a role or copy a fallback value locally. In production, those actions can make the incident harder to explain. If the real cause is Key Vault throttling, network latency, an unstable managed identity, an expired application cache or a firewall rule, rotating the secret fixes nothing and adds another moving part.
The use case is an Azure application that reads secrets at startup and during selected processing paths. After a deployment, errors increase: timeouts, 429, 403, SecretNotFound, slower dependencies or restarting pods. The runbook goal is to decide whether to fix access, reduce pressure, restore configuration, enable a bounded fallback or roll back the application change.
Freeze the symptom before rotation
Before changing a secret, describe the exact failure. Read latency, authorization denial, missing secret and quota pressure do not have the same correction.
Observed symptom
Application: api-checkout-prod
Key Vault: kv-shared-prod
Secret involved: payment-provider-token
First detection: 2026-07-11 08:40 UTC
Visible errors: timeout, 429, 403, SecretNotFound or TLS failure
Last change: application release, role assignment, firewall, private DNS, secret version, SDK
Questions before action
Does the secret exist in the expected version?
Is the calling identity the production identity?
Does the issue affect one secret, one vault or several applications?
Does it happen at startup, on every request or in bursts?
Is a previous configuration or secret version available for rollback?This record prevents the team from turning an access incident into a rotation incident. The secret value can be valid while the path used to read it is broken.
Separate existence, identity and permissions
The first check should be read-only. It confirms that the secret, its version and the caller identity still exist in the expected scope.
VAULT="kv-shared-prod"
SECRET="payment-provider-token"
APP_IDENTITY_CLIENT_ID="00000000-0000-0000-0000-000000000000"
az keyvault secret show --vault-name "$VAULT" --name "$SECRET" --query "{name:name, enabled:attributes.enabled, created:attributes.created, updated:attributes.updated, expires:attributes.expires, version:id}" --output json
az role assignment list --assignee "$APP_IDENTITY_CLIENT_ID" --scope "$(az keyvault show --name "$VAULT" --query id --output tsv)" --query "[].{role:roleDefinitionName, scope:scope}" --output table
az keyvault show --name "$VAULT" --query "{name:name, publicNetworkAccess:properties.publicNetworkAccess, enableRbacAuthorization:properties.enableRbacAuthorization, networkAcls:properties.networkAcls}" --output jsonA 403 should be handled as an identity or policy issue, not as proof that the secret value is wrong. A SecretNotFound should verify name, casing, version, slot and environment before anyone recreates a value manually.
Measure latency and throttling
Key Vault may answer, but too slowly or with 429 responses. In that case, the diagnosis is about call volume, retries, cache behavior, simultaneous startups and quotas, not about the secret value.
let Window = 2h;
AzureDiagnostics
| where TimeGenerated > ago(Window)
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName has_any ("SecretGet", "SecretList")
| summarize
Calls = count(),
Throttled = countif(httpStatusCode_d == 429),
Forbidden = countif(httpStatusCode_d == 403),
NotFound = countif(httpStatusCode_d == 404),
P95LatencyMs = percentile(DurationMs, 95)
by bin(TimeGenerated, 5m), identity_claim_appid_g, OperationName, requestUri_s
| order by TimeGenerated ascIf the table or columns differ because of Diagnostic Settings, keep the same logic: group by time bucket, operation, identity, HTTP status and duration. The signal you need is a burst or a clear regression after a change.
Check the network without making it the whole story
Private Endpoint, firewall and private DNS can be involved in Key Vault access, but they are only part of the path. Prove whether the application reaches the right FQDN, through the right resolver, with the expected certificate and status.
VAULT_FQDN="kv-shared-prod.vault.azure.net"
getent hosts "$VAULT_FQDN" || nslookup "$VAULT_FQDN"
timeout 5 bash -lc "cat </dev/null >/dev/tcp/$VAULT_FQDN/443" && echo "tcp_connect_ok=true" || echo "tcp_connect_ok=false"
openssl s_client -connect "$VAULT_FQDN:443" -servername "$VAULT_FQDN" </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer
curl -sS -o /dev/null -w "status=%{http_code} time=%{time_total}s
" "https://$VAULT_FQDN/secrets?api-version=7.4"A 401 can be healthy for an unauthenticated probe: it proves the service responds. A timeout, unexpected resolution or TLS error points toward DNS, firewall, proxy or inspection. Private Endpoint then clarifies the path; it should not hide an identity problem.
Inspect application behavior
Many Key Vault incidents come from aggressive application behavior: reading on every request, missing cache, retry loops without backoff, simultaneous restarts across many instances or cache invalidation after deployment.
Application-side checks
Secret read at startup or on every request
Local cache and expiration policy
Retry with backoff and jitter
Number of instances restarting at the same time
SDK version and configured timeout
Controlled fallback when Key Vault is temporarily slow
Logs with Key Vault request ID and caller identity
Block rotation when
429 responses increase after scale-out
The same secret is read multiple times per request
Cache was disabled by a release
Errors disappear when concurrency is reducedRotating a secret under load can force even more reads and amplify throttling. The correction may be cache restoration, progressive warm-up, backoff or application rollback.
Decide fix, pressure reduction or rollback
Keep the decision bounded. A 429 is not fixed with a wider role, and a 403 is not fixed with a secret rotation.
Fix identity or permissions
Errors are 403
The observed identity is not the expected one
Role assignment or access policy changed recently
The correction is scoped to the required vault or secret
Reduce pressure
Errors are 429 or bursty timeouts
Reads increased after scale-out or redeployment
Cache or backoff is missing
Validation passes after concurrency is limited
Fix network or DNS
The workload probe cannot reach vault.azure.net
Resolution points to an unexpected path
Key Vault logs do not see the application requests
The correction is a documented route, firewall rule or DNS zone
Roll back
A release changed cache, SDK, timeout or secret name
The previous version restores reads without changing the secret
Rotation would add unnecessary risk during the incident
Rotate the secret
The secret is compromised, expired or invalid at the target provider
Key Vault access is healthy
Propagation and rollback plans are readySecret rotation becomes a security and business decision, not a diagnostic gesture. If the read path is the problem, stabilize that path before changing the value.
Validate after correction
Validation must prove that the application reads the secret cleanly and that the next rotation remains possible.
Minimum validation
Secret read with the expected production identity
429 rate back to zero or normal baseline
Read P95 acceptable for application startup
No remaining 403 or 404 on the affected secret
Cache and backoff confirmed in active configuration
Application logs linked to Key Vault request IDs
No secret value copied into a file or durable variable
Clean rollback
Restore previous application version or configuration
Replay one controlled secret read
Keep before/after Key Vault status evidence
Open follow-up work if cache, retries or Diagnostic Settings are missingIf a temporary firewall, role or code exception was added, it needs an owner and removal date. Otherwise the incident becomes permanent drift.
Conclusion
A Key Vault read failure is not automatically a secret failure. The diagnosis should separate existence, identity, permission, network path, latency, throttling, application cache and recent change.
The healthy outcome is a verifiable decision: fix a permission, restore the network path, reduce pressure, roll back a release or rotate the secret only when the value is truly at fault. That discipline prevents an observable degradation from becoming a risky rotation that nobody can explain later.