Infrastructure
Azure App Service: diagnose a Managed Identity token failure before bringing back a client secret
A production runbook for qualifying an Azure App Service Managed Identity failure with IMDS endpoint, Entra ID, RBAC, Key Vault or target API evidence, logs, validation and rollback before reintroducing a client secret.
When an Azure App Service application can no longer obtain a token through Managed Identity, the fastest recovery idea is often to put a client secret back into configuration. It may restore the call, but it also turns an operations incident into security debt: a secret to store, rotate, monitor, scope and eventually remove, while the original identity failure remains unclear.
The use case is an API running on App Service that calls Key Vault, Storage, Microsoft Graph or an internal API protected by Entra ID. After a deployment, identity rotation, RBAC change or slot swap, the application starts returning 401, 403, CredentialUnavailableException, ManagedIdentityCredential authentication unavailable or token acquisition timeouts. The runbook goal is to decide whether to fix the App Service identity, the target authorization scope, the credential chain, the target service, or use a tightly bounded temporary workaround.
Freeze the expected identity contract
Start by writing down the identity that should have acted. A managed identity is not only a portal toggle. It defines an Entra principal, a calling resource, a token audience, an authorization scope and an audit trail on the target service.
Application
App Service: app-orders-prod
Slot: production or staging
Resource group: rg-prod-app
Runtime: production code, WebJob or Function in the same plan
Expected identity
Type: system-assigned or user-assigned
Expected client ID when user-assigned
Expected object ID / principal ID
Token audience: https://vault.azure.net, https://storage.azure.com or api://...
Target resource: Key Vault, Storage, internal API, Graph or third-party service
Expected permission: RBAC, access policy, app role or API scope
Questions before bringing back a client secret
Is the identity still enabled on the right slot ?
Does the code request a token for the right audience ?
Does the principal have access on the exact target scope ?
Is the target rejecting the identity or is no token being issued ?
Did a slot, role assignment, Key Vault, API or SDK change just happen ?If this contract is not explicit, a client secret will hide the outage without explaining which runtime identity was supposed to own the action.
Separate token acquisition from target authorization
A 403 from Key Vault or an API does not prove Managed Identity is broken. It may mean the token was issued correctly, but the target rejected the principal, role, scope, audience or network condition.
Failure families
Token unavailable
Managed Identity endpoint unreachable
Identity disabled on App Service or on the slot
Wrong client ID for a user-assigned identity
Credential chain ignores ManagedIdentityCredential
Token issued but target rejects the call
Role assignment missing or at the wrong scope
Key Vault access policy still in use
App role not assigned on the internal API
Token audience is wrong
RBAC propagation or service-side cache delay
Application or network failure
App calls the wrong URL
DNS or route to private API fails
TLS, proxy or firewall breaks the path
Code reuses an expired token or bad cache entry
Blocker
Reintroducing a client secret before locating the failure familyThis separation avoids two common mistakes: widening target permissions while no token is being requested, or adding a secret when only a target-side role is missing.
Capture the identity exposed by App Service
First verify the real application and slot state. Incidents often follow a swap, a user-assigned identity change, or an IaC script that recreated the identity but did not restore the role assignments.
SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
RG="rg-prod-app"
APP="app-orders-prod"
SLOT="production"
az account set --subscription "$SUBSCRIPTION"
az webapp identity show --resource-group "$RG" --name "$APP" --query "{type:type,principalId:principalId,tenantId:tenantId,userAssignedIdentities:userAssignedIdentities}" --output json
az webapp config appsettings list --resource-group "$RG" --name "$APP" --query "[?contains(name, 'IDENTITY') || contains(name, 'AZURE_CLIENT_ID') || contains(name, 'KEYVAULT') || contains(name, 'AUTH')].[name,value]" --output tableWhen the application uses a user-assigned identity, confirm that AZURE_CLIENT_ID points to the intended client ID. After a swap, slot settings and attached identities can drift independently.
Test the token from the same runtime
Testing from an administrator laptop only proves the administrator path. The useful test runs from the same App Service, with the same variables, slot and credential chain.
# Run from an App Service console, a controlled diagnostic endpoint
# or a temporary revision that uses the same slot and identity.
TOKEN_ENDPOINT="$IDENTITY_ENDPOINT"
TOKEN_HEADER="$IDENTITY_HEADER"
RESOURCE="https://vault.azure.net"
curl -sS -H "X-IDENTITY-HEADER: $TOKEN_HEADER" "$TOKEN_ENDPOINT?api-version=2019-08-01&resource=$RESOURCE" | jq '{expires_on, resource, client_id, access_token_present: (.access_token != null)}'The expected result is not to print the full token. It is to prove the audience, client ID and endpoint availability. If this fails, fix the App Service identity before changing target permissions.
Read denials on Entra, Key Vault or the API
When a token is issued, evidence must come from the target. For Key Vault, Storage, API Management, an internal API or Microsoft Graph, look for the principal, audience and rejection reason in the available logs.
let StartTime = datetime(2026-07-16T08:00:00Z);
let EndTime = datetime(2026-07-16T09:00:00Z);
let AppPrincipalId = "00000000-0000-0000-0000-000000000000";
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where ResourceProvider has_any ("MICROSOFT.KEYVAULT", "MICROSOFT.APIMANAGEMENT", "MICROSOFT.STORAGE")
| where ResultType has_any ("401", "403", "Forbidden", "Unauthorized")
or OperationName has_any ("SecretGet", "SecretList", "GetBlob", "GatewayLogs")
| extend Raw = tostring(Properties)
| where Raw has AppPrincipalId or Identity has AppPrincipalId or CallerIPAddress != ""
| project TimeGenerated, ResourceProvider, Resource, OperationName, ResultType, ResultSignature, Identity, CallerIPAddress, Raw
| order by TimeGenerated descAdapt the query to the tables that are actually enabled. The important point is to tie the denial to the App Service principal, not only to observe an application-side 403.
Check roles at the right scope
Roles are rarely missing everywhere. They are missing at the relevant scope: exact vault, exact secret, storage account, API app registration, resource group or subscription. Also check recent removals.
PRINCIPAL_ID="00000000-0000-0000-0000-000000000000"
TARGET_SCOPE="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod-sec/providers/Microsoft.KeyVault/vaults/kv-prod"
az role assignment list --assignee "$PRINCIPAL_ID" --all --query "[].{role:roleDefinitionName,scope:scope,principalType:principalType,createdOn:createdOn}" --output table
az role assignment list --assignee "$PRINCIPAL_ID" --scope "$TARGET_SCOPE" --include-inherited --query "[].{role:roleDefinitionName,scope:scope}" --output tableIf the target still uses Key Vault access policies or Entra app roles, RBAC commands are not enough. The diagnosis must follow the authorization model that is really active.
Control the application credential chain
Many incidents come from a credential chain that behaves differently in local, staging and production. DefaultAzureCredential can try several sources. In production, the application should fail clearly when the expected identity is unavailable, instead of silently falling back to another credential.
Check in code and configuration
ManagedIdentityCredential explicitly uses the expected client ID when user-assigned
No residual client secret exists in production app settings
Logs show which credential was selected
Token errors are not hidden by endless application retries
Token cache is invalidated after identity or slot changes
Startup tests call a representative dependency with the runtime identity
Drift signals
AZURE_CLIENT_SECRET reappears in configuration
AZURE_CLIENT_ID points to a staging identity
Production and staging slots do not have the same identities
Logs do not distinguish token acquisition from target callA client secret workaround often becomes permanent because it is invisible. The durable fix starts by making the credential chain observable.
Decide fix, workaround or rollback
The decision should state which evidence is still missing and which risk surface is accepted. A client secret is acceptable only as a bounded temporary workaround, with an owner, expiry and removal change.
decision:
fix_app_service_identity:
when:
- managed_identity_endpoint_fails_from_runtime
- identity_disabled_on_app_or_slot
- wrong_user_assigned_client_id
validation:
- token_probe_returns_expected_client_id
- app_restart_or_slot_swap_keeps_identity
fix_target_authorization:
when:
- token_is_issued_for_expected_audience
- target_logs_show_401_or_403_for_app_principal
- role_assignment_or_app_role_is_missing
validation:
- target_accepts_runtime_identity
- audit_log_contains_app_principal
rollback_recent_change:
when:
- identity_or_role_changed_in_recent_deployment
- slot_swap_changed_effective_identity
- no_safe_equivalent_signal_exists
validation:
- previous_identity_path_works
- removed_change_is_documented
temporary_client_secret:
allowed_only_if:
- user_impact_is_active
- managed_identity_fix_needs_external_delay
- secret_is_scoped_to_same_or_narrower_permissions
- expiry_owner_and_removal_change_are_created
blocked_if:
- root_cause_is_unknown
- secret_would_need_broader_permissions
- audit_would_no_longer_identify_the_applicationA secret workaround without a removal date is an architecture regression. Treat it as a production exception, not a new default.
Validate before closure
Incident closure must prove three things: the runtime obtains the expected token, the target accepts the principal, and the application runs without a residual client secret.
Final validation
Token obtained from production slot with the expected client ID
Token audience matches the target
Role, access policy or app role confirmed at the right scope
Target log shows App Service principal and successful operation
Production app settings contain no temporary client secret
Representative application test passes after restart or swap
Alert or KQL query added to detect client secret reintroduction
Rollback ready
Previous identity documented
Previous role assignment known
Temporary secret expired or removed
IaC change aligned with the corrected stateIf the fix was applied manually in the portal, it remains incomplete until IaC, the runbook and monitoring reflect the expected state.
Conclusion
A Managed Identity failure does not automatically justify bringing back a client secret. First locate the break: identity exposed by App Service, token audience, credential chain, target authorization, audit logs or slot change.
The right runbook restores Managed Identity when possible, fixes target scope when the token is valid, rolls back the recent change when production is exposed, and allows a client secret only as a temporary, bounded and monitored workaround. The final decision should leave the application operable without losing control of its identity.