Infrastructure

Azure Container Apps: diagnose Managed Identity before widening a role

A production runbook for qualifying Azure Container Apps 401/403 failures with Managed Identity, token evidence, RBAC scope, Key Vault, ACR, logs, validation and rollback before widening permissions.

25 Jul 2026 azurecontainer-appsmanaged-identityidentityentra-idrbackey-vaultacrobservabilitykqlsecurityrunbookrollbackproduction

When an Azure Container Apps revision starts returning 401, 403, Key Vault failures, ACR denials or rejected calls to an internal API, the fastest recovery idea is often to add a broader role to the managed identity. That should not be the first move. A broader role can make the service pass while hiding revision drift, a wrong client ID, a token requested for the wrong audience, an over-scoped RBAC assignment or configuration that no longer matches IaC.

The use case is a Container App named orders-worker-prod that reads a Key Vault secret, pulls an image from ACR and calls an internal API protected by Entra ID. After a deployment, a new revision starts but fails on dependencies. The runbook goal is to decide whether to fix the attached identity, the target scope, the credential code, the revision configuration, or roll back the revision before widening permissions.

Freeze the identity contract

Start by writing which identity should act, against which target, with which permission. Container Apps can use a system-assigned or user-assigned identity, and the active revision is not always the one the team is looking at in the portal.

yaml container-apps-identity-contract.yml

container_app:
name: orders-worker-prod
environment: cae-platform-prod
resource_group: rg-platform-prod
active_revision: orders-worker-prod--20260725-1

expected_identity:
type: user_assigned
client_id: 00000000-0000-0000-0000-000000000000
principal_id: 11111111-1111-1111-1111-111111111111
token_audiences:
  - https://vault.azure.net
  - https://management.azure.com
  - api://internal-operations

expected_targets:
key_vault: read selected secrets only
acr: AcrPull on the registry
internal_api: app role operations.read

decision_needed:
fix_identity_or_scope
rollback_revision
temporary_exception_with_expiry

Without that contract, the team may treat every 403 as missing access. A 403 can also prove that the wrong principal is acting correctly, but against the wrong target.

Separate token, authorization and application path

Classify the failure before changing RBAC.

text identity-failure-classes.txt

Token not obtained
Identity is not attached to the Container App
AZURE_CLIENT_ID points to a missing or staging identity
Code does not use ManagedIdentityCredential
Runtime does not receive expected identity variables

Token obtained but target rejects it
Role is missing at the exact scope
App role or API permission is not assigned
Token audience does not match the resource
RBAC propagation or service cache has not completed

Application path or configuration
Revision calls a staging URL
Secret or registry name changed
ACR, Key Vault or the API sees another caller
Application logs hide the difference between token and target call

This separation avoids adding Contributor on a resource group when the application is simply asking for a token for the wrong audience.

Read the identity attached to the Container App

First verify the real Container App state, not only the expected configuration in the ticket.

bash 01-container-app-identity-state.sh

SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
RG="rg-platform-prod"
APP="orders-worker-prod"

az account set --subscription "$SUBSCRIPTION"

az containerapp identity show --resource-group "$RG" --name "$APP" --query "{type:type,principalId:principalId,tenantId:tenantId,userAssignedIdentities:userAssignedIdentities}" --output json

az containerapp revision list --resource-group "$RG" --name "$APP" --query "[].{name:name,active:properties.active,trafficWeight:properties.trafficWeight,createdTime:properties.createdTime}" --output table

az containerapp show --resource-group "$RG" --name "$APP" --query "properties.template.containers[].env[]" --output table

Focus on AZURE_CLIENT_ID, target variables, referenced secrets and active revisions. A revision with 10 percent of traffic can be enough to create an intermittent incident.

Test the token from the revision

Testing from an administrator workstation does not prove the runtime. The useful test starts from the revision, or from an equivalent diagnostic container, with the same variables.

bash 02-runtime-token-probe.sh

# Run from a controlled diagnostic shell in the same app
# or from a temporary revision carrying the same identity.

RESOURCE="https://vault.azure.net"

curl -sS -H "Metadata: true" "$IDENTITY_ENDPOINT?api-version=2019-08-01&resource=$RESOURCE&client_id=$AZURE_CLIENT_ID" | jq '{client_id, resource, expires_on, token_present: (.access_token != null)}'

The goal is not to expose the token. The goal is to prove client ID, audience and identity endpoint availability. If this step fails, widening a target role will not fix the incident.

Correlate denials on the target

If the token exists, move to target logs. Key Vault, ACR, API Management, the internal API or Storage should show the principal, operation and result.

kusto 03-target-denials-container-apps.kql

let StartTime = datetime(2026-07-25T08:00:00Z);
let EndTime = datetime(2026-07-25T10:00:00Z);
let PrincipalId = "11111111-1111-1111-1111-111111111111";
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where ResultType has_any ("401", "403", "Forbidden", "Unauthorized")
 or OperationName has_any ("SecretGet", "RepositoryPull", "GatewayLogs")
| extend Raw = tostring(Properties)
| where Raw has PrincipalId or Identity has PrincipalId or CallerIdentity has PrincipalId
| project TimeGenerated, ResourceProvider, Resource, OperationName, ResultType, Identity, CallerIPAddress, Raw
| order by TimeGenerated desc

Adapt the query to the telemetry that is actually enabled. The key evidence does not change: did the target reject the expected identity, another identity, or does no call reach the target at all?

Verify the minimal scope

When the target-side denial is confirmed, verify the exact scope. An identity may have a role on the wrong ACR registry, the staging vault, or a resource group while the target actually uses an Entra app role.

bash 04-rbac-scope-check.sh

PRINCIPAL_ID="11111111-1111-1111-1111-111111111111"
TARGET_SCOPE="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sec-prod/providers/Microsoft.KeyVault/vaults/kv-orders-prod"

az role assignment list --assignee "$PRINCIPAL_ID" --all --query "[].{role:roleDefinitionName,scope:scope,condition:condition}" --output table

az role assignment list --assignee "$PRINCIPAL_ID" --scope "$TARGET_SCOPE" --include-inherited --query "[].{role:roleDefinitionName,scope:scope}" --output table

The minimal fix is often a precise role at the right scope, a missing app role, or a return to the previous identity. The broadest fix is rarely needed.

Decide fix, rollback or exception

The decision must stay usable for on-call and readable after the incident.

yaml container-apps-identity-decision.yml

fix_container_app_identity:
when:
  - identity_not_attached_to_app
  - wrong_user_assigned_client_id
  - active_revision_uses_staging_configuration
validation:
  - runtime_token_probe_returns_expected_client_id
  - active_revision_reaches_dependency

fix_target_authorization:
when:
  - token_is_issued_for_expected_audience
  - target_logs_show_denial_for_expected_principal
  - minimal_role_or_app_role_is_missing
validation:
  - target_log_shows_success_for_same_principal
  - permission_scope_matches_iac

rollback_revision:
when:
  - failure_started_with_new_revision
  - previous_revision_has_expected_identity_contract
  - identity_fix_cannot_be_validated_quickly
validation:
  - traffic_returns_to_previous_revision
  - dependency_calls_succeed

temporary_exception:
allowed_only_if:
  - user_impact_is_active
  - scope_is_same_or_narrower_than_expected
  - expiry_and_owner_are_recorded
blocked_if:
  - root_cause_is_unknown
  - broader_role_would_hide_wrong_identity

An RBAC exception without an expiry becomes security debt. Treat it as a temporary measure, not an architecture fix.

Validate before closure

Closure must prove runtime, target and rollback.

text identity-closure-checklist.txt

Final validation
Active revision carries the expected identity
Token is obtained from the runtime for the right audience
Target accepts the expected principal at the minimal scope
Logs show success with the same principal
Staging variables or temporary secrets have been removed
IaC reflects the corrected state
Rollback to the previous revision remains possible until stabilization

If the fix was applied in the portal, it remains incomplete until IaC, alerts and the runbook describe the new state.

Conclusion

An Azure Container Apps 401 or 403 should not automatically trigger a broader role assignment. The right sequence is to prove the active identity, test the token from the revision, read target-side denials, verify the minimal scope, then decide between a fix, rollback or bounded exception.

The final decision should leave the platform more explainable than it was before the incident: a known identity, a justified scope, correlatable logs and a rollback path ready if the new revision remains unstable.