Azure DevOps WIF: diagnose issuer or subject drift before bringing back a secret

An Azure DevOps pipeline that used to deploy without a secret suddenly fails with AADSTS70021, AADSTS700211 or AADSTS700213. The service connection still exists, the identity still has its Azure roles, and the YAML did not change. Under release pressure, the obvious workaround is to create another client secret and rerun production.

That workaround conflates three separate contracts. Azure DevOps issues a token with an issuer, subject and audience. Microsoft Entra looks for a federated credential that matches those claims. Azure RBAC then authorizes the resulting identity against a target. This runbook follows an Azure Resource Manager service connection that uses workload identity federation and drifted after a conversion, recreation or configuration change. It must end with an explicit decision: repair the mapping, complete the issuer migration, roll back the change, or stop the release without silently returning to a long-lived secret.

Freeze the failure before changing the identity

Preserve one pipeline run, its UTC timestamp, the failing task and the complete Entra error. Record the organization, project, service connection ID, identity type, tenant, client ID and expected Azure scope. A display name is not enough: the same name can exist in multiple projects or be reused after recreation.

incident: INC-CI-492
pipeline_run: 20260929.3
failed_at_utc: 2026-09-29T06:42:18Z
azure_devops:
organization: platform-ops
project: delivery
service_connection_id: <endpoint-guid>
service_connection_name: azure-prod
identity:
type: app-registration-or-user-assigned-managed-identity
tenant_id: <tenant-guid>
client_id: <application-client-guid>
azure_scope: /subscriptions/<subscription-guid>/resourceGroups/rg-app-prod
error:
code: AADSTS700213
correlation_id: <correlation-guid>
trace_id: <trace-guid>
assertion_issuer: <issuer-from-error>
assertion_subject: <subject-from-error>
stop_conditions:
- service connection id is unknown
- assertion claims were not preserved
- target identity is ambiguous
- proposed workaround creates a non-expiring secret

Do not select Verify and save yet. Saving can cause Azure DevOps to regenerate values and remove the comparison with the failing state. First capture the configuration screen, the endpoint through the Azure DevOps API or CLI, and the federated credential in Entra.

Use the error to locate the broken stage

A missing federated identity record means Microsoft Entra could not find a configured tuple matching the presented assertion. Compare the exact values, including case and punctuation.

• AADSTS700211 points toward the issuer;
• AADSTS700213 points toward the subject;
• AADSTS70021 is broader and requires checking issuer, subject and audience;
• AuthorizationFailed or an ARM 403 happens later: token exchange succeeded, but the principal cannot perform the requested action at the target scope.

This distinction prevents teams from granting a role to an identity that never authenticated or recreating a federated credential for a pure authorization failure.

Compare both sides of the contract

Export the service connection without writing any secret to pipeline logs. Then inventory federated credentials on the app registration or managed identity that the endpoint actually references.

ORG_URL="https://dev.azure.com/platform-ops"
PROJECT="delivery"
ENDPOINT_ID="<endpoint-guid>"
APP_ID="<application-client-guid>"

az devops configure --defaults organization="$ORG_URL" project="$PROJECT"
az devops service-endpoint show --id "$ENDPOINT_ID" --query '{id:id,name:name,type:type,authorization:authorization.scheme,servicePrincipalId:authorization.parameters.serviceprincipalid,tenantId:authorization.parameters.tenantid}' --output yaml

az ad app federated-credential list --id "$APP_ID" --query '[].{name:name,issuer:issuer,subject:subject,audiences:audiences}' --output yaml

For a user-assigned managed identity, query that identity’s federated credentials instead of an app registration. Also confirm that the client ID still resolves to the intended object. A deleted and recreated identity can retain a familiar name while representing a different principal.

The contract must match on issuer, subject and audience. The usual audience is api://AzureADTokenExchange. Do not normalize a subject manually or replace an identifier with a friendlier name. Configure Entra with the value generated by the service connection.

Identify the issuer generation without mixing formats

Azure DevOps WIF service connections can still expose two contract families. Older connections use the Azure DevOps issuer at https://vstoken.dev.azure.com/... and a readable subject such as sc://organization/project/service-connection. New connections use a Microsoft Entra issuer beginning with https://login.microsoftonline.com/... and a subject based on Azure DevOps identifiers.

Do not assemble a hybrid of the two. Copying the new issuer while retaining the old subject, or the reverse, creates a credential that looks plausible but cannot match. Azure DevOps has announced that the Azure DevOps issuer retires on July 1, 2027 for applicable service connections in Azure public cloud. A matching failure can therefore expose an incomplete migration, but it does not justify an improvised conversion during a critical deployment.

Assertion presented by the failed run
issuer: exact value from the Entra error or Azure DevOps configuration
subject: exact value from the Entra error or Azure DevOps configuration
audience: api://AzureADTokenExchange

Credential configured on the identity
issuer: value exported from Entra
subject: value exported from Entra
audiences: list exported from Entra

Verdict
exact_match: continue diagnosis at RBAC
issuer_mismatch: repair or complete the issuer migration
subject_mismatch: recreate the mapping from Azure DevOps values
audience_mismatch: fix the audience without broadening issuer or subject
identity_mismatch: stop and locate the actual object

If the organization, project or service connection was renamed, compare the emitted claim with the stored credential. Do not infer rename impact from the visible label alone: the historical format contains names, while the newer contract relies more heavily on identifiers.

Prepare a reversible correction

Separate service connection control, identity ownership and target RBAC. An Azure DevOps administrator can edit the endpoint without permission to add a federated credential to the app registration or managed identity. That separation is expected and should be explicit in the change record.

For an automatically managed connection, use the Azure DevOps Update flow when issuer conversion is offered. For a manually configured connection, copy issuer, subject and audience from Azure DevOps, create a new federated credential on the same identity, and return to finish the setup. Keep the old credential through the canary window when policy allows it; remove the previous path only after the new one is proven.

change:
service_connection_id: <endpoint-guid>
identity_client_id: <application-client-guid>
from_credential: ado-wif-legacy
to_credential: ado-wif-entra-issuer
target_scope_unchanged: true

preconditions:
- current issuer subject and audience exported
- current Azure role assignments exported
- identity owner and service connection admin available
- canary pipeline authorized explicitly
- no client secret added

rollback:
- stop production pipeline authorization
- restore previous service connection configuration
- keep or restore previous federated credential while supported
- remove only the failed candidate credential
- verify no RBAC scope changed

Do not change the issuer, target identity and Azure role in the same operation. A passing canary would not prove which change was required, while a failure would leave no legible rollback.

Canary with a read before deploying

Create or reuse a qualification pipeline that is explicitly authorized for the service connection. The canary should obtain a token, confirm tenant and principal, and read one known resource at the minimum scope. Add a negative control against an unauthorized scope. Do not run the full deployment template merely to test authentication.

trigger: none

steps:
- task: AzureCLI@2
displayName: Validate federated service connection
inputs:
  azureSubscription: azure-prod
  scriptType: bash
  scriptLocation: inlineScript
  inlineScript: |
    set -euo pipefail
    az account show       --query '{tenantId:tenantId,subscriptionId:id,user:user.name,userType:user.type}'       --output yaml
    az group show       --name rg-app-prod       --query '{id:id,name:name,location:location}'       --output yaml

    if az group show --name rg-forbidden-control >/dev/null 2>&1; then
      echo "Negative RBAC control unexpectedly succeeded" >&2
      exit 1
    fi

Minimum success proves five facts: Azure DevOps issued an assertion, Entra mapped it to the right principal, the token targets the expected tenant, the intended scope is readable, and the negative control remains denied. Run the canary with the same agent type and Azure DevOps task used in production. An outdated extension might not support WIF even when the connection is correct.

Correlate evidence in Entra and Azure DevOps

Preserve the run ID, service connection ID, client ID, correlation ID and UTC window. Service principal sign-in logs can confirm the principal, resource and result. Adapt the table to the diagnostics actually routed to the workspace.

let Start = datetime(2026-09-29T06:30:00Z);
let End = Start + 2h;
let ClientId = "<application-client-guid>";
AADServicePrincipalSignInLogs
| where TimeGenerated between (Start .. End)
| where AppId == ClientId or ServicePrincipalId == ClientId
| project TimeGenerated,
        AppDisplayName,
        ServicePrincipalId,
        ResourceDisplayName,
        ResultType,
        ResultDescription,
        CorrelationId,
        IPAddress
| order by TimeGenerated desc

No sign-in matching the run points back to the task, assertion or service connection. A successful sign-in followed by an ARM denial points to the role and its scope. Keep these verdicts separate in the incident record.

Decide, validate or roll back

Repair the federated credential when the client ID is correct and one contract field no longer matches the value generated by Azure DevOps. Complete the issuer migration when the connection is eligible, both sides have been inventoried, and the canary passes without an RBAC change. Correct RBAC only after proving successful authentication and one denied action at the exact scope.

Roll back the conversion if the identity becomes ambiguous, the production task does not support WIF, the negative control succeeds, or traces can no longer attribute the principal. Restore the previous valid WIF configuration, remove the candidate credential, and reauthorize production only after another canary.

A temporary secret is acceptable only under an existing emergency process, with an owner, minimum scope, short expiry and verified deletion. It should not be the default rollback for issuer or subject drift. The normal rollback is a known, observable and bounded federation contract.

Conclusion

A WIF service connection works when two independently managed configurations describe the exact same contract. The connection name, existing identity and role assignments are insufficient: issuer, subject and audience must match before RBAC is even evaluated.

Freeze the assertion, compare both sides, migrate one layer at a time, and canary a read with a negative control. The production decision then becomes defensible: validate the new federation contract, repair one precise mapping, or roll back without quietly reintroducing a durable secret.