Azure DevOps: diagnose a Key Vault secret after rotation before rerunning production

A secret rotation can succeed in Azure Key Vault and still break the next deployment. The pipeline sees an empty variable, receives a 401, or fails before the first script runs. The quick response is often to rerun the job, recreate the secret or widen the service connection permissions. Those actions collapse several possible failures into one: a disabled version, a name missing from the variable group, read authorization, network path, variable collision, or an application that still expects the previous value.

The use case is an Azure DevOps pipeline deploying payments-api-prod. It consumes payments-api-key from Key Vault through the vg-payments-prod variable group. Rotation completed at 08:30 UTC; the next deployment fails during its smoke test. This runbook ends with a concrete decision: repair the Azure DevOps binding, temporarily restore the previous version, fix the consumer, or stop the rerun until the evidence is complete.

Freeze The Rotation Contract

Before changing the vault or pipeline, collect the identifiers that connect rotation to failure. The secret value is never part of that evidence.

incident:
pipeline: payments-api-prod
run_id: 18452
stage: deploy_westeurope
variable_group: vg-payments-prod
key_vault: kv-payments-prod
secret_name: payments-api-key
rotation_time_utc: 2026-08-07T08:30:00Z
first_failed_run_utc: 2026-08-07T08:42:00Z

evidence_without_secret_value:
- pipeline commit and template version
- variable group identifier and authorization
- service connection principal identifier
- secret version, enabled state and dates
- agent pool and network path
- failing task and downstream HTTP status

decisions:
- repair mapping or authorization
- deploy with current version
- restore previous version temporarily
- stop and correct the application contract

This contract prevents a common false diagnosis: blaming the secret because the error appeared in the same window, even though the pipeline also changed its template, pool or variable group.

Separate Name, Value And Consumer

A variable group linked to Key Vault stores a selection of secret names. Values are fetched at runtime. A new version under an already selected name can therefore become available without editing the group. Adding a new name or deleting the old one in Key Vault does not automatically update that selection.

Follow three distinct objects through the incident:

1. the name selected in vg-payments-prod;
2. the current version returned by Key Vault for that name;
3. the variable injected into the task running the smoke test.

An application 401 proves only that the supplied value was rejected. It does not prove that Azure DevOps read the wrong version. A higher-precedence variable may have replaced it, or the target service may not yet accept the newly rotated credential.

Inspect Versions Without Reading The Value

Start with secret metadata. The latest version should be enabled, inside its validity window, created at the expected time and traceable to the rotation record.

SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
VAULT="kv-payments-prod"
SECRET="payments-api-key"

az account set --subscription "$SUBSCRIPTION"

az keyvault secret list-versions --vault-name "$VAULT" --name "$SECRET" --query "[].{version:id,enabled:attributes.enabled,created:attributes.created,updated:attributes.updated,notBefore:attributes.notBefore,expires:attributes.expires}" --output table

Retain the version identifier, but never the output of az keyvault secret show --query value. The pipeline should prove presence and use without printing the secret, copying it into an artifact or passing it through a visible command-line argument.

If the latest version is disabled or expired, another rerun cannot fix the incident. If it is valid, continue into binding and identity before creating yet another version.

Prove Binding, Identity And Path

The variable group must be authorized for the pipeline and linked to the intended vault and secret name. The principal behind the service connection needs the required read permissions under the vault authorization model. Record its object ID: the display name of a service connection is not identity evidence.

Variable group
Group: vg-payments-prod
Type: AzureKeyVault
Vault: kv-payments-prod
Selected name: payments-api-key
Pipeline permission: payments-api-prod authorized

Service connection
Connection used by the variable group recorded
Tenant and subscription match production
Principal object ID recorded
Get/List or equivalent RBAC access proven

Execution path
Microsoft-hosted or self-hosted agent identified
Vault firewall and public/private access model recorded
DNS and TCP/TLS path tested from the actual self-hosted pool when applicable
No temporary broad firewall opening kept after diagnosis

A private vault changes the operating path. An AzureKeyVault@2 task on a self-hosted agent inside an allowed network may fit when the variable-group path cannot reach the vault under its access model. Treat that as a tested architecture decision, not an incident workaround quietly left in the pipeline.

Eliminate Variable Collisions

A correct value can be overwritten after retrieval. Azure Pipelines applies precedence across queue-time variables, YAML variables and variable groups. Two groups defining the same name in the same scope make the result difficult to reason about.

Inspect the final YAML after template expansion and find every definition of payments-api-key or its alias. Prefer a specific name and map the secret into an environment variable only for the task that requires it.

variables:
- group: vg-payments-prod

steps:
- bash: |
    set -euo pipefail
    if [ -z "$PAYMENTS_API_KEY" ]; then
      echo "Secret injection failed: PAYMENTS_API_KEY is empty"
      exit 20
    fi
    echo "Secret injection present; value intentionally not logged"
  displayName: Validate secret injection
  env:
    PAYMENTS_API_KEY: $(payments-api-key)

This check separates a missing variable from a secret rejected by the target service. It does not yet prove the value is correct; the bounded functional test does that.

Build A Controlled Rerun

Do not rerun the full deployment first. Add a preflight stage using the same template, variable group, service connection and pool as production, but making no resource changes. It should verify injection, then call a non-destructive validation endpoint or authentication check provided by the target service.

stages:
- stage: secret_preflight
  variables:
  - group: vg-payments-prod
  jobs:
  - job: validate_rotated_secret
    pool: prod-private-agents
    steps:
    - bash: |
        set -euo pipefail
        test -n "$PAYMENTS_API_KEY"
        status=$(curl --silent --show-error --output /dev/null           --write-out "%{http_code}"           --connect-timeout 5           --max-time 15           --header "Authorization: Bearer $PAYMENTS_API_KEY"           "https://payments.internal.example.com/auth/check")
        test "$status" = "204"
      displayName: Validate current secret against bounded endpoint
      env:
        PAYMENTS_API_KEY: $(payments-api-key)

Adapt the endpoint and authentication scheme to the real service. The stable principle is to exercise the same retrieval path as production, avoid business writes and enforce a short timeout. A green preflight permits the next decision; it does not automatically start the deployment.

Correlate Rotation And Runs

The timeline should show which run consumed which logical generation of the secret without storing its value. Record a rotation ID from the process, the Key Vault version ID from the control plane, the Azure DevOps run and preflight outcome in the change ticket or operational telemetry.

{
"rotationId": "rot-payments-20260807-0830",
"vault": "kv-payments-prod",
"secretName": "payments-api-key",
"secretVersionId": "version-id-without-value",
"variableGroup": "vg-payments-prod",
"pipelineRunId": 18452,
"preflightRunId": 18457,
"injectionCheck": "present",
"boundedAuthCheck": "passed",
"productionDeploymentApproved": false,
"rollbackVersionId": "previous-version-id"
}

Do not log a hash of the secret to identify it. That creates derived sensitive data and still does not establish that the consumer accepts the value. A version ID and a functional check are sufficient.

Decide Validation Or Rollback

Close the runbook with an explicit decision.

Validate rotation
Name is selected in the variable group
Latest version is enabled and inside its validity window
Identity and access path are proven
No higher-precedence variable replaces the secret
Non-destructive preflight passes from the production pool
Deployment remains subject to its normal approval

Repair and retest
New name is missing from the variable group
Pipeline or group is not authorized
Service connection resolves to the wrong principal or subscription
Network blocks retrieval from the real execution path
YAML collision replaces the variable

Roll back rotation
Consumer does not accept the new value
Application fix cannot fit inside the incident window
Previous version remains allowed by security policy
Reactivation is bounded, approved and followed by another preflight

Stop the rerun
Consumed version cannot be identified
Value was exposed in logs or an artifact
Rollback is undefined
Concurrent changes prevent attribution

A sound rollback does not copy the old value into YAML. It temporarily restores a controlled Key Vault version, preserves the same consumption path and sets a deadline for the next rotation. Once preflight is green, production can be rerun with a clear chain of evidence: selected name, active version, correct identity, reachable path and compatible consumer.