Automation
Azure DevOps: diagnose a Variable Group before rerunning deployment
A production runbook for qualifying Azure DevOps Variable Group drift with variables, secrets, Key Vault, scopes, logs, validation and rollback before rerunning a pipeline.
An Azure DevOps pipeline can fail after a change that does not look like a deployment: a modified variable, an expired secret, a broken Key Vault link, a moved scope, a removed library permission or an override passed at run time. The tempting answer is to rerun the pipeline after “fixing the value”. In production, that shortcut can spread a bad configuration across several environments.
The use case is a release-prod pipeline deploying an Azure API and reading a shared Variable Group: internal API URL, Key Vault name, feature flags, connection identifiers, telemetry endpoints and rollback parameters. Deployment fails on an authentication step or health check. The runbook goal is to decide whether the pipeline can be rerun, whether the Variable Group should be rolled back, whether a secret should be restored, or whether deployment must stay blocked until the configuration contract is clear again.
Freeze the configuration contract
Start by treating the Variable Group as a production dependency. Knowing that a variable exists is not enough. The team needs to know where it is used, which environment it targets, who owns it, how it changes and what happens when it changes.
pipeline: release-prod
environment: production
variable_group: vg-api-prod
linked_key_vault: kv-prod-app
last_successful_run: 2026-07-21T21:14:00Z
failed_run: 2026-07-22T05:40:00Z
configuration_contract:
variables_used_by_deploy:
- API_BASE_URL
- KEY_VAULT_NAME
- FEATURE_MODE
- APPINSIGHTS_CONNECTION_STRING
- ROLLBACK_SLOT
secrets_used_by_runtime:
- DB_CONNECTION
- PARTNER_API_TOKEN
expected_scope:
project: platform-prod
pipelines:
- release-prod
- hotfix-prod
owners:
technical: platform-engineering
operational: sre-oncall
rollback_reference:
source: last successful run and approved change ticketThis step avoids two common mistakes: rerunning with a corrected but unvalidated value, or changing a shared Variable Group without checking the other pipelines that consume it.
Compare the failed run with the last healthy run
Useful diagnosis starts with a comparison. The pipeline, commit, YAML template, Variable Group, resolved secrets and run parameters may all have changed between two executions.
Compare before rerun
Deployed commit or tag
YAML template version
Variable Groups referenced by the pipeline
Variables overridden at run time
Target Azure DevOps environment
Service connection in use
Linked Key Vault and resolved secret names
Approvals and checks crossed
Agent or execution pool
Change window and related request
Block rerun when
The failed run does not read the same Variable Group
A runtime variable masks the library value
The secret resolves but points to the wrong version
The group is shared with another critical pipeline
The last healthy run cannot be identifiedA deployment failure after a configuration change is not fixed by rerunning blindly. The rerun should reproduce a state the team understands, not hope to land on a value that became correct again.
Separate plain variable, secret and Key Vault reference
A Variable Group often mixes several kinds of data. A plain variable can be read in logs. A secret is masked but may be empty, expired or badly encoded. A Key Vault reference depends on Azure DevOps authorization, identity, secret name and sometimes a specific version.
Plain variable
Symptoms: wrong URL, unexpected feature flag, incorrect resource name
Decision: fix or roll back the value with consumer impact documented
Azure DevOps secret
Symptoms: masked value but authentication denied, empty string, incomplete rotation
Decision: restore known version or rerun only after a bounded test
Key Vault linked secret
Symptoms: secret not found, access denied, disabled version, firewall or network issue
Decision: verify Key Vault, identity and version before touching the pipeline
Run override
Symptoms: healthy pipeline with wrong manual parameter
Decision: rerun with explicit parameter and restrict free-form override if recurring
Library scope or permission
Symptoms: pipeline cannot access variable group, unexpected approval, permission denied
Decision: restore minimal scope, do not open the library to the whole projectThe important point is not to widen permissions to hide a configuration problem. If the secret exists but the pipeline should not access it, the correct state may be the block.
Check access and audit before correction
A Variable Group change should leave usable evidence: who changed what, when, from which request and with which validation. Without audit, the fix becomes a second drift.
ORG="https://dev.azure.com/example"
PROJECT="platform-prod"
GROUP_ID="42"
az devops configure --defaults organization="$ORG" project="$PROJECT"
az pipelines variable-group show --group-id "$GROUP_ID" --output json
az pipelines runs list --pipeline-name "release-prod" --top 5 --output table
# Add Azure DevOps audit or organization logs when available.
# Keep change time, author, touched group and impacted run.If the group is linked to Key Vault, also verify on the Azure side that the secret is enabled, the expected version exists, and RBAC or network policy did not change in the same window.
VAULT="kv-prod-app"
SECRET="PARTNER_API_TOKEN"
az keyvault secret show --vault-name "$VAULT" --name "$SECRET" --query "{id:id,enabled:attributes.enabled,created:attributes.created,updated:attributes.updated,recoveryLevel:attributes.recoveryLevel}" --output json
az keyvault secret list-versions --vault-name "$VAULT" --name "$SECRET" --query "[].{id:id,enabled:attributes.enabled,created:attributes.created,updated:attributes.updated}" --output tableThe diagnosis should prove whether the pipeline reads a wrong value, can no longer read the value, or reads a correct value and fails later.
Correlate pipeline, deployment and runtime
A Variable Group is not valid just because the pipeline can read it. It is valid when deployment and runtime confirm the expected configuration. Correlate the Azure DevOps run with Azure Activity, application logs and health checks.
let DeploymentStart = datetime(2026-07-22T05:40:00Z);
let DeploymentEnd = datetime(2026-07-22T06:05:00Z);
AzureActivity
| where TimeGenerated between ((DeploymentStart - 10m) .. (DeploymentEnd + 20m))
| where Caller has_any ("AzureDevOps", "release-prod", "00000000-0000-0000-0000-000000000000")
| project TimeGenerated,
OperationNameValue,
ActivityStatusValue,
Caller,
ResourceGroup,
ResourceId,
CorrelationId,
Properties
| order by TimeGenerated ascThen add application or health-check logs over the same window. A runtime 401 after deployment can come from a secret, but also from managed identity, API scope, application cache or configuration not being reloaded.
Decide rerun, rollback or block
The runbook output should be a short and defensible decision. A rerun is acceptable only when the configuration state is proven, the scope is bounded and rollback is ready.
Rerun the pipeline
Expected Variable Group is identified
Corrected or restored value is approved
Key Vault secrets are enabled and readable
Run overrides are explicit
Last healthy run is the reference
Roll back the Variable Group
Value drift is the likely cause
Several pipelines consume the group
Previous version is known
Application fix would take longer than configuration rollback
Block deployment
Change author or reason is unknown
Expected value is not documented
Secret may have been compromised
Group is too shared for a fast correction
Pipeline would hide an approval or permission errorThe safest rollback may be to restore the Variable Group to the version used by the last healthy run, then replay only the validation stage. If the pipeline cannot work at that level of granularity, the deployment design is part of the problem.
Add a durable guardrail
After the incident, avoid leaving the Variable Group as a black box. Critical variables should have an owner, an environment, a change rule and a consumption test.
guardrails:
ownership:
every_variable_has_owner: true
production_group_has_change_ticket: true
scope:
no_cross_environment_group: true
pipeline_access_is_explicit: true
validation:
dry_run_stage_reads_configuration: true
health_check_uses_same_values: true
rollback_values_are_documented: true
secrets:
key_vault_link_preferred_for_rotated_secrets: true
secret_version_change_is_a_deployment_signal: true
audit:
variable_group_change_is_reviewed: true
failed_run_keeps_configuration_snapshot: trueThese guardrails do not make configuration immutable. They make configuration changes readable and reversible.
Conclusion
An Azure DevOps Variable Group is a production surface. It influences deployed code, resolved secrets, called dependencies, feature flags and sometimes rollback itself.
Before rerunning a pipeline, prove the group being read, the effective value, the secret version, the scope, the audit trail and the runtime signals. The right decision may be a bounded rerun, a configuration rollback, a secret restore, or an explicit block until the configuration contract is operable again.