Azure DevOps: prove artifact provenance before production promotion

A staging deployment passes, but the production job cannot prove which package it is about to release. The pipeline points to a successful build, the filename looks right, and the commit was approved. Yet the package may have been rebuilt from the same commit, downloaded from another run, repacked by a deployment stage, or mixed with files left on a self-hosted agent.

The running case is an Azure DevOps pipeline that builds a web API once, publishes a named Pipeline Artifact, deploys it to staging, then promotes the same bytes to production after approval. This runbook establishes a verifiable chain from source to runtime. Its outcome is explicit: promote the artifact, quarantine it and rebuild, or roll back to the last artifact whose provenance is complete.

Freeze promotion and name the candidate

Stop the production stage before another run changes the evidence. Record the exact pipeline, run ID, source commit, artifact name, target environment and pending deployment job. A build number or filename alone is not an identity: both can be copied into another context.

candidate:
pipeline_id: <pipeline-id>
build_run_id: <run-id>
source_repository: <repository>
source_commit: <full-sha>
artifact_name: api-drop
artifact_sha256: <sha256>
build_definition_revision: <revision-or-commit>

promotion:
target: production
deployment_job: <job-name>
approval_reference: <approval-or-change-id>
staging_validation: <evidence-link>

decision_owner: <role>
rollback_artifact_sha256: <last-known-good-sha256>

Preserve the original run and its logs while the decision is open. Deleting an Azure DevOps run also removes its associated Pipeline Artifacts, so cleanup is not containment when that run is part of the evidence.

Prove the source and the build contract separately

The source commit answers what code was checked out. It does not prove what the build produced. Dependencies, generated files, container base images, tool versions and pipeline templates can change the output without changing the application commit.

Compare the candidate run with the approved contract:

• full commit SHA and repository, not only branch name;
• pipeline YAML or definition revision used by the run;
• parameter values and variable groups that affect compilation or packaging;
• agent image or self-hosted pool and the build toolchain versions;
• resolved dependencies, lockfiles and base image digest when applicable;
• test and security gates attached to that same run.

If one of these inputs is unknown, mark it unknown. Do not replace missing provenance with “the run succeeded.” Success proves task completion, not artifact identity.

Download from the exact run and compute the digest

Address the artifact by run ID and artifact name. Download it into a clean investigation directory, then compute a deterministic digest for the deployable file. If the artifact is a directory, package it deterministically during the build or publish a manifest containing hashes for every file.

set -euo pipefail

rm -rf ./candidate-artifact
mkdir -p ./candidate-artifact

az pipelines runs artifact download \
--run-id <run-id> \
--artifact-name api-drop \
--path ./candidate-artifact \
--organization https://dev.azure.com/<organization> \
--project <project>

find ./candidate-artifact -type f -print0 \
| sort -z \
| xargs -0 sha256sum \
> artifact-files.sha256

sha256sum artifact-files.sha256

Store the resulting manifest as promotion evidence, not as an editable pipeline variable. Recompute the same digest after staging download and immediately before production deployment. A mismatch is a stop condition, even if both packages carry the same semantic version.

Inspect content without executing it

Provenance can be consistent and still point to an unsafe package. Inspect the archive or directory without loading binaries or running installation hooks. Compare its shape with the previous approved artifact and the expected build output.

Look for:

• unexpected executables, scripts or deployment manifests;
• configuration files containing environment-specific endpoints or secrets;
• missing lockfile, SBOM, signature or dependency inventory when the pipeline normally produces one;
• debug symbols, test fixtures or source files that should not ship;
• unexplained size, file-count or dependency changes.

An SBOM improves the evidence but does not replace the digest or the link to the producing run. A signature is useful only when the verifier also checks the expected signer, trust policy and payload digest.

Keep build and promotion as different authorities

The deployment stage should consume a specific artifact. It should not check out source and rebuild “for convenience.” Rebuilding after approval creates new bytes outside the reviewed build, even when the source SHA is unchanged.

Separate permissions as well. The build identity needs to publish the artifact; the deployment identity needs to read the approved artifact and write to the target environment. Neither needs permission to edit approvals. Restrict manual artifact uploads or run selection so an operator cannot silently replace the candidate.

On self-hosted agents, use a clean destination and remove it after the job. A reused workspace must never become an implicit artifact source. The deployment log should record the run ID, artifact name and digest before any production write.

Validate the same bytes in staging and production canary

Deploy the verified candidate to staging without repackaging it. Record the runtime version endpoint, image digest, assembly version or another marker that can be tied back to the artifact manifest. Functional tests alone are insufficient if they cannot identify the bytes under test.

Promotion begins with one bounded production canary: one instance, slot, ring or low-risk tenant. During the observation window, validate both behavior and identity:

Identity checks
Production downloaded the recorded run ID and artifact name
Digest before deployment equals the approved digest
Runtime marker maps to the same artifact

Operational checks
Canary health, latency and error rate remain within the agreed envelope
Required migrations are backward compatible or separately controlled
No unexpected dependency or authorization failure appears

Decision
PROMOTE: provenance and canary checks pass
HOLD: evidence is incomplete but no production write occurred
QUARANTINE_AND_REBUILD: digest or content differs
ROLL_BACK: production health regresses after the canary write

Do not overwrite the candidate when a gate fails. Preserve it under restricted access so the difference can be explained, then start a new build with a new run ID.

Roll back to bytes that are already known

Rollback must reference the last known-good run ID, artifact name and digest. Rebuilding an old commit is not the same rollback: registries, dependencies and build images may have moved since the original release.

Before promotion, confirm that the previous artifact is still retained and deployable, that its runtime dependencies remain available, and that database or configuration changes preserve backward compatibility. If rollback requires a compensating migration, treat it as a separate approved action with its own validation.

After rollback, verify the runtime marker and service health. Keep the failed candidate quarantined until the team can distinguish a pipeline selection error, build-environment drift, workspace contamination or an unauthorized change.

Conclusion

An approved commit is necessary but not sufficient evidence for production promotion. The operational unit is the exact artifact produced by an identified run, with known build inputs, a stable digest and a deployment path that does not rebuild or repackage it.

The decision is mechanical: promote only when source, run, content digest, approval and canary all refer to the same bytes. Quarantine and rebuild when identity is ambiguous. If the canary regresses, roll back to the retained artifact already proven in production, not to a fresh reconstruction of old source.