Azure ACR and AKS: verify an image signature before production promotion

An image can pass its vulnerability scan and still be unfit for production. The expected tag may have moved, the build may come from an unapproved pipeline, or the signature may be cryptographically valid but issued to an identity that has no authority to publish into that repository. Redeploying latest or bypassing admission might restore service, but it removes the provenance evidence the team needs most.

The use case is a pipeline that builds an API, pushes its image to Azure Container Registry, and deploys it to AKS. Promotion should be allowed only when the digest is immutable, a Notation signature exists, that signature passes a scoped trust policy, and the exact digest reaches the canary. The runbook ends with one of three decisions: promote, hold, or restore the last known trust policy without accepting an unverified image.

Freeze the promotion contract

Do not start with the signature. Start with the exact object that may enter production. A tag is mutable; a digest identifies the OCI manifest that verification covers.

artifact:
registry: acrprodweu.azurecr.io
repository: payments/api
candidate_digest: sha256:<candidate-digest>
source_pipeline_run: build-20260825.4
trust:
policy_version: prod-images-v7
registry_scope: acrprodweu.azurecr.io/payments/api
expected_publisher: x509.subject:<approved-subject>
controls:
verify_in_pipeline: true
deploy_by_digest: true
canary_namespace: payments-canary
admission_rollout: audit-before-enforce
decision:
promote_only_if:
  - digest resolved from the candidate tag
  - signature passes strict verification
  - publisher matches the approved identity
  - canary runs the verified digest
rollback:
last_known_good_digest: sha256:<previous-approved-digest>
last_known_good_policy: prod-images-v6

Attach the digest, pipeline run, policy version, expected identity and verification output to the promotion request. Without them, a successful notation verify is difficult to connect to the deployment that actually occurred.

Resolve the digest before verification

Find the manifest referenced by the candidate tag, then stop using that tag for the rest of the runbook. Azure CLI inventory commands may differ across versions; the operational requirement is to resolve one digest from ACR and build a fully qualified reference.

ACR="acrprodweu"
LOGIN_SERVER="acrprodweu.azurecr.io"
REPOSITORY="payments/api"
TAG="build-20260825.4"

az acr manifest list-metadata --registry "$ACR" --name "$REPOSITORY" --query "[?contains(tags, '$TAG')].{digest:digest,tags:tags,created:createdTime,lastUpdate:lastUpdateTime}" --output table

CANDIDATE_DIGEST="sha256:<digest-returned-by-acr>"
IMAGE="$LOGIN_SERVER/$REPOSITORY@$CANDIDATE_DIGEST"
printf '%s
' "$IMAGE"

Stop promotion if the tag resolves unexpectedly, if the digest differs from the build output, or if the manifest changed after signing. Do not repair that divergence by signing an existing digest after the fact when its provenance has not been established.

Separate signature presence from trust

notation ls inventories signatures attached to a manifest. Presence does not prove validity or publisher identity. It is only the first inspection step.

az acr login --name "$ACR"

notation ls "$IMAGE"

# Verification must use an imported and versioned policy.
notation policy show

A missing signature is useful evidence: hold the candidate and repair the publishing pipeline. It is not a reason to add a wildcard signer or move the policy to a permissive verification level.

Scope the trust policy

A useful policy restricts repository scope, verification level, trust store and allowed publisher identity. Avoid * for both scope and identity in production. That would reduce provenance control to a certificate-presence check.

{
"version": "1.0",
"trustPolicies": [
  {
    "name": "payments-production",
    "registryScopes": [
      "acrprodweu.azurecr.io/payments/api"
    ],
    "signatureVerification": {
      "level": "strict"
    },
    "trustStores": [
      "ca:payments-builders"
    ],
    "trustedIdentities": [
      "x509.subject: <approved-subject>"
    ]
  }
]
}

Install the root or intermediate certificate through a controlled trust-store process, then import the policy from a reviewed version. Verification must target the digest reference.

POLICY_FILE="trustpolicy.json"
EVIDENCE_DIR="promotion-evidence"

mkdir -p "$EVIDENCE_DIR"
notation policy import "$POLICY_FILE"
notation policy show > "$EVIDENCE_DIR/trust-policy.txt"

notation verify "$IMAGE" > "$EVIDENCE_DIR/notation-verify.txt" 2>&1

printf '%s
' "$IMAGE" > "$EVIDENCE_DIR/verified-image.txt"

Retain this evidence with the GitOps commit or change record. An exit code 0 alone is not enough. The digest, policy and verified identity are what make the decision explainable.

Diagnose failure without widening trust

Verification failures usually belong to one of four families. Keeping them separate prevents a local fault from becoming a global exception.

Symptom                         Evidence to inspect                 Bounded action
No signature found               notation ls, build publication       fix signing stage; hold image
Signature invalid                digest, signature, certificate        reject candidate; rebuild
Identity not trusted             policy scope, X.509 subject           correct reviewed policy or signer
Registry access failed           ACR login, RBAC, network path         restore read access; retry verify
Certificate expired or rotated   chain, validity, trust-store version  deploy reviewed trust bundle

Never solve these failures by trusting every identity or deploying by tag.

Certificate rotation needs a bounded overlap. Old and new trust bundles may coexist for a controlled window, but the new publisher identity must be approved before it signs release candidates. Rollback then restores the previous trust-store version rather than disabling verification.

Put the gate in the pipeline

Run verification after push and signing but before changing deployment state. The pipeline should hand a verified digest to the GitOps repository, not a tag that the cluster resolves later.

stages:
- build_and_push
- sign_digest
- verify_digest:
    inputs:
      - fully_qualified_digest
      - trust_policy_version
    outputs:
      - verification_evidence
      - verified_digest
    stop_conditions:
      - signature_missing
      - publisher_not_trusted
      - digest_changed
- prepare_gitops_change:
    image_reference: acrprodweu.azurecr.io/payments/api@sha256:<verified-digest>
- deploy_canary
- validate_or_rollback

Keep signing authority separate from promotion authority. The signer attests origin; the promoter decides whether that origin and digest satisfy the environment policy. This makes it possible to revoke one capability without making the registry unusable.

Roll out admission without freezing the cluster

Pipeline verification reduces risk but does not cover a manual deployment or another controller. Admission policy can verify signatures when a workload is created. Start in a preproduction cluster or namespace, use audit mode, and exercise both signed and deliberately unsigned images.

The managed AKS Image Integrity experience uses Ratify, Azure Policy and Gatekeeper. It is currently a preview with audit-only policy behavior and limitations that make it unsuitable as a production dependency. Use it to evaluate the signal. For blocking control in production, choose a mechanism supported by your organization, test its availability, and retain the CI/CD gate as the first barrier.

Moving from audit to enforcement requires four proofs: the verifier can reach ACR, the trust store is available, system workloads have an explicit policy, and verifier failure has a decided behavior. A webhook outage without a tested failure strategy can stop every deployment, including rollback.

Validate the canary by runtime digest

After promotion, compare desired state with the imageID reported by the runtime. A tag printed in a manifest is not enough.

NAMESPACE="payments-canary"
DEPLOYMENT="payments-api"

kubectl rollout status --namespace "$NAMESPACE" deployment/"$DEPLOYMENT" --timeout=10m

kubectl get pods --namespace "$NAMESPACE" -l app=payments-api -o custom-columns='POD:.metadata.name,DESIRED:.spec.containers[*].image,RUNTIME:.status.containerStatuses[*].imageID,READY:.status.containerStatuses[*].ready'

kubectl get events --namespace "$NAMESPACE" --sort-by=.lastTimestamp

Functional validation still matters: availability, errors, dependency health and the canary’s business signals. A signature proves integrity and identity under the selected policy. It does not prove that the application behaves correctly.

Decide promote, hold or rollback

Promote only when strict verification succeeds, the runtime digest matches the verified digest, and the canary is healthy. Hold when origin cannot be proven, even if the scanner reports no known vulnerability.

If a new policy rejects previously approved images, restore the last known trust store or policy version and verify the same digest again. Do not disable admission globally. If the image itself regresses, deploy the last healthy digest that still satisfies the current trust policy. Keep these two rollback paths independent.

Conclusion

ACR image promotion should not depend on trust in a tag. It should connect an immutable digest, a valid signature, an approved publisher identity, a versioned trust policy and the digest that AKS actually runs.

The runbook has a clear outcome: promote with evidence, hold the candidate, or restore a known policy and verify again. A useful rollback preserves provenance control; urgency never becomes permission to deploy an unverified image.