Azure ACR and AKS: contain a vulnerable image digest before redeployment

A vulnerability assessment flags a digest in Azure Container Registry. The repository has several tags, AKS pods are already running, and the delivery pipeline can still promote the image. Deleting the manifest immediately may look like containment. It does not replace running containers, and it can make their next restart fail before a safe digest is ready.

Consider a service deployed to several AKS namespaces from an ACR image. The team must find every place where the suspect digest runs, stop further propagation, move to a known artifact and prove that the cluster cannot recreate the affected version. The runbook ends with an explicit decision: retain the manifest temporarily for investigation, delete it from the registry, or isolate the workload because no safe release is available.

Anchor the incident on the digest

A tag expresses release intent; a digest identifies the manifest that actually runs. One digest can have several tags, and a mutable tag may already point somewhere else. Start the incident record with the full digest, repository, registry and the evidence that raised the alert.

incident: inc-container-2026-08-08-01
registry: acrprodweu
repository: payments/api
suspect_digest: sha256:<suspect-digest>
finding:
source: approved-vulnerability-assessment
severity: <severity>
affected_component: <package-or-layer>
exploitable_context: under-review
scope:
clusters: [aks-prod-weu, aks-prod-neu]
environments: [production]
containment_owner: platform-oncall
application_owner: payments-team
known_good_digest: sha256:<known-good-digest>
deletion_authorized: false

Do not turn a high severity label into an automatic registry deletion. Confirm that the finding belongs to this digest, that the vulnerable component is present in the execution path and that the image is not merely a parent used during a build stage. That qualification changes urgency; it should not delay the promotion freeze.

Map the ACR manifest to AKS workloads

First, inspect the manifest metadata and every tag currently attached to it. This operation is read-only.

ACR="acrprodweu"
REPOSITORY="payments/api"
SUSPECT_DIGEST="sha256:<suspect-digest>"

az acr manifest list-metadata --registry "$ACR" --name "$REPOSITORY" --query "[?digest=='$SUSPECT_DIGEST']" --output json

Then inspect pods for both the requested image and the image identifier resolved by the runtime. The image field may still display a tag while imageID exposes the digest that was pulled.

kubectl get pods --all-namespaces -o json | jq -r '
.items[] as $pod
| ($pod.status.containerStatuses // [])[]
| select(.imageID | contains("sha256:<suspect-digest>"))
| [
    $pod.metadata.namespace,
    $pod.metadata.name,
    .name,
    .image,
    .imageID,
    (.ready | tostring),
    (.restartCount | tostring)
  ]
| @tsv'

Repeat the inventory for every cluster in scope. Map each pod to its controller and then to the GitOps repository or release that owns desired state. Editing a standalone pod will not hold: a Deployment, StatefulSet, DaemonSet or operator can recreate it from its own image reference.

Include zero-replica workloads and CronJobs. They produce no pod during the initial investigation but can reintroduce the digest on the next scale-out or scheduled execution.

Freeze propagation without destroying evidence

The first change should stop another promotion, not erase the artifact under investigation. Pause the job that moves release tags, deny the digest in the deployment policy and prevent nonessential automated restarts. Keep ACR reads and log collection available.

blocked:
- promote suspect digest to an environment tag
- create a new workload revision with suspect digest
- restart or scale out affected workloads without approval
- rebuild from an unpinned base image

still_allowed:
- read registry metadata and vulnerability evidence
- export manifests, SBOM and deployment history
- build and scan a remediation candidate
- deploy the approved known-good digest to a canary

exit_conditions:
- every desired-state reference is known
- a deployable safe digest is selected
- rollback and isolation paths are assigned
- validation commands are ready before rollout

Moving a tag does not clean existing pods. Deleting from ACR does not stop them either. Containment is complete only when the delivery path rejects the digest and controller desired state can no longer recreate it.

Select a safe return point, not merely an older image

Identify the return candidate by digest and reassess it using the evidence available during the incident. “Deploy the previous version” is not enough if that version shares the same vulnerable layer or cannot run after an irreversible schema migration.

At minimum, verify build provenance, current scan results, configuration compatibility, external dependencies, irreversible migrations and startup against the present state. When no previous digest is defensible, build a patched image from a pinned base or isolate the service. Do not restore the suspect digest merely because the remediation rollout fails.

Return to a known digest
digest rescanned and provenance verified
compatible with the current schema and configuration
startup probe available
another safe release is ready if it regresses

Build a patched digest
no previous release is compatible
dependency or base-image patch is required
reproducible pipeline and attestations are available

Isolate the workload
active exploitation or high impact is confirmed
no safe digest can be deployed immediately
network exposure or business action can be suspended
restoration requires a new validation decision

Redeploy by digest and start with a canary

Change the source of truth, not just the live cluster. A digest reference ensures that every new pod requests the same manifest even when someone later moves a tag.

spec:
template:
  metadata:
    annotations:
      naxaya.com/image-incident: inc-container-2026-08-08-01
  spec:
    containers:
    - name: api
      image: acrprodweu.azurecr.io/payments/api@sha256:<known-good-digest>
      imagePullPolicy: IfNotPresent

Apply the change to one replica or a bounded traffic segment first. Validate startup, readiness, application errors, dependencies and a reversible business transaction. Expand in explicit stages. A Deployment becoming Available is not enough; the chosen digest must process representative traffic correctly.

NAMESPACE="payments"
DEPLOYMENT="payments-api"

kubectl rollout status --namespace "$NAMESPACE" deployment/"$DEPLOYMENT" --timeout=10m

kubectl get pods --namespace "$NAMESPACE" -l app=payments-api -o custom-columns='POD:.metadata.name,IMAGE:.spec.containers[*].image,IMAGE_ID:.status.containerStatuses[*].imageID,READY:.status.containerStatuses[*].ready'

After rollout, run the global suspect-digest inventory again. It should return no current pod, newly scheduled pod or job started during the change window. Also verify that GitOps manifests, CronJobs and suspended workloads no longer reference it.

Decide the fate of the suspect manifest

Deleting by digest from ACR removes the manifest and every tag that references it. The operation is destructive. It prevents future pulls from that registry, but it neither stops existing containers nor protects a restart that still points at the deleted digest.

Require three proofs before deletion: no approved desired state references the digest, no recovery dependency needs it, and the evidence required for the investigation has been retained under the organization’s policy. Display the exact target first, then obtain authorization for the destructive step.

ACR="acrprodweu"
REPOSITORY="payments/api"
SUSPECT_DIGEST="sha256:<suspect-digest>"

az acr manifest list-metadata --registry "$ACR" --name "$REPOSITORY" --query "[?digest=='$SUSPECT_DIGEST'].[digest,tags,lastUpdateTime]" --output table

# Run only after formal validation of the target.
az acr repository delete --name "$ACR" --image "$REPOSITORY@$SUSPECT_DIGEST" --yes

When the investigation requires artifact retention, do not let retention become a promotion path. Keep the pipeline and admission block in place, restrict write permissions and record a review date.

Validate containment and prepare rollback

Rolling back the rollout must never mean “restore the suspect digest.” Prepare two paths: switch to a second safe digest if the selected release regresses, or isolate the workload when no clean image works with the current state.

Close the incident only when the suspect digest is no longer running, no source of truth references it, a promotion attempt is denied, the replacement digest is observed in every cluster, probes remain healthy, and the retention or deletion decision is recorded.

After deletion, test a controlled pull of the suspect digest from an environment with no local cache: it must fail. Then reschedule the healthy canary on a node that does not already hold its layers. This second test proves that the replacement digest remains retrievable from ACR.

Conclusion

A container finding becomes actionable when it is tied to a digest, the pods that run it and the controllers that can recreate it. The safe order is to freeze promotion, inventory exposure, select a clean digest, redeploy by digest, validate behavior and only then decide what happens to the suspect manifest.

The final decision must prove three separate properties: the vulnerable digest no longer runs, it cannot be promoted again, and the service has a rollback path that does not reintroduce the same risk.