Azure Container Registry: diagnose an image pull failure before redeploying

An image pull failure often looks like an application problem: a pod stays in ImagePullBackOff, a Container Apps revision never starts, a private runner fails before deployment, or an AKS release is cancelled because the expected image never reaches the node. The tempting reaction is to republish the image, rerun the pipeline or redeploy the workload.

In production, that reaction can hide the real risk. An ACR pull crosses several layers: image name, tag or digest, identity, AcrPull permissions, DNS resolution, registry firewall, node or runner network, registry availability, local cache and orchestrator logs. A useful runbook should not only make the pull pass. It must prove which layer blocks, which correction is minimal and how to return to a known image if the fix shifts the incident.

The use case is an Azure platform with AKS, Azure Container Apps or private CI runners pulling images from Azure Container Registry. Some environments use filtered public access, others use a private path or an egress firewall. Private Endpoint may exist in the path, but it is only one possible component: the failure may just as well come from identity, a missing tag, an ACR rule or outbound routing.

Name the exact symptom

Before changing the registry or the cluster, classify the error message. ImagePullBackOff, ErrImagePull, 401 Unauthorized, 403 Forbidden, manifest unknown, connection timed out and no such host do not point to the same causes.

Observed symptom
ImagePullBackOff or ErrImagePull
  Read the Kubernetes or Container Apps event before rerunning

401 Unauthorized
  Check the identity used and the authentication mechanism

403 Forbidden
  Check AcrPull, scope, ACR firewall and network access

manifest unknown or tag not found
  Check repository, tag, digest and image promotion

no such host
  Check DNS from the node, runner or Container Apps environment

connection timed out
  Check routing, egress firewall, NAT, proxy and registry network rules

This step avoids a common mistake: granting AcrPull to an identity that already has it while the tag does not exist, or republishing an image when the nodes can no longer resolve the registry.

Pin the expected image

A mutable tag makes diagnosis fragile. The first control is to write down the exact expected image, then verify whether it exists in ACR. Whenever possible, diagnose with a digest rather than a tag.

ACR_NAME="acrprod01"
REPOSITORY="payments/api"
TAG="2026.06.28.4"

az acr repository show-tags --name "$ACR_NAME" --repository "$REPOSITORY" --orderby time_desc --output table

az acr manifest list-metadata --registry "$ACR_NAME" --name "$REPOSITORY" --query "[?tags[?@=='$TAG']].{digest:digest,tags:tags,createdTime:createdTime}" --output table

If the expected digest is absent, the problem is upstream: build, push, promotion or retention. Rerunning deployment will not fix a registry that does not contain the requested object.

Identify the real pull identity

The pull is not always executed by the identity the team has in mind. AKS may use a kubelet identity or a dedicated managed identity. Container Apps may use a managed identity configured on the app or environment. A private runner may use OIDC, a service principal, az acr login or a Docker secret.

AKS_RG="rg-aks-prod"
AKS_NAME="aks-prod"
ACR_ID="$(az acr show --name acrprod01 --query id -o tsv)"

az aks show --resource-group "$AKS_RG" --name "$AKS_NAME" --query "{kubeletIdentity:identityProfile.kubeletidentity.objectId,managedIdentity:identity.principalId}" --output json

KUBELET_OBJECT_ID="$(az aks show --resource-group "$AKS_RG" --name "$AKS_NAME" --query "identityProfile.kubeletidentity.objectId" -o tsv)"

az role assignment list --assignee "$KUBELET_OBJECT_ID" --scope "$ACR_ID" --query "[].{role:roleDefinitionName,scope:scope}" --output table

The useful criterion is not “an identity has AcrPull”. It is “the identity that really performs the pull has AcrPull on the right registry or scope”. If the registry is shared across environments, avoid widening access to the whole resource group without evidence.

Test the network path from the right place

Testing from an admin workstation or Cloud Shell is not enough. The pull starts from the node, Container Apps environment or runner. DNS, TLS and egress must be tested from that path.

REGISTRY="acrprod01.azurecr.io"

getent hosts "$REGISTRY"

timeout 5 bash -c "cat < /dev/null > /dev/tcp/$REGISTRY/443"

openssl s_client -connect "$REGISTRY:443" -servername "$REGISTRY" </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer

If ACR is exposed through Private Endpoint, verify the privatelink.azurecr.io zone, VNet links, forwarders and resolution from the consuming network. If the registry remains public but filtered, verify the real outbound IP, NAT Gateway, Azure Firewall, proxy and ACR allowlist.

Read orchestrator events

Kubernetes events and Container Apps logs often contain the raw cause. Keep them in the ticket before any correction, because a redeployment can erase useful context.

NAMESPACE="payments"
DEPLOYMENT="payments-api"

kubectl -n "$NAMESPACE" get pods -l app="$DEPLOYMENT" -o wide

kubectl -n "$NAMESPACE" describe pod "$(kubectl -n "$NAMESPACE" get pod -l app="$DEPLOYMENT" -o jsonpath='{.items[0].metadata.name}')" | sed -n '/Events:/,$p'

kubectl -n "$NAMESPACE" get events --sort-by=.lastTimestamp --field-selector type=Warning

For Container Apps, read the active revision, system errors and console logs. The symptom may be an impossible pull, but it may also be a revision that keeps running an older image because the new one was never available.

Correlate ACR, Azure activity and network logs

The registry can prove whether the request arrives, which identity is seen and whether the denial comes from the service or the network. Depending on the setup, ACR logs, AzureActivity, Firewall or Application Insights provide the useful clue.

let Start = datetime(2026-06-28 08:00:00);
let End = datetime(2026-06-28 09:00:00);
ContainerRegistryRepositoryEvents
| where TimeGenerated between (Start .. End)
| where Repository has "payments/api"
| project TimeGenerated, OperationName, Repository, Tag, Digest, ResultType, CallerIpAddress, Identity
| order by TimeGenerated desc

Adapt the query to the tables available in the workspace. If no ACR log appears during the pull window, traffic probably does not reach the registry or diagnostics do not cover that event. In that case, firewall logs, node events and DNS tests become more important.

Choose the smallest correction

The correction should follow the diagnosis, not the broad symptom. A blocked pull does not automatically justify a full redeployment, temporary public access or an overbroad role.

Fix without application rollback
The tag or digest exists and the real identity lacks AcrPull
An ACR firewall rule blocks the expected outbound IP
A DNS link or forwarder is missing for the private registry
The runner uses an old Docker secret that can be replaced

Rollback the workload
The new image is absent or corrupted
The active revision cannot start inside the change window
The expected digest does not match the validated image
Several critical environments can no longer pull from ACR

Refuse an overbroad correction
Enabling publicNetworkAccess without a window and evidence
Granting AcrPull to the whole group by default
Replacing the tag without attaching the expected digest
Purging node cache before capturing events

The cleanest rollback is often to return to a previously validated digest or revision while keeping the ACR diagnosis open. That restores service without deleting the evidence.

Validate after correction

After the fix, prove three things: the registry contains the image, the pull path works from the real environment, and the workload starts with the expected digest.

NAMESPACE="payments"
DEPLOYMENT="payments-api"
EXPECTED_DIGEST="sha256:0123456789abcdef..."

kubectl -n "$NAMESPACE" rollout status deploy/"$DEPLOYMENT" --timeout=120s

kubectl -n "$NAMESPACE" get pods -l app="$DEPLOYMENT" -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].imageID}{"\n"}{end}'

kubectl -n "$NAMESPACE" get pods -l app="$DEPLOYMENT" -o jsonpath='{range .items[*]}{.status.containerStatuses[*].ready}{"\n"}{end}'

If pods start with a different digest, the CI/CD chain or tagging strategy remains ambiguous. Validation should then block incident closure even if the service appears to be back.

Conclusion

An ACR pull failure is an execution-chain incident, not just a container problem. Diagnosis should start from the exact symptom, verify image existence, identify the real identity, test the network from the right place and correlate logs before correcting.

The healthy decision is direct: fix the proven layer when the image is good, return to a validated digest when the release is uncertain, and refuse broad openings that turn a pull incident into durable security debt.