Azure deployment from a private runner: diagnose the pipeline before rerun

An Azure deployment can fail even when application code has not changed. The pipeline runs the same template, script or Terraform plan, but from a private runner inside a VNet, behind a firewall, using federated or managed identity, reading Key Vault, pushing to a private registry and sometimes reaching filtered Azure control-plane endpoints. Rerunning the job without diagnosis can hide a network issue, consume a change window or roll back the wrong thing.

The use case is a GitHub Actions or Azure DevOps pipeline executed on a private runner. It deploys an Azure application, reads secrets, pushes an image, applies infrastructure and triggers validation. The runbook has one goal: decide whether the failure comes from the runner, identity, network path, called dependency, deployment code or a partial change already applied.

Freeze the exact deployment state

Before rerun, capture what really happened. A pipeline can fail before authentication, while reading secrets, when reaching Azure Resource Manager, after a partial apply, or during post-deployment validation. These states do not require the same rollback.

State to capture
Run ID and deployed commit
Runner, runner group and subnet
Requested Azure identity
Exact failed step
Resources created, modified or untouched
Azure correlation ID or client request ID
Latest network, secret, role or policy change

Possible decision
Failure before authentication: check federation, secret or identity
Failure before Azure change: fix runner path or dependency
Failure during apply/deploy: qualify partial state before rerun
Failure after deploy: validate application impact before rollback

This record protects against the reflex to press rerun. If the first pass already changed a route, an App Service setting or a container image, the second pass can make the diagnosis worse instead of cleaner.

Separate runner, identity and authorization

A private runner creates a common ambiguity: the pipeline error often mixes the execution host, identity token and Azure permissions. Prove separately that the runner is healthy, that the effective identity is expected, and that the assigned rights match the operation.

az account show --query '{tenant:tenantId, subscription:id, user:user.name}' --output json

az ad signed-in-user show --query '{id:id, userPrincipalName:userPrincipalName}' --output json 2>/dev/null || echo "No interactive user, check service principal, workload identity or managed identity"

az role assignment list --assignee "$AZURE_CLIENT_ID" --scope "/subscriptions/$AZURE_SUBSCRIPTION_ID" --query '[].{role:roleDefinitionName, scope:scope}' --output table

az deployment sub list --query '[0:5].{name:name, state:properties.provisioningState, timestamp:properties.timestamp}' --output table

With OIDC, also check the federated subject, allowed GitHub or Azure DevOps environment, branch and workflow. A 403 may come from a missing role, but also from a federation rule that no longer matches the job context.

Prove the runner network path

The private runner must reach several target families: Azure Resource Manager, Entra ID, Key Vault, Container Registry, the Terraform backend, the application validation endpoint and sometimes an internal API. One broken access path can fail the whole deployment.

Private runner
Subnet, NSG, route table, DNS resolver
Internet egress through NAT Gateway or firewall
Explicit proxy or TLS inspection

Deployment targets
management.azure.com for ARM
login.microsoftonline.com for authentication
vault.azure.net for secrets
*.azurecr.io for images
Terraform state backend or artifact storage
Private or public application validation endpoint

Expected evidence
DNS resolution from the runner
TCP/TLS connection with the real FQDN
Outbound IP observed by allowlisted services
Firewall or NSG rule explaining allow or block

Private Endpoint may be involved for Key Vault, Storage or ACR, but it should not become the only diagnostic lens. The runner may also fail on a public Azure endpoint, a partner allowlist, a UDR toward a firewall, or TLS inspection that breaks the flow.

Test DNS, TLS and egress without deploying

The pipeline rerun should not be the first network test. A read-only diagnostic job on the same runner or runner group can verify dependencies without applying any change.

for host in management.azure.com login.microsoftonline.com "$KEYVAULT_NAME.vault.azure.net" "$ACR_NAME.azurecr.io"
do
echo "== $host =="
getent hosts "$host" || nslookup "$host"
timeout 5 bash -lc "cat </dev/null >/dev/tcp/$host/443"   && echo "tcp_connect_ok=true"   || echo "tcp_connect_ok=false"
openssl s_client -connect "$host:443" -servername "$host" </dev/null 2>/dev/null   | openssl x509 -noout -subject -issuer
done

curl -sS --connect-timeout 5 https://ifconfig.me || true

Compare the result with the expected design: private or public resolution, NAT Gateway egress, firewall path, proxy, or documented refusal. Resolution that “works” is not enough if it exits through the wrong path.

Read logs as a timeline

A deployment run should be correlated with Azure logs and network logs when available. The goal is not to collect every signal, but to rebuild the order: authentication, dependency read, ARM call, resource change, validation.

let Window = 2h;
let CorrelationId = "00000000-0000-0000-0000-000000000000";
AzureActivity
| where TimeGenerated > ago(Window)
| where CorrelationId == CorrelationId
 or Caller has_any ("github", "devops", "spn", "managedidentity")
| project TimeGenerated,
        OperationNameValue,
        ActivityStatusValue,
        ResourceGroup,
        ResourceProviderValue,
        ResourceId,
        Caller,
        CorrelationId,
        Properties
| order by TimeGenerated asc

For Key Vault, ACR, Storage or firewall, add a parallel view using the same time window, runner identity, source IP or client request ID. The right decision rarely comes from the pipeline log alone.

Decide rerun, fix or rollback

A rerun is acceptable only when the action is idempotent and the cause is understood. Otherwise, fix the path, reduce the scope or roll back the partial change.

Rerun the pipeline
The failure is proven transient
No dangerous partial change exists
The plan or template is unchanged
Read-only validations pass from the runner

Fix the runner or network
DNS, NSG, UDR, firewall, proxy or NAT explains the failure
The deployment did not touch the target resource
The network test passes before a new execution

Fix identity or authorization
The effective identity is proven
The denial comes from RBAC, OIDC federation, policy or Key Vault
No network opening is added to hide a 403

Roll back
A partial change modified production
Application validation fails
Rerun may widen the impact
Reverting is safer than a second apply

Do not confuse pipeline rollback with production rollback. Canceling a workflow does not remove an already modified resource. Conversely, rolling back the application will not fix a broken OIDC federation rule or a bad route applied to the runner subnet.

Keep recovery bounded

After the fix, recovery should remain controlled. A production pipeline should be able to replay a targeted stage, run a plan without apply, verify dependencies from the runner and produce an impact summary.

rerun_controls:
require_same_commit: true
require_plan_before_apply: true
require_runner_network_probe: true
allow_targeted_stage:
  - authenticate
  - validate_dependencies
  - plan
  - apply_scoped
  - post_deploy_checks
block:
  - apply_all_after_unknown_partial_failure
  - rotate_secret_during_network_incident
  - widen_firewall_without_correlation_id
rollback_record:
  include:
    - changed_resources
    - deployment_correlation_id
    - identity_used
    - validation_result

This recovery mode turns the pipeline into an operations interface, not just a deployment button. It gives the team a clear path: prove, fix, rerun within limits, then keep the evidence.

Conclusion

An Azure pipeline executed from a private runner is a production component. It depends on a network, an identity, routes, DNS, logs and services called during deployment. When it fails, immediate rerun often destroys the most useful evidence.

The runbook should therefore end in an explicit decision: rerun if the failure is understood and idempotent, fix the runner or identity if the path is wrong, or roll back if a partial change touched production. Deployment quality is not only whether the job turns green, but whether the team can prove what happened before running it again.