Azure DevOps: diagnose an offline self-hosted agent before recreating it

When an Azure DevOps self-hosted agent goes offline, the usual reaction is simple: restart the VM, remove the agent from the pool, or recreate a clean runner. That may unblock a pipeline, but it also destroys evidence. If the cause is an expired token, a proxy change, a full disk, an identity that can no longer reach repositories, a stopped systemd service, or a broken outbound route, recreating the agent does not fix the underlying problem.

The scenario is familiar: a team deploys to production through self-hosted agents placed inside a private Azure network. Jobs remain queued, the agent appears offline or appears online but no longer accepts work, and an urgent release is waiting. The runbook must decide whether to restart the service, clean a cache, fix networking, re-authorize the agent, switch to a standby pool, or roll back the latest infrastructure change.

Treat the incident as a contract

Before touching the runner, define the scope. An offline agent is not precise enough: the whole pool may be unavailable, one agent may be degraded, only one type of job may be failing, or Azure DevOps may be rejecting authentication.

Incident to qualify
Organization: naxaya-devops
Pool: prod-private-linux
Agent: ado-prod-agent-03
Host: vm-ado-agent-03
Symptoms:
  - agent offline since 09:42
  - deployment jobs queued
  - last completed job: deploy-api-prod #1842
Recent change:
  - VM image rotation yesterday evening
  - new outbound route to firewall

Expected decision
Restart the agent service
Fix disk, cache, network or identity
Switch to a standby pool
Recreate the agent only after preserving evidence

This framing prevents the team from confusing speed with diagnosis. It also helps communicate clearly with the product team: the pipeline is not just slow, it is waiting for a private execution capacity that must be restored or replaced.

Read the Azure DevOps side first

The first check is on the service side: does the pool see the agent, since when, with which version, demands and capabilities? An online agent can still be unusable if job demands no longer match its capabilities. An offline agent may simply have lost its long-running session.

Azure DevOps checks
Target pool and active queues
Agent online/offline state and last contact
Agent version and pending update
Capabilities exposed by the agent
Demands used by the pipeline
Recent jobs on the same agent
Recent jobs on other agents in the pool

If every agent in the pool is offline, the problem looks like a network, proxy, DNS or organization-level incident. If only one agent is affected, focus on the host, local service, disk, cache or runtime identity.

Check the local service before cleaning anything

On the host, the goal is not to restart immediately. Capture state first. The service may be stopped, stuck in a reconnect loop, unable to read its working directory, or wedged after a partial update.

sudo systemctl status vsts.agent.* --no-pager
sudo journalctl -u 'vsts.agent.*' --since '2 hours ago' --no-pager
ps aux | grep '[A]gent.Listener'
df -h
find /opt/azdo-agent/_diag -type f -mtime -1 -maxdepth 1 -print

The _diag files are often more useful than an immediate restart. They show whether the agent can no longer reach Azure DevOps, refuses a job, fails while preparing the workspace, or hits an authentication error.

Separate outbound networking, proxy and DNS

A private agent almost always needs outbound access to Azure DevOps, artifact registries, repositories, Key Vault, container registries or security endpoints. A UDR, proxy, firewall rule or DNS resolution problem can be enough to make it appear offline.

Outbound path to validate
DNS resolver used by the agent
Proxy configured in .env, systemd or shell variables
Effective route from the NIC
Firewall rules for dev.azure.com and dependencies
TLS inspection or enterprise certificate
Access to registries and feeds used by jobs

Useful evidence
DNS resolution from the host
curl with HTTP code and connection timing
Firewall logs for the agent source IP
Comparison with a healthy agent in the same pool

The right test is not just whether the host can reach the Internet. Test the endpoints actually used by the runner and by the job. An agent can stay online and still fail as soon as it must download an artifact, pull an image or read a secret.

Separate runner identity from job identity

The Azure DevOps agent uses a pool registration and an authentication mechanism. Jobs usually use other identities: service connections, managed identities, OIDC federation, feed keys, registry tokens. Keep the runner identity and the deployment identity separate.

Runner identity
Lets the agent connect to the Azure DevOps pool
Can break because of an expired token, reconfiguration or pool deletion

Job identity
Lets the pipeline deploy, read Key Vault, push an image or call Azure
Can break because of RBAC, OIDC federation, secret rotation or policy

Decision
Do not recreate the agent when only the job is denied
Do not broaden Azure RBAC when only the runner is offline

This avoids two opposite mistakes: recreating the agent while a service connection is broken, or widening Azure permissions while the runner cannot even register with the pool.

Read disk, workspace and cache without wiping everything

Self-hosted agents accumulate workspaces, tools, npm caches, Docker layers, repositories and temporary artifacts. A full disk can destabilize the agent, but a blunt cleanup can also hide the job that filled the host.

df -h
sudo du -xh /opt/azdo-agent/_work --max-depth=2 | sort -h | tail -20
sudo du -xh /var/lib/docker --max-depth=1 2>/dev/null | sort -h | tail -10
sudo find /opt/azdo-agent/_work -maxdepth 2 -mtime -1 -type d -print | head

The fix should be targeted: remove old workspaces, purge a documented Docker cache, increase the disk if the workload changed, or move selected artifacts. If the agent repeatedly runs out of space, rollback is not a one-time cleanup; it is reverting the job or image change that introduced the drift.

Decide between restart, failover and recreation

After collecting evidence, make the decision explicit. Restarting the service is acceptable when the agent lost its session but disk, network and identity are healthy. Switching to a standby pool is better when a production delivery is urgent and the alternate pool has the same controls. Recreating the agent is a last resort when local registration is corrupted or the image must be rebuilt.

Restart the service
Service stuck, logs captured, network OK, disk OK

Clean or resize
Disk full, cause identified, impacted jobs known

Fix networking
DNS, proxy, route or firewall proven by logs

Switch to standby pool
Production urgency, validated standby pool, compatible secrets and demands

Recreate the agent
Corrupted registration, non-compliant image, evidence preserved, old agent removed cleanly

Each option needs validation: a short diagnostic job, a non-destructive pipeline, artifact retrieval, Key Vault access if needed, and only then a production deployment retry.

Prepare rollback

Rollback depends on the cause. If a recent UDR broke outbound access, restore the previous route. If a VM image broke the service, reactivate the previous image or replace the host with a known-good agent. If cache cleanup removed a required tool, run the controlled installer instead of patching the host by hand.

Minimal rollback
Network: restore previous route, proxy or firewall rule
Image: return to the previous scale set or template
Agent: remove the corrupted agent from the pool after saving _diag
Pipeline: restore previous demands or pool
Identity: restore the validated service connection or federation

Post-rollback validation
Agent online in the pool
Diagnostic job completed
Artifact and secret access confirmed
No production job restarted without approval

Conclusion

An offline Azure DevOps self-hosted agent is not just a component to recreate. It is a private execution capacity that depends on a local service, a pool, outbound networking, disk, cache and several identities. A useful runbook preserves evidence, separates runner and job, tests the real path and chooses a reversible action.

The right decision is not always the most dramatic one. If the service can be restarted with clean logs, restart it. If the outbound route is broken, roll back the route. If the image is bad, replace the host with a known-good version. Recreating the runner then becomes a documented decision, not a reflex that erases the incident.