Automation
Azure DevOps: diagnose a self-hosted agent before rerunning the pipeline
A production runbook for qualifying an Azure DevOps failure on a self-hosted agent with disk, workspace, cache, local services, identity, logs, controlled cleanup, validation and rollback.
An Azure DevOps job that fails on a self-hosted agent often triggers the same reflex: click rerun. If the agent only hit a transient error, that may be enough. If the disk is full, the workspace still contains state from a previous build, a cache is corrupted, Docker kept obsolete layers, or the local service lost the expected identity, the rerun only replays the same broken state with more noise.
The use case is a product or platform team running CI/CD pipelines on a private agent pool, usually because the jobs need internal network access, specific tooling, proximity to Azure dependencies or tighter governance than hosted agents provide. The runbook goal is to decide whether to rerun the job, clean the agent, remove it from the pool, fix the base image, or roll back a recent pipeline change before the delivery chain becomes unreliable.
Treat the agent as a production dependency
A self-hosted agent is not only a machine that runs commands. It is an operational dependency: disk, workspace, runtime, local secrets, network path, identity, agent version, tools, caches and permissions on target resources.
Pipeline incident
Azure DevOps project: platform-prod
Pipeline: build-and-deploy-orders
Run: 20260715.2
Failed stage: package-and-publish
Pool: private-linux-prod
Agent: azdo-agent-prod-03
Last known healthy run: 20260714.6
Recent change: SDK image update + npm cache enabled
Questions before rerun
Is the failure caused by the code or by the agent?
Are disk, workspace or cache full or corrupted?
Did the agent keep files from a previous run?
Is the agent service running under the expected identity?
Does the issue affect one agent or the whole pool?
Which cleanup is allowed without destroying useful evidence?If the contract is not written down, rerun becomes a guess. The point is not to block every rerun, but to avoid replaying degraded local state in production.
Separate application failure, agent failure and pool drift
Start by classifying the failure. A build error that reproduces on every agent is not handled like a broken Maven cache on a single machine.
Likely application failure
Same commit fails on several agents
Message matches test, compilation or packaging failure
Clean workspace and dependencies restored correctly
No disk, permission or missing tool signal
Likely local agent failure
Only one agent fails in the pool
Logs include no space left, permission denied, file locked, checksum mismatch
Cache or workspace reused across runs
Local tool differs from the expected version
Agent service restarted or updated recently
Likely pool drift
Several agents fail after image or bootstrap change
Same tool missing across the pool
Service connection, proxy, DNS or firewall changed
New cleanup or cache policy applied everywhereThe decision follows that classification. Rerunning on another agent can be useful to isolate a machine. Rerunning on the same agent without evidence only adds another failed run.
Capture evidence before cleanup
Cleanup is useful, but it often destroys evidence. Before deleting a workspace, capture the elements that explain the failure and prevent it from coming back.
AGENT_HOME="/opt/azdo-agent"
WORK="/opt/azdo-agent/_work"
RUN_ID="20260715.2"
hostname
whoami
date -u
systemctl status vsts.agent.* --no-pager || true
printf "
Disk usage
"
df -h
printf "
Workspace size
"
du -sh "$WORK"/* 2>/dev/null | sort -h | tail -20
printf "
Recent agent logs
"
find "$AGENT_HOME/_diag" -type f -mtime -2 -maxdepth 1 -print | sort | tail -10
printf "
Large files in workspace
"
find "$WORK" -type f -size +500M -printf "%s %p
" 2>/dev/null | sort -n | tail -20Attach this evidence to the Azure DevOps run: job id, exact agent, commit, artifact, failed step and UTC time. Without that mapping, a successful cleanup does not prove the cause.
Check disk, workspace and caches
The most ordinary failures are often the most expensive: disk filled by artifacts, corrupted cache, full temporary directory, dependencies restored in a partial state, Docker layers never pruned.
WORK="/opt/azdo-agent/_work"
# Adjust thresholds to the pool.
df -h / /tmp "$WORK"
# Heaviest workspaces.
du -xhd 2 "$WORK" 2>/dev/null | sort -h | tail -30
# Common caches depending on workloads.
du -sh ~/.npm ~/.cache ~/.m2 ~/.nuget ~/.gradle 2>/dev/null || true
# Docker when the agent builds images.
docker system df 2>/dev/null || true
# Recent files that explain a sudden growth.
find "$WORK" -type f -mtime -2 -printf "%TY-%Tm-%Td %TH:%TM %s %p
" 2>/dev/null | sort | tail -50A cache is not wrong by default. It becomes an incident when nobody knows what it contains, how large it is, who prunes it and how the pipeline behaves without it.
Verify identity and local services
An agent can have enough disk and still fail because the service no longer runs under the right identity, workspace permissions drifted, or a local tool changed version.
AGENT_HOME="/opt/azdo-agent"
WORK="/opt/azdo-agent/_work"
systemctl cat vsts.agent.* --no-pager || true
ps -eo user,group,pid,cmd | grep -E "Agent.Listener|Agent.Worker" | grep -v grep || true
stat -c "%U %G %a %n" "$AGENT_HOME" "$WORK"
find "$WORK" -maxdepth 2 -type d -printf "%u %g %m %p
" 2>/dev/null | head -40
node --version 2>/dev/null || true
dotnet --info 2>/dev/null | head -40 || true
docker version 2>/dev/null || true
az version 2>/dev/null || trueIf the agent writes files as root during a Docker step and later tries to clean them as an unprivileged user, the next rerun can fail before the application even builds.
Compare with a healthy agent
The most useful test is often to compare the suspicious agent with a healthy agent from the same pool. Do not chase every difference. Look for the differences that explain the run.
compare:
failing_agent: azdo-agent-prod-03
healthy_agent: azdo-agent-prod-01
same_pool: private-linux-prod
same_pipeline_capabilities: true
checks:
- agent_version
- service_user
- free_disk_percent
- workspace_size
- docker_cache_size
- tool_versions
- proxy_dns_firewall_path
- last_bootstrap_commit
- last_successful_pipeline_run
decision:
same_state: investigate pipeline or artifact
local_drift: quarantine failing agent and clean
pool_drift: rollback bootstrap or image changeThis comparison avoids a common mistake: changing the pipeline when one agent is dirty, or cleaning a machine when the whole pool received a bad image.
Clean with a controlled scope
Cleanup must be explicit. Deleting everything on disk can hide a leak, break a shared tool or remove logs needed for the incident record.
WORK="/opt/azdo-agent/_work"
# Safer pattern: remove the agent from the pool before cleanup through Azure DevOps UI/API,
# then clean completed job workspaces after evidence has been saved.
systemctl stop vsts.agent.*
# Adapt the scope to the pool convention.
find "$WORK" -mindepth 1 -maxdepth 1 -type d -name "[0-9]*" -mtime +2 -print
# Run deletion only after operator validation.
# find "$WORK" -mindepth 1 -maxdepth 1 -type d -name "[0-9]*" -mtime +2 -exec rm -rf {} +
# Controlled Docker cleanup when the pool builds images.
docker system df 2>/dev/null || true
# docker builder prune --filter "until=72h" --force
systemctl start vsts.agent.*For a production agent, it is often better to remove the agent from the pool, clean it, run a validation job, then return it to service. That prevents a critical job from landing on a machine under repair.
Validate before rerunning the real pipeline
A production rerun is not an agent health check. First run a short job that validates the required dependencies: checkout, dependency restore, workspace write, network access, registry access and publication of a test artifact.
steps:
- checkout: self
clean: true
- script: |
set -euo pipefail
echo "agent=$(Agent.Name)"
df -h
mkdir -p "$(Pipeline.Workspace)/agent-validation"
echo "validation" > "$(Pipeline.Workspace)/agent-validation/probe.txt"
test -s "$(Pipeline.Workspace)/agent-validation/probe.txt"
displayName: Validate workspace and disk
- script: |
set -euo pipefail
node --version || true
dotnet --info | head -20 || true
az version | head -20 || true
displayName: Validate toolchain
- publish: $(Pipeline.Workspace)/agent-validation/probe.txt
artifact: agent-validation
displayName: Publish validation artifactIf this job fails, the application pipeline is not the right place to continue the diagnosis.
Decide rerun, quarantine or rollback
Finish with a decision that can be read later. The runbook should produce more than a manually cleaned machine.
Rerun on the same agent
Disk healthy
Workspace clean
Cache validated or temporarily disabled
Agent service and identity compliant
Validation job succeeded
Rerun on another agent
Local suspicion is not blocking
Need to separate code from machine state
Suspicious agent removed from the pool for investigation
Quarantine the agent
Disk or cache corrupted
Permissions inconsistent
Local tools outside expected version
Logs insufficient to authorize a critical job
Roll back pipeline or image change
Several agents affected
Failure appeared after bootstrap, cache policy or toolchain update
Validation fails on a pool that was healthy before the changeThe right result may be a rerun, but only after proving that the agent is not replaying a broken local state.
Conclusion
A self-hosted Azure DevOps agent gives control: private network access, managed tooling, proximity to dependencies and pool governance. It also adds an operational responsibility. Disk, cache, workspace, identity and bootstrap must be treated as production components.
Before rerunning a failed pipeline, qualify local state, compare with a healthy agent, clean with a documented scope and validate with a short job. The final decision becomes defensible: rerun, quarantine, pool correction or rollback of the change that made the agent unreliable.