AWX: rerun a failed job without replaying a partial action

A failed AWX job creates a very simple temptation: click rerun. Sometimes that is the right move, but only if the failure happened before any state change or if the playbook is genuinely idempotent. In production, a blind rerun can restart a service twice, reapply configuration to only part of the estate, overwrite useful evidence or hide the original cause.

The use case is a bounded AWX maintenance template that applies a controlled fix to a group of servers. The job fails halfway through execution: two hosts reached changed, one host failed, the remaining hosts were not touched. The runbook should help decide whether to rerun the whole job, rerun only the failed hosts, fix variables, trigger rollback or stop automation long enough to review the playbook.

Freeze context before any rerun

The first action is not to fix. It is to preserve diagnostic state. An AWX job already contains a lot of evidence: template, inventory, project, Git revision, launch variables, credential, Ansible events and affected hosts. If the team reruns too quickly, it mixes the original failure with a second attempt.

JOB_ID="12345"

awx jobs get "$JOB_ID" --format json | jq '{id,name,status,failed,started,finished,elapsed,launch_type,inventory,project,job_template,job_explanation,scm_revision,extra_vars}'

awx job_events list --job "$JOB_ID" --format json | jq '.results[] | {counter,event,host:.host_name,task:.event_data.task,changed:.event_data.res.changed,failed:.event_data.res.failed,msg:.event_data.res.msg}' > "awx-job-$JOB_ID-events.jsonl"

This capture does not replace analysis, but it keeps the thread intact. It also answers the first operational question: did the failure happen before or after an action changed target state?

Split changed, failed and untouched hosts

An AWX job does not always fail uniformly. A playbook may succeed on a few hosts, fail on one host, then stop the rest depending on the Ansible strategy. The rerun decision depends on real scope, not only on the global red status.

JOB_ID="12345"

awx job_events list --job "$JOB_ID" --event runner_on_ok --format json | jq -r '.results[] | select(.event_data.res.changed == true) | [.host_name, .event_data.task] | @tsv' | sort -u > changed-hosts.tsv

awx job_events list --job "$JOB_ID" --event runner_on_failed --format json | jq -r '.results[] | [.host_name, .event_data.task, (.event_data.res.msg // .event_data.res.stderr // "no message")] | @tsv' | sort -u > failed-hosts.tsv

awx job_events list --job "$JOB_ID" --format json | jq -r '.results[] | select(.host_name != null) | .host_name' | sort -u > seen-hosts.txt

Three cases change the reading. If no host changed, rerun after fixing the cause is often acceptable. If some hosts changed before failure, idempotence must be proven or the next step must be limited. If many hosts were not touched, the rerun should avoid reprocessing hosts that already applied the change.

Check variables and executed revision

An AWX rerun should start from the same contract unless the correction is deliberate. Compare the Git revision, variables, credential and inventory. A failure caused by a bad variable is not handled like a temporary network failure.

Context to compare
Same job template
AWX project and Git revision identified
Same inventory or documented limited rerun
Extra vars preserved or explicitly corrected
Same credential or approved credential change
Same execution environment
Host limit documented when rerun is partial

Questions before rerun
Is the playbook idempotent on tasks already changed?
Did an incorrect variable cause the failure?
Is rollback safer than rerun?
Should the rerun target all hosts, only failed hosts, or only unreachable hosts?

If a variable changes between attempts, this is no longer a simple rerun. It is a new operational change. It needs its own evidence and, when appropriate, its own approval.

Classify the cause before choosing the action

The failed status is not enough. Classify the failure family: parameter error, unreachable host, permission issue, external dependency, non-idempotent task, role bug or missing prerequisite. Each family calls for a different response.

Parameter error
Fix extra_vars or survey input
Do not rerun without new approval if scope changes

Unreachable host
Check DNS, SSH, bastion, credential and maintenance window
Rerun only unreachable hosts if no task changed

Permission denied
Check AWX credential and target sudoers
Do not broaden rights without proof of the exact need

Task changed then failed
Prove idempotence or prepare rollback
Limited and monitored rerun

Role bug or dependency failure
Fix the repository, sync the AWX project, rerun on bounded scope
Keep the faulty revision in the ticket

This classification avoids two extremes: rerunning everything by reflex or freezing all automation when a targeted rerun would be enough.

Choose a limited rerun when possible

The safest rerun is often the smallest one that restores the expected state. If one host failed after a temporary dependency issue, there is no need to process the full inventory again. If several hosts already changed, rerunning only failed hosts reduces the risk of double action.

TEMPLATE_ID="42"
FAILED_LIMIT="web03.example.local"

awx job_templates launch "$TEMPLATE_ID" --limit "$FAILED_LIMIT" --extra_vars @approved-extra-vars.yml --monitor

The limit must be written into the ticket or incident note. An undocumented partial rerun becomes hard to explain if post-action validation finds divergent states across hosts.

Validate final state, not only green status

A rerun that ends green does not prove the operation is healthy. Validate the expected effect on hosts already changed, hosts rerun and hosts never touched. The exact check depends on the action, but it must be explicit.

validation:
changed_hosts:
  - check service is active and expected version is present
  - confirm no unintended double restart occurred
failed_hosts_rerun:
  - verify the task that originally failed
  - compare final configuration with baseline
untouched_hosts:
  - confirm absence of unexpected change
evidence:
  - initial job and rerun job linked in the ticket
  - Git revision and extra_vars preserved
  - rollback decision documented if drift is detected

Validation should also look at side effects: alerts, application logs, service metrics, authentication errors or processing queues depending on the playbook scope.

Prepare rollback when rerun is not safe

Some situations should not be rerun. If a non-idempotent task modified part of the estate, if variables were wrong or if the role applied an incomplete configuration, the right move may be a controlled rollback before any new attempt.

Rerun
No changed task before failure
Temporary cause fixed
Same revision and same variables
Limited rerun possible

Rerun with human validation
Partial changed state but playbook is idempotent
Isolated failed host
Post-action validation available
Change window still open

Rollback
Partial changed state is not idempotent
Incorrect variables were applied
Final configuration diverges
User impact or monitoring degradation is present

Stop and fix code
Defective role
Missing precondition
Overly permissive template
Inconsistent credential or inventory

Rollback should be as bounded as the rerun: same hosts, same evidence, same validation requirement. Otherwise the team replaces one partial action with an opposite action that is just as hard to explain.

Conclusion

Rerunning a failed AWX job is an operations decision, not a UI gesture. Before clicking, know what changed, on which hosts, with which variables and which revision. The red status tells you there is a problem; Ansible events tell you whether rerun is safe.

The healthy decision keeps scope explainable: full rerun if nothing changed, limited rerun if the cause is isolated, variable correction if the contract was wrong, rollback if the partial state is dangerous. That discipline keeps AWX useful as a reliable operations tool instead of a risky repeat button.