AWX: validate dynamic inventory before a production job

AWX dynamic inventory feels comfortable: hosts come from the source of truth, groups refresh automatically and jobs no longer depend on static files. In production, that comfort becomes risky when a filter, credential or variable change silently brings new hosts into scope.

The use case is common: a team synchronizes inventory from Azure, VMware, a CMDB or an internal script. A maintenance job must restart an agent, apply a baseline or verify configuration. Before launching the template, the team notices that the latest sync added hosts, removed a group or changed variables. The right move is not to run the job “to see what happens”. Inventory must be qualified as a production dependency.

This runbook helps decide whether the inventory is safe for the intended job, whether the host limit must be narrowed, whether the source should be rolled back or whether execution should be blocked.

Capture the source before syncing

Before clicking sync, identify what produces the inventory: plugin, credential, AWX organization, project, configuration file, API query, tags or groups used as filters. Dynamic inventory is not neutral: it translates external metadata into executable targets.

INVENTORY_ID="17"

awx inventories get "$INVENTORY_ID" --format json | jq '{id,name,organization,variables,kind}'

awx inventory_sources list --inventory "$INVENTORY_ID" --format json | jq '.results[] | {id,name,source,credential,source_project,source_path,source_vars,update_on_launch,overwrite,overwrite_vars}'

Two options deserve particular attention. overwrite can remove hosts that disappear from the source. overwrite_vars can replace variables already used by playbooks. Both settings directly change job behavior.

Compare inventory before and after

A successful sync does not prove that scope is acceptable. It only proves the source answered and AWX produced a new view. Compare hosts and groups before running an action.

INVENTORY_ID="17"

awx hosts list --inventory "$INVENTORY_ID" --all --format json | jq -r '.results[] | [.name, (.enabled|tostring), (.variables|tostring)] | @tsv' | sort > hosts-before.tsv

awx inventory_sources update "23" --monitor

awx hosts list --inventory "$INVENTORY_ID" --all --format json | jq -r '.results[] | [.name, (.enabled|tostring), (.variables|tostring)] | @tsv' | sort > hosts-after.tsv

diff -u hosts-before.tsv hosts-after.tsv || true

Read that diff as change evidence. An added host may be normal after scale-out. A removed host may point to a deleted Azure tag, a CMDB error or a credential that no longer sees the full scope.

Check the groups that feed the job

An AWX template rarely targets the whole inventory. It targets a group, pattern or launch-time limit. The useful check is therefore the groups actually used by the job, not only the total host count.

JOB_TEMPLATE_ID="42"

awx job_templates get "$JOB_TEMPLATE_ID" --format json | jq '{id,name,inventory,limit,ask_limit_on_launch,extra_vars,job_tags,skip_tags}'

awx groups list --inventory "17" --all --format json | jq -r '.results[] | [.name, .total_hosts] | @tsv' | sort

If production_web grows from 12 to 38 hosts, the topic is no longer only technical. The change alters the operational risk of the job. The decision may be to launch with a strict limit, fix the source or create a wider change window.

Control variables and identities

Dynamic inventory carries more than hostnames. It can inject ansible_host, ansible_user, role variables, application tags or connection settings. A changed variable can make a job fail or, worse, succeed through the wrong path.

Variables to check
ansible_host changes to an unexpected IP
ansible_user changes to a more privileged account
application groups change through tag or label drift
role variables are replaced by overwrite_vars
detected environment differs from group name
disabled host becomes enabled again without approval

Identities to check
AWX credential used by the source
SSH or WinRM credential used by the job
API rights of the external source
scope visible to the identity after the change

The validation criterion is not “the inventory syncs”. It is “the job will see the right targets with the right variables and credentials”.

Run a no-effect preflight

Before a job that changes state, run a read-only preflight on the same scope. It should confirm connectivity, host count, expected environment and critical variables.

---
- name: Validate dynamic inventory scope
hosts: "{{ target_group }}"
gather_facts: false
tasks:
  - name: Show resolved target and environment
    ansible.builtin.debug:
      msg: "{{ inventory_hostname }} env={{ env | default('missing') }} ansible_host={{ ansible_host | default('missing') }}"

  - name: Reject missing environment marker
    ansible.builtin.fail:
      msg: "Environment marker is missing"
    when: env is not defined

  - name: Reject production database hosts for this job
    ansible.builtin.fail:
      msg: "This job cannot target production databases"
    when: group_names is search('production_databases')

The preflight does not need to test everything. Its purpose is to make the scope resolved by AWX visible. If the preflight surprises the team, the production job should be paused.

Decide execution, limit or rollback

The decision must be explicit. Dynamic inventory may change for good reasons, but it should not silently widen an action.

Execute
Diff is expected and documented
Target group is stable or change is approved
Critical variables are unchanged
Read-only preflight is green
Source rollback is identified

Execute with a limit
New hosts are legitimate but not yet validated
Urgent maintenance on a known subset
Limit written in the ticket and AWX launch

Block
Production group widened without approval
Connection variables changed
Sensitive host enters scope
External source is inconsistent or credential is degraded

Rollback inventory
Wrong source filter
External tag or label removed by mistake
overwrite_vars replaced stable variables
Sync removes critical hosts

Rollback can mean returning to the previous source configuration, temporarily disabling update_on_launch, applying a manual limit or reverting the inventory project to the previous commit.

Validate after the job

After execution, link three pieces of evidence: inventory version, launched job and observed effect. Without that link, it becomes difficult to explain why a host was touched or missed.

JOB_ID="12345"

awx jobs get "$JOB_ID" --format json | jq '{id,name,status,inventory,limit,scm_revision,extra_vars,started,finished}'

awx job_events list --job "$JOB_ID" --format json | jq -r '.results[] | select(.host_name != null) | [.host_name, .event, (.event_data.task // "")] | @tsv' | sort -u > job-host-events.tsv

If an expected host was not touched, do not only fix the playbook. Returning to the inventory diff tells whether the problem came from AWX scope, the external source or the launch limit.

Conclusion

An AWX dynamic inventory is an operations interface. It decides which systems become reachable by a job, with which variables and credentials. Treating it as a simple host list creates quiet incidents: correct playbook, wrong scope.

The healthy decision is simple: compare before and after, validate the template groups, check critical variables, run a read-only preflight, then execute only when the scope is explainable. If inventory drifts, rolling back the source is often safer than improvising a correction inside the job.