Azure Automation: prove the published version before running a runbook

An Azure Automation runbook has been fixed in Git, reviewed and merged. Its next job will change a production allowlist. Yet nobody can prove that the version published in the Automation account matches the approved commit. The sync may have failed, the new code may still be Draft, or a portal hotfix may have created drift that the repository cannot show.

The running case is a PowerShell runbook named reconcile-firewall-allowlist. It calculates a diff and applies only approved entries. This is not another code review. The operational goal is to bind an immutable commit to the Azure artifact, complete a no-effect test, publish one candidate, and decide whether to run, resync, block or roll back.

Freeze the release contract

Name the expected version before starting another sync. A branch name is not enough because it moves. Preserve the commit, exact script path, hash, target Automation account, runbook name, runtime and intended sync behavior.

runbook: reconcile-firewall-allowlist
automation_account: aa-platform-prod
resource_group: rg-automation-prod
source_control: platform-runbooks
branch: main
source_path: /Runbooks/reconcile-firewall-allowlist.ps1
approved_commit: 7c91e1f
approved_sha256: <sha256-of-reviewed-script>
runtime: PowerShell-5.1
sync_should_publish: false
execution_mode: Plan
target_scope: /subscriptions/.../resourceGroups/rg-network-prod
success:
- sync job completes for the approved commit
- draft hash equals the reviewed script hash
- no-effect test returns the expected bounded diff
- published hash is captured after promotion
rollback: import and publish the last approved artifact

Azure Automation keeps separate Draft and Published versions. The test pane exercises the Draft; normal jobs execute the Published version. A successful synchronization therefore does not, by itself, prove what production will run.

Qualify the synchronization boundary

Source control integration is a one-way flow from the repository into Azure Automation. Verify the configured branch and folder, auto-sync state and automatic publication setting. A file outside the configured folder, a different production branch or an expired webhook can leave Azure on an older version even though the Git merge is correct.

Do not assume the integration covers every runtime. Native source control sync supports PowerShell 5.1 runbooks. For newer PowerShell runtimes, use an explicit import-and-publish pipeline and keep the same hash, test and rollback evidence.

Do not combine recovery of the sync channel, credential rotation and runbook promotion into one incident change. Repair synchronization first; promote code as a separate, observable decision.

Read the sync job and streams before retrying

A Completed state must belong to the right source control, commit and file set. List recent jobs, inspect the candidate job and preserve its streams. Starting another sync before reading the previous one can erase the most useful diagnostic context.

RG="rg-automation-prod"
ACCOUNT="aa-platform-prod"
SOURCE="platform-runbooks"

az automation source-control sync-job list --resource-group "$RG" --automation-account-name "$ACCOUNT" --source-control-name "$SOURCE" --output table

JOB_ID="<sync-job-id>"

az automation source-control sync-job show --resource-group "$RG" --automation-account-name "$ACCOUNT" --source-control-name "$SOURCE" --job-id "$JOB_ID" --output json

az automation source-control sync-job stream list --resource-group "$RG" --automation-account-name "$ACCOUNT" --source-control-name "$SOURCE" --sync-job-id "$JOB_ID" --output json

Look for authentication failures, skipped paths, unsupported runbook types, partial imports or missing publication. If the job does not identify the expected commit, stop comparing application behavior: the release chain has already selected the wrong artifact.

Compare repository, Draft and Published

Export both slots separately. Hash them without editing the files, then compare each to the script from the approved commit. All three states can legitimately differ during diagnosis: correct repository, correct Draft, stale Published.

$ResourceGroup = "rg-automation-prod"
$Account = "aa-platform-prod"
$Runbook = "reconcile-firewall-allowlist"
$Output = Join-Path $PWD "runbook-evidence"

New-Item -ItemType Directory -Path $Output -Force | Out-Null

Export-AzAutomationRunbook `
-ResourceGroupName $ResourceGroup `
-AutomationAccountName $Account `
-Name $Runbook `
-Slot Draft `
-OutputFolder (Join-Path $Output "draft") `
-Force

Export-AzAutomationRunbook `
-ResourceGroupName $ResourceGroup `
-AutomationAccountName $Account `
-Name $Runbook `
-Slot Published `
-OutputFolder (Join-Path $Output "published") `
-Force

Get-ChildItem $Output -Recurse -File |
Get-FileHash -Algorithm SHA256 |
Select-Object Path, Hash

Comments or encoding can change a hash without changing runtime behavior. Keep the hash as a drift signal and add a readable diff when that happens. Do not silently normalize code before comparison, because normalization can also hide an executable difference.

Emit a non-secret release identifier at the start of every job. It connects execution streams to the intended source version without relying on an operator’s memory.

$Release = "git:7c91e1f"
Write-Output "release=$Release runbook=reconcile-firewall-allowlist mode=$Mode"

if ($Mode -eq "Plan") {
Write-Output "No production write is allowed in Plan mode"
}

Test the Draft without side effects

The test should use production-shaped inputs and the same logical identity, but a mode that prevents every write. For reconcile-firewall-allowlist, the runbook reads effective configuration, calculates additions and removals, then stops before the mutation call.

Success is more than Completed. The output must include the release marker, target scope, number of objects read, calculated diff and proof that no write was attempted. Check the Activity Log over the target scope as an independent control: a no-effect test is valid only when the control plane shows no matching mutation.

If the runbook has no Plan mode, do not improvise one during the incident. Use an isolated canary target or block publication until the execution contract is fixed.

Publish and validate a bounded canary

When the Draft matches the approved commit and its no-effect test is clean, publish that runbook explicitly. Export the Published slot again and calculate its hash. A successful publish operation in the portal or API is not a substitute for artifact evidence.

Publish-AzAutomationRunbook `
-ResourceGroupName "rg-automation-prod" `
-AutomationAccountName "aa-platform-prod" `
-Name "reconcile-firewall-allowlist"

Export-AzAutomationRunbook `
-ResourceGroupName "rg-automation-prod" `
-AutomationAccountName "aa-platform-prod" `
-Name "reconcile-firewall-allowlist" `
-Slot Published `
-OutputFolder "$PWD/published-after" `
-Force

Get-FileHash "$PWD/published-after/reconcile-firewall-allowlist.ps1" `
-Algorithm SHA256

Run the published version in Plan first, then allow one reversible canary entry. Correlate job ID, release marker, identity, target, applied diff and Activity Log. Expand scope only after both artifact identity and effect are proven.

Decide run, resync or rollback

Run
approved commit, Draft and Published are tied together
Plan test and published canary are clean
release marker and Activity Log agree

Resync
job is missing, failed or tied to another commit
connection, branch and folder are confirmed before retry
production jobs cannot start while the channel is repaired

Publish
Draft equals the approved commit
Published is still stale
no-effect Draft test has passed

Rollback
Published is wrong or the canary regresses
import and publish the previous known artifact
rerun Plan and canary before schedules or webhooks resume

Block
hash or commit cannot be established
test side effects are not bounded
runtime is unsupported by the selected sync mechanism

Rollback to a known artifact, not whatever main contains at incident time. Import the last approved version, publish it, export it again and confirm its release marker. Temporarily suspend schedules and webhooks when an automatic job could start between import and validation.

Conclusion

In Azure Automation, “the code was merged” and “this code will execute” are different claims. An operable release chain connects commit, sync job, Draft, no-effect test, Published artifact, canary and execution trace.

The final decision is then defensible: run the proven version, resync the intended commit, publish a validated Draft, restore the previous artifact, or keep production blocked while version identity remains ambiguous.