Azure Automation: validate a runtime migration before cutting production runbooks over

An Azure Automation runbook still works in its established environment, but the team must move to another PowerShell version or update the Az package. A manual check passes, then the first production job fails because a module is missing, a type is serialized differently, or a command is unavailable on one Hybrid Runbook Worker. Reverting the script does not help: the code stayed the same while its execution environment changed.

The running example is a nightly reconcile-private-dns-records runbook. It reads an inventory, calculates a diff and updates only approved records. The team wants to move it to a current PowerShell runtime without using production as the compatibility test. This runbook builds an immutable candidate, exercises it with the same inputs and identities, limits the first real effect, then makes an explicit promote, hold or rollback decision.

Treat the runtime as a versioned dependency

An Azure Automation runtime environment groups the language, its version and the packages required by a runbook. Changing it can affect authentication, module resolution, JSON serialization, exception behavior and command availability. The migration contract therefore needs to say more than “PowerShell upgraded.”

runbook: reconcile-private-dns-records
current_runtime: ps-old-approved
candidate_runtime: ps-candidate-2026-09
execution_targets:
- azure_sandbox
- hybrid_worker_group: hwg-network-prod
packages_to_freeze:
- Az
- Az.Accounts
- Az.PrivateDns
identity: aa-network-prod-managed-identity
inputs_fixture: fixtures/private-dns-small.json
allowed_canary_scope: privatelink.example.internal
success_signal: expected_diff_applied_and_second_run_empty
rollback: relink_runbook_to_ps-old-approved

Create a separate candidate environment instead of editing the one already shared by production runbooks. The language version is immutable, but a package update on a shared environment reaches every linked runbook. A separate candidate makes the diff, canary and return path reviewable.

Inventory consumers before changing anything

List the runtime environments and affected runbooks before changing an association. The risk unit is not just the script; it includes every consumer of a shared environment.

SUBSCRIPTION="<subscription-id>"
RG="rg-automation-prod"
ACCOUNT="aa-network-prod"
API="2024-10-23"
BASE="https://management.azure.com/subscriptions/$SUBSCRIPTION/resourceGroups/$RG/providers/Microsoft.Automation/automationAccounts/$ACCOUNT"

az rest --method get --url "$BASE/runtimeEnvironments?api-version=$API" --query "value[].{name:name, runtime:properties.runtime, defaultPackages:properties.defaultPackages}" --output json

az rest --method get --url "$BASE/runbooks?api-version=$API" --query "value[].{name:name, state:properties.state, type:properties.runbookType, runtime:properties.runtimeEnvironment, modified:properties.lastModifiedTime}" --output table

Keep the exact previous environment name, package versions and linked-runbook list. If the candidate changes during qualification, give it a new manifest version or name. Evidence collected against a moving target is not promotion evidence.

Build a compatibility matrix that exercises the contract

Tests must cover the branches that depend on the runtime. A successful Get-Date proves neither identity nor runbook effects.

Surface                 Required evidence
Language                Exact PowerShell version and edition
Packages                Name, version and provisioning state
Imports                 Every required module loads without implicit fallback
Authentication          Expected principal, tenant and subscription
Azure reads             Inventory returns with the same scope
Serialization           Input fixture and normalized diff match
Error handling          Expected refusal keeps an actionable code and message
Hybrid Worker           Runtime binary exists on every target worker
Canary effect           One bounded target, followed by an empty second run
Observability           Job ID, runtime, worker, correlation and outcome retained

Include negative cases: an out-of-scope resource, a deliberately missing package in a test environment, an identity without write permission and invalid input. A candidate is safe when it performs allowed work and still refuses what the contract forbids.

Test the candidate without changing the published association

Azure Automation can start a draft test job with a different runtime environment before the published runbook is relinked. Use a no-effect fixture or a WhatIf mode that the runbook actually implements.

SUBSCRIPTION="<subscription-id>"
RG="rg-automation-prod"
ACCOUNT="aa-network-prod"
RUNBOOK="reconcile-private-dns-records"
CANDIDATE="ps-candidate-2026-09"
API="2024-10-23"
TEST_URL="https://management.azure.com/subscriptions/$SUBSCRIPTION/resourceGroups/$RG/providers/Microsoft.Automation/automationAccounts/$ACCOUNT/runbooks/$RUNBOOK/draft/testJob?api-version=$API"

az rest --method put --url "$TEST_URL" --headers "Content-Type=application/json" --body "{"properties":{"runtimeEnvironment":"$CANDIDATE","runOn":""}}"

The Test pane must not become a route around production controls. Use bounded inputs and identity, retain the streams, and never emit secrets. If the runbook cannot simulate writes, create an isolated canary target before testing the candidate.

Qualify every Hybrid Worker

A logical runtime association does not guarantee that every worker has the correct executable or matching local dependencies. PowerShell 7.4 on a Hybrid Worker requires the binary to be installed and its path declared on the machine. Check every group member before allowing job dispatch.

$expectedMajor = 7
$expectedMinor = 4
$runtimePath = [Environment]::GetEnvironmentVariable(
'powershell_7_4_path',
'Machine'
)

[pscustomobject]@{
ComputerName = $env:COMPUTERNAME
RuntimePath = $runtimePath
PathExists = Test-Path $runtimePath
CurrentVersion = $PSVersionTable.PSVersion.ToString()
Compatible = (
  $PSVersionTable.PSVersion.Major -eq $expectedMajor -and
  $PSVersionTable.PSVersion.Minor -eq $expectedMinor
)
} | ConvertTo-Json -Compress

Get-Module -ListAvailable Az.Accounts, Az.PrivateDns |
Select-Object Name, Version, Path |
ConvertTo-Json -Compress

Run this as a no-effect job on every intended worker. Remove a divergent worker from the group or remediate it before the canary. Do not rely on the scheduler to happen to select the correct machine.

Compare normalized outputs

Two runtimes may produce technically different objects that represent the same state. Compare a normalized result: identifiers, sorted business properties, proposed diff and error codes. Ignore job timestamps and non-contractual property order.

{
"fixture": "private-dns-small-v3",
"oldRuntime": {
  "readCount": 12,
  "proposedChanges": 1,
  "deniedOutOfScope": true
},
"candidateRuntime": {
  "readCount": 12,
  "proposedChanges": 1,
  "deniedOutOfScope": true
},
"normalizedDiff": [],
"decision": "eligible_for_bounded_canary"
}

Explain every mismatch before promotion. A missing property can expose a module change; a new result order can break non-deterministic logic; a changed error message can disable an alert or parser.

Relink one runbook and bound the first effect

Once no-effect tests pass, relink only the canary runbook to the candidate. A running job is not changed by a new association, so validation applies to jobs started after cutover.

SUBSCRIPTION="<subscription-id>"
RG="rg-automation-prod"
ACCOUNT="aa-network-prod"
RUNBOOK="reconcile-private-dns-records"
CANDIDATE="ps-candidate-2026-09"
API="2024-10-23"
RUNBOOK_URL="https://management.azure.com/subscriptions/$SUBSCRIPTION/resourceGroups/$RG/providers/Microsoft.Automation/automationAccounts/$ACCOUNT/runbooks/$RUNBOOK?api-version=$API"

az rest --method patch --url "$RUNBOOK_URL" --headers "Content-Type=application/json" --body "{"properties":{"type":"PowerShell","runtimeEnvironment":"$CANDIDATE"}}"

The first real job needs a reduced scope, an idempotency key and business validation. In the DNS example, limit it to one controlled zone and record. Run it a second time: an empty diff is stronger evidence than Completed alone.

Record the runtime that actually executed

Add a proof envelope at the beginning of the runbook without logging tokens or secrets. It must connect the migration decision to the observed job.

$evidence = [ordered]@{
correlationId = [guid]::NewGuid().ToString()
runbook = 'reconcile-private-dns-records'
runtime = $PSVersionTable.PSVersion.ToString()
edition = $PSVersionTable.PSEdition
worker = $env:COMPUTERNAME
azAccounts = (Get-Module -ListAvailable Az.Accounts |
  Sort-Object Version -Descending |
  Select-Object -First 1 -ExpandProperty Version).ToString()
startedAtUtc = [DateTime]::UtcNow.ToString('o')
}

Write-Output ($evidence | ConvertTo-Json -Compress)

Correlate this envelope with the Job ID, the identity recorded in Azure Activity and the business signal. A green job under an unknown runtime or on an unidentified worker does not close the migration.

Decide promotion, hold or rollback

Promote progressively
Candidate runtime and packages are frozen
Positive and negative tests match the contract
Every target Hybrid Worker is qualified
Canary applies only the expected diff
Second run has no effect
Technical traces and business validation agree

Hold the current runtime
Candidate works but one package remains uncontrolled
Outputs differ without an explanation
One worker in the group is not qualified
Tests miss a branch with an external effect

Rollback immediately
Identity, scope or expected refusals changed
Canary touches a target outside the contract
A worker lacks a local dependency
Traces cannot attribute the effect
Second run repeats a write

Rollback means relinking the runbook to the previous environment, then running a read-only check and handling any already-applied effect separately. Do not immediately delete the candidate: retain its manifest and outputs to explain the failure. Deleting a runtime environment is not a rollback mechanism.

Conclusion

An Azure Automation runtime migration is a production release even when the script is unchanged. The useful scope is the combination of code, environment and execution target, completed by identity and a business-level success signal.

The decision is then defensible: promote when the candidate is frozen, tested for refusals as well as success, qualified on every worker and validated by an idempotent canary; hold when evidence is missing; relink the previous environment immediately when scope, identity or effects drift. The runtime becomes an operable dependency instead of hidden context discovered during an incident.