Automation

Azure Automation: diagnose a runbook before rerunning the job

A production runbook for qualifying an Azure Automation failure with managed identity, parameters, modules, Hybrid Worker, webhooks, logs, validation and rollback before rerun.

03 Jul 2026 azureazure-automationrunbookautomationmanaged-identityhybrid-workermoduleswebhookobservabilitykqlrollbackproduction

A failed Azure Automation runbook often creates pressure to click Rerun too quickly. The job may have timed out, a module may have changed, a managed identity may have lost a role, a Hybrid Worker may no longer have the same connectivity, an Automation variable may contain an old value, or the webhook may have received an incomplete payload. Rerunning without qualifying state can replay a partial action: secret rotation, resource cleanup, configuration switch, RBAC correction or maintenance task.

The use case is a PowerShell or Python runbook used by operations to apply a bounded action in Azure. It runs either in Azure Automation or on a Hybrid Runbook Worker to reach a private network. The diagnostic goal is to decide whether the job can be rerun as-is, rerun with corrected input, rolled back, or kept blocked until the automation contract is fixed.

Freeze the job contract

Start by describing what the job was supposed to do, not only the error message. A runbook is a production interface: parameters, identity, target, preconditions, side effects, idempotence and validation.

text automation-job-scope.txt

Incident
Automation Account: aa-prod-ops
Runbook: rotate-expiring-app-secret
JobId: 7f7f3c1a-0000-0000-0000-000000000000
Mode: Azure sandbox or Hybrid Worker
Trigger: manual, schedule, webhook or pipeline
Target: application, resource group, subscription, tenant
Expected effect: prepare or apply the rotation
Last certain step before failure
Expected validation after execution
Known rollback or point of no return

Questions before rerun
Is the job idempotent?
Was a partial action already applied?
Do parameters come from a webhook, variable or manual call?
Is the runtime identity the expected one?
Is Hybrid Worker connectivity required?

If the team cannot answer those questions, rerun is already a production change, not a simple technical retry.

Read the real job state

Capture the job state, received parameters, streams and execution context. Do not rely only on the status shown in the portal.

bash 01-automation-job-state.sh

RG="rg-automation-prod"
ACCOUNT="aa-prod-ops"
JOB_ID="7f7f3c1a-0000-0000-0000-000000000000"

az automation job show --resource-group "$RG" --automation-account-name "$ACCOUNT" --name "$JOB_ID" --output json

az automation job output --resource-group "$RG" --automation-account-name "$ACCOUNT" --job-id "$JOB_ID" --stream Any --output table

Keep the start time, end time, published runbook, parameters and worker used. These fields make a rerun comparable with the failed execution.

Separate parameters, variables and secrets

A runbook often fails because its input drifted. The code did not change, but the Automation variable, webhook, schedule or referenced secret no longer matches the expected contract.

text automation-input-checklist.txt

Check inputs
Parameters supplied to the job
Defaults in the published runbook
Automation variables read during execution
Automation credentials or certificates still in use
Key Vault secrets referenced by the runbook
Webhook payload and expected schema
Trigger time and schedule timezone

Block rerun when
A target parameter is empty or too broad
The webhook payload has no change identifier
A global variable changed during the incident
The runbook can touch several environments with the same input
Rerun would replay a completed step

The right fix may be a rerun with bounded input, not a script change. That decision is safe only when the target and partial state are proven.

Prove the runtime identity

Azure Automation may use a system-assigned managed identity, a user-assigned managed identity or an older Run As account. On a Hybrid Worker, the script may also depend on local context, installed modules or specific network access. The rerun must prove which identity will act.

bash 02-automation-identity-scope.sh

RG="rg-automation-prod"
ACCOUNT="aa-prod-ops"

az automation account show --resource-group "$RG" --name "$ACCOUNT" --query "{identity:identity, publicNetworkAccess:publicNetworkAccess}" --output json

PRINCIPAL_ID="00000000-0000-0000-0000-000000000000"
az role assignment list --assignee "$PRINCIPAL_ID" --all --query "[].{scope:scope, role:roleDefinitionName}" --output table

An AuthorizationFailed error should not immediately lead to a wider role. First verify that the runbook uses the right identity, tenant, subscription and target.

Check modules and published version

The published runbook version and imported modules are a change surface. An update to Az.Accounts, Az.Resources or an internal module can change authentication, serialization or error handling without changing the runbook itself.

bash 03-automation-runtime-version.sh

RG="rg-automation-prod"
ACCOUNT="aa-prod-ops"
RUNBOOK="rotate-expiring-app-secret"

az automation runbook show --resource-group "$RG" --automation-account-name "$ACCOUNT" --name "$RUNBOOK" --query "{state:state, runbookType:runbookType, lastModifiedTime:lastModifiedTime, logProgress:logProgress, logVerbose:logVerbose}"

az automation module list --resource-group "$RG" --automation-account-name "$ACCOUNT" --query "[].{name:name,version:version,provisioningState:provisioningState,lastModifiedTime:lastModifiedTime}" --output table

If a module changed just before the incident, the rerun should go through a controlled test or module rollback. Replaying the job in the same unstable runtime does not validate anything.

Qualify the Hybrid Worker before blaming the script

When a runbook depends on a Hybrid Worker, the failure may come from the execution host: stopped service, degraded extension, private connectivity, DNS, proxy, local permissions or missing module. Treat the worker as a production component.

text hybrid-worker-checks.txt

Check the worker
Expected Hybrid Worker group
Machine that picked up the job
Agent service running
Connectivity to Azure Automation
DNS and routing to private target resources
PowerShell or Python version available
Expected local modules
Access to Key Vault, Storage, internal API or private endpoint

Rerun is forbidden when
The job can land on an unqualified worker
Local DNS does not resolve the private target
The worker lost controlled outbound access
Local modules differ inside the same worker group
Logs do not say which worker executed the job

If the worker is the cause, rollback may mean temporarily removing it from the group, fixing connectivity or forcing execution on a healthy group, not changing the runbook.

Correlate job, Azure activity and application logs

Useful evidence links the job to its effects. Look for Azure writes, authorization errors, dependency calls and application traces around the execution window.

kusto 04-automation-job-effects.kql

let JobStart = datetime(2026-07-03T06:10:00Z);
let JobEnd = datetime(2026-07-03T06:24:00Z);
AzureActivity
| where TimeGenerated between ((JobStart - 10m) .. (JobEnd + 20m))
| where Caller has_any ("aa-prod-ops", "00000000-0000-0000-0000-000000000000")
 or CorrelationId == "<job-correlation-id>"
| project TimeGenerated,
        OperationNameValue,
        ActivityStatusValue,
        Caller,
        ResourceGroup,
        ResourceId,
        CorrelationId,
        Properties
| order by TimeGenerated asc

If the job already changed a resource, rerun must be treated as state recovery, not as a brand-new full execution.

Decide rerun, resume or rollback

The decision should be explicit. A failed job may be rerun, resumed from a step, corrected at input level, rolled back or frozen.

text automation-rerun-decision.txt

Rerun as-is
The job is idempotent
No partial action was applied
Identity, modules and worker are healthy
The failure comes from a proven timeout or temporary dependency issue

Rerun with corrected input
Payload was incomplete or too broad
Exact target is now proven
Rerun will not replay a completed step
Change is attached to a validated incident or request

Resume manually or by step
A partial action is complete
The runbook has no reliable checkpoint
Next action depends on observed state
Human validation is required before continuing

Rollback or block
Runtime identity is wrong
A module or worker drifted
The job touched an unexpected target
Logs do not prove applied state
Rerun could duplicate a side effect

The central criterion is real idempotence. If the runbook cannot detect what it has already done, rerun should become controlled recovery.

Automate the diagnostic pack

A good operating mode is not a magic rerun button. It is a collection step that produces the decision: rerun, resume, rollback or block.

yaml automation-diagnostic-pack.yml

diagnostic_pack:
collect:
  - job_state_and_streams
  - input_parameters_and_webhook_payload
  - runbook_published_version
  - automation_variables_used
  - module_versions
  - runtime_identity_and_role_scope
  - hybrid_worker_group_and_machine
  - azure_activity_changes
  - validation_signal_after_each_step
decide:
  - rerun_same_inputs
  - rerun_with_corrected_inputs
  - resume_from_verified_state
  - rollback_previous_step
  - block_until_contract_fix
require_human_validation:
  - production_write
  - secret_rotation
  - rbac_change
  - network_or_firewall_change
  - non_idempotent_rerun

This automation stays safe because it does not hide state. It gives the operator enough evidence to act without improvising.

Validate after action

After rerun or rollback, validate the business effect and Azure state. A Completed job status is not enough.

text automation-post-action-validation.txt

Minimum validation
Job completes with the expected runbook and modules
Runtime identity matches the authorized principal
Target resources are exactly within scope
AzureActivity logs confirm expected operations
Secrets, roles, routes or configurations reached the expected final state
Application checks or probes recover
Incident record includes input, decision, action and evidence

Rollback is incomplete when
The job is green but target state did not change
A partial action remains ownerless
The healthy worker is not identified
Modules remain inconsistent
The next rerun can reproduce the same incident

Conclusion

Rerunning an Azure Automation runbook is an operations decision. It should prove the job contract, partial state, parameters, identity, modules, worker, logs and validation path.

The right outcome is not always Rerun. It may be a bounded rerun, manual recovery, configuration rollback, module correction or deliberate block until the runbook becomes operable again. That discipline keeps automation useful without turning a one-time failure into a replayed incident.