Automation

Azure Durable Functions: diagnose an orchestration before replay or purge

A production runbook for qualifying a stuck Azure Durable Functions orchestration with instance history, activity state, storage, idempotency, KQL, validation and rollback before replay, terminate or purge.

26 Jul 2026 azuredurable-functionsazure-functionsorchestrationautomationobservabilityapplication-insightskqlstorageidempotencyrunbookrollbackproduction

A Durable Functions orchestration that stays Running, restarts from history, fails after an activity, or never reaches its expected output can look like a simple runtime issue. The fast answers are tempting: replay the event, terminate the instance, purge history, redeploy the Function App or start a new orchestration with the same input. In production, each shortcut can duplicate a side effect, hide a poison activity, erase evidence or leave an external system in an intermediate state.

The use case is a production workflow coordinating several steps: read a request, call internal APIs, fan out activities, update a database, send a notification, rotate a configuration value or start a controlled remediation. The runbook goal is to decide whether the instance should be allowed to continue, retried from a known boundary, terminated, compensated, purged after evidence capture, or blocked until the orchestrator contract is corrected.

Freeze the orchestration boundary

Start with one instance. Durable Functions can run many orchestrations and activities at the same time, and the same code path may be healthy for most inputs. The first useful action is to capture the instance boundary before changing state.

yaml durable-orchestration-scope.yml

incident:
function_app: func-prod-automation
task_hub: prodhub
orchestration: OrderReconciliationOrchestrator
instance_id: order-20260726-0842
runtime: production
symptom: running_without_progress_or_failed_after_activity
last_known_healthy_instance: order-20260726-0810

state_to_freeze:
orchestration_input_hash
created_time
last_updated_time
runtime_status
custom_status
failed_activity
external_event_name
external_correlation_id
side_effects_already_observed
replay_or_termination_owner
rollback_or_compensation_reference

If the team cannot name the instance and its expected side effects, do not replay globally. A broad replay can process new work, old work and broken work in the same movement.

Read status before changing status

Use the Durable instance state as the first source of truth, then correlate with Application Insights. The important question is not only whether the orchestration failed. It is where it stopped and whether the last activity may already have changed an external system.

bash 01-durable-instance-state.sh

RESOURCE_GROUP="rg-automation-prod"
FUNCTION_APP="func-prod-automation"
INSTANCE_ID="order-20260726-0842"

az functionapp function show --resource-group "$RESOURCE_GROUP" --name "$FUNCTION_APP" --function-name "OrderReconciliationOrchestrator" --output json

curl -sS "https://$FUNCTION_APP.azurewebsites.net/runtime/webhooks/durabletask/instances/$INSTANCE_ID?code=<system-key>" | jq '{instanceId, runtimeStatus, createdTime, lastUpdatedTime, customStatus, output}'

Keep credentials out of incident notes, but keep the returned timestamps, status, custom status and correlation IDs. If status endpoints are not available from the operator network, capture the equivalent evidence through the approved management path.

Rebuild the execution timeline in KQL

Durable Functions replays orchestrator code by design. That means duplicated-looking orchestrator logs are not always duplicate business actions. Activity logs, dependency calls and custom correlation IDs matter more than a single repeated message.

kusto 02-durable-orchestration-timeline.kql

let InstanceId = "order-20260726-0842";
union traces, exceptions, requests, dependencies
| where timestamp > ago(24h)
| where tostring(customDimensions["prop__instanceId"]) == InstanceId
 or tostring(customDimensions["InstanceId"]) == InstanceId
 or operation_Id == InstanceId
| project timestamp,
        itemType,
        operation_Id,
        name,
        message,
        success,
        resultCode,
        DurationMs = duration,
        FunctionName = tostring(customDimensions["prop__functionName"]),
        State = tostring(customDimensions["prop__state"]),
        Reason = tostring(customDimensions["prop__reason"])
| order by timestamp asc

Look for the last non-replay activity, the last dependency call and the last external effect. If all you see is orchestrator replay noise, add activity-level logging before deciding that the workflow is looping.

Separate orchestration, activity and dependency failure

A durable incident can sit in different layers. Treating every symptom as an orchestrator bug leads to unsafe purges and unnecessary redeployments.

text durable-failure-classes.txt

Orchestrator contract failure
Non-deterministic code changed inside the orchestrator
Date, random value or external call is executed in orchestrator code
History no longer matches the deployed orchestration logic
Decision: stop the rollout, restore compatible code or terminate with compensation

Activity failure
One activity returns an exception, timeout or invalid output
Retries may already be exhausted or still running
Decision: fix the dependency or input, then retry from a known boundary

External event wait
Instance is waiting for an approval, callback or event that never arrived
Decision: prove the event source before raising a replacement event

Storage or task hub failure
Durable history, queues, leases or tables cannot be read or updated reliably
Decision: fix storage and task hub health before replay or purge

Side-effect ambiguity
Activity timed out after calling an external system
Decision: query the external system before retrying the activity

This classification protects the team from the worst production mistake: terminating or purging the instance while the real problem is a dependency state that still needs compensation.

Verify idempotency before replay

Durable Functions gives structure to retries, but it does not make every activity idempotent. A replay can be safe for a read or a deterministic calculation. It can be dangerous for billing, notification, provisioning, deletion, ticket creation, secret rotation or data export.

yaml durable-idempotency-contract.yml

orchestration: OrderReconciliationOrchestrator
instance_id: order-20260726-0842
idempotency:
orchestration_key: order_batch_id
activity_keys:
  LoadOrders: batch_id
  ReserveProcessingWindow: batch_id + window
  UpdateBillingState: order_id + target_state
  NotifyOperations: incident_id + notification_type
safe_to_replay_when:
  - activity_has_no_external_side_effect
  - external_system_reports_no_operation_created
  - idempotency_key_already_maps_to_same_result
unsafe_to_replay_when:
  - timeout_after_external_write
  - missing_operation_id
  - notification_or_ticket_created_without_correlation
  - compensation_plan_unknown

If the activity cannot prove idempotency, the safer path is often manual reconciliation or compensation, not automatic replay.

Check storage and task hub health

The orchestration depends on the task hub storage account. Queue backlogs, poison messages, throttling, firewall changes, key rotation or task hub name drift can make healthy code look stuck.

text task-hub-health-checks.txt

Task hub checks
AzureWebJobsStorage points to the expected production storage account
Task hub name matches the deployed environment
Control and work-item queues are not blocked by poison messages
History and Instances tables are readable
Storage firewall, private path or identity changes are known
Host logs show the Functions runtime acquiring leases
No deployment is running against the same task hub with incompatible code

Block purge when
Storage health is unknown
The task hub may be shared by another environment
Instance history is the only evidence of side effects
A host restart may allow the instance to continue safely

Private networking can be part of the evidence when storage is restricted, but it is only one boundary. The point is to prove the task hub is healthy before changing orchestration state.

Decide continue, retry, terminate, compensate or purge

Make the operational decision explicit. Purge is rarely the first decision. It removes history after the team has already decided that the history is no longer needed for recovery or audit.

text durable-decision-table.txt

Let the instance continue
Runtime and storage recovered
Activity retries are still inside the expected policy
No duplicate side effect is possible
Business deadline still allows waiting

Retry from a known boundary
Failed activity is idempotent or external state proves no effect
Input and dependency issue are corrected
Correlation ID and expected output are known
Post-checks can distinguish old and new attempt

Terminate with compensation
Instance cannot progress safely
Partial side effect exists
Compensation or manual reconciliation is documented
New orchestration would otherwise duplicate work

Start a replacement instance
Original instance is blocked on missing input or obsolete code
Replacement input is bounded and linked to the original instance
Old instance is terminated or held with clear ownership

Purge only after evidence capture
Instance is completed, failed or terminated with no remaining recovery value
Timeline, inputs hash, side effects and decision are stored outside Durable history
Purge scope is limited to selected instance IDs or time window

A good decision tells operators what not to do as much as what to do. Do not purge to make dashboards clean. Do not restart the Function App to hide a non-deterministic orchestrator change.

Validate after action

After continue, retry, termination or replacement, validate the target state and the orchestration control plane together.

text post-durable-action-validation.txt

Target validation
External system contains one expected operation per idempotency key
No duplicate notification, ticket, billing update or configuration change exists
Business state matches the expected final state or documented compensation
Downstream consumers are not waiting on the old instance silently

Durable validation
Original instance status is known and recorded
Replacement instance, if any, links back to the original instance ID
Task hub queues and history are healthy
Application Insights timeline shows the final decision
Incident note stores KQL, instance status, side effects and rollback reference

If validation cannot prove those points, the incident should remain open. A green orchestration status is not enough when the workflow has already touched external systems.

Conclusion

A Durable Functions orchestration is both code and operational state. Replay, terminate and purge are production actions, not cleanup buttons. The safe path starts with the instance boundary, rebuilds the timeline, separates orchestrator, activity, dependency and storage causes, then proves idempotency before any replay.

The decision becomes defensible when the team can say: continue because retries are safe, retry because the boundary is known, terminate because compensation is ready, replace because the old instance is bounded, or purge because evidence has already been preserved. That is how Durable Functions remains a reliable automation engine instead of an opaque workflow that operators are afraid to touch.