Azure Functions: diagnose a poison queue before replaying messages

An Azure Function fed by Storage Queue still processes most traffic, but its poison queue starts growing after a deployment. Moving the messages straight back to the source queue looks like a quick recovery. It can also repeat a write that already succeeded, hide a schema defect, or loop a message that the new release will never parse.

The use case is a Queue trigger driving a business operation such as document generation, order synchronization, or provisioning. The runbook must attribute failures to a message, a release, and a processing stage, then choose among a fix, quarantine, bounded replay, or worker rollback.

Freeze the contract and incident window

Start with one Function and one queue. Capture the bundle that is actually deployed, the trigger configuration, the first failure window, and the expected side effect.

incident:
start: <timestamp>
first_poison_message: <timestamp>
function_app: func-orders-prod
function: ProcessOrder

queue_contract:
source: orders
poison: orders-poison
schema_version: <version>
business_key: orderId
expected_side_effect: create-or-update-order

active_bundle:
application_version: <immutable-version>
host_configuration: <commit-or-artifact-hash>
extension_bundle: <version-range>
app_settings_snapshot: <redacted-reference>

evidence:
- poison queue count and safe sample
- invocation, exception and dependency logs
- deployment and configuration timeline
- downstream state for sampled business keys
- previous deployable bundle

Do not copy application secrets into the incident record. Preserve useful setting names, versions, and fingerprints, then reference the vault or configuration mechanism.

Sample without consuming the evidence

A peek reads a sample without making messages invisible or changing their dequeue count. Start there; do not use a destructive retrieval command while qualifying the incident.

STORAGE_ACCOUNT="stprodorders"
POISON_QUEUE="orders-poison"

az storage message peek --account-name "$STORAGE_ACCOUNT" --queue-name "$POISON_QUEUE" --num-messages 16 --auth-mode login --output json

az monitor activity-log list --resource-group "rg-prod-orders" --offset 24h --query "[?contains(resourceId, 'func-orders-prod')].{time:eventTimestamp,operation:operationName.value,caller:caller,status:status.value}" --output table

For each sample, record message ID, insertion time, business key, schema version, payload size, and payload fingerprint. If the content is sensitive, work from a redacted copy. The goal is to group messages by failure signature, not to create a business-data export.

Separate five failure classes

A poison queue commonly mixes several causes. Classify before changing anything:

• deterministic contract failure: malformed JSON, missing required field, or unknown schema version;
• transient dependency failure: timeout, throttling, or outage that outlasted the retry window;
• identity or network failure: Storage, Key Vault, API, or database access denied to the actual runtime identity;
• resource failure: Function timeout, memory pressure, scaling behavior, or processing duration incompatible with message visibility;
• partial side effect: the dependency accepted the write, then the invocation failed before recording success.

The last class is the dangerous one. A Function exception does not prove that the downstream operation failed. Query business state or the idempotency ledger before replaying anything.

Correlate message, invocation, and dependency

The following query assumes the application emits messageId, businessKey, schemaVersion, and appVersion as custom properties. Adapt table names to your workspace and always bound the time window.

let StartTime = datetime(2026-08-05T08:00:00Z);
let EndTime = datetime(2026-08-05T10:00:00Z);
AppTraces
| where TimeGenerated between (StartTime .. EndTime)
| where AppRoleName == "func-orders-prod"
| extend MessageId = tostring(Properties.messageId),
       BusinessKey = tostring(Properties.businessKey),
       SchemaVersion = tostring(Properties.schemaVersion),
       AppVersion = tostring(Properties.appVersion),
       FailureClass = tostring(Properties.failureClass)
| where isnotempty(MessageId)
| summarize Attempts=count(),
          FirstSeen=min(TimeGenerated),
          LastSeen=max(TimeGenerated),
          Messages=make_set(MessageId, 20)
by AppVersion, SchemaVersion, FailureClass, BusinessKey
| order by Attempts desc

Then correlate the invocation with AppExceptions and AppDependencies. The same exception on one schema version suggests a deterministic defect. 429 responses or timeouts isolated to one dependency indicate a transient incident. A successful dependency call followed by a local exception requires an idempotency check.

Review retries and concurrency as one system

Maximum attempts, message visibility, batching, and concurrency form one operating contract. More retries do not fix an unreadable payload. Higher concurrency can accelerate downstream saturation. Visibility that is shorter than processing can make a message available while its first invocation is still running.

{
"extensions": {
  "queues": {
    "maxDequeueCount": "<derived-attempt-budget>",
    "visibilityTimeout": "<greater-than-tested-processing-window>",
    "batchSize": "<bounded-for-downstream-capacity>",
    "newBatchThreshold": "<tested-concurrency-threshold>"
  }
}
}

This is a decision model, not a configuration to paste. Verify the options supported by the extension version in production. Derive the attempt budget from acceptable recovery time, downstream capacity, and which failures are genuinely retryable.

Fix the defect before designing replay

For a contract failure, make the consumer compatible with supported versions or explicitly archive obsolete messages. For a transient failure, prove that the dependency has recovered. For identity failures, test with the Function identity rather than an operator account. For partial side effects, add or use a stable idempotency key.

{
"replay": {
  "incidentId": "func-poison-20260805-01",
  "originalMessageId": "<message-id>",
  "businessKey": "<stable-business-key>",
  "sourceSchemaVersion": "<version>",
  "targetConsumerVersion": "<version>",
  "replayAttempt": 1
},
"payload": "<validated-original-payload>"
}

Do not silently rewrite a historical payload. If transformation is required, version it, keep the original fingerprint, and produce a dry-run report showing every inclusion, exclusion, and reason.

Replay through a canary with a stop condition

Replay must use a bounded tool or canary queue, never a loop that drains the entire poison queue. Start with messages that have no irreversible effect and one representative from each failure signature.

replay:
incident_id: func-poison-20260805-01
include:
  failure_class: transient_dependency
  schema_versions: [v3]
  inserted_at: [<start>, <end>]
exclude:
  unknown_schema: true
  business_keys_already_completed: true
  irreversible_side_effect_unverified: true
canary:
  messages: 5
  pause_after_batch: 10m
acceptance:
  - every replay has a new invocation correlation
  - business state changes at most once
  - no replayed message returns to poison
  - latency and dependency errors stay inside baseline
stop:
  - duplicate side effect detected
  - unknown failure signature appears
  - poison queue resumes growing
  - correlation with downstream state is lost

A lower poison queue count is not a success criterion. Reconcile selected messages, completed invocations, and the business states actually produced.

Decide, validate, and roll back

Keep the fix when the canary processes every expected signature exactly once, new messages no longer reach poison, and dependencies remain stable at normal throughput. Then expand in small batches with the same exclusions.

Roll back the worker when poison messages began with the new release and the previous bundle still handles the current contract. Restore code, host.json, and compatible settings together; a partial rollback can retain the wrong concurrency or retry budget. Do not replay until the rollback has processed one synthetic message and one redacted sample.

Keep messages quarantined when the schema is unknown, the business effect is ambiguous, or idempotency is missing. Rolling back a Function does not undo a write already accepted by a dependency.

Conclusion

A poison queue is more than a backlog. It collects incompatible contracts, unstable dependencies, and partial executions that the nominal path could not resolve.

The production decision must remain reviewable: classify each signature, verify downstream state, fix the cause, replay a bounded canary, then keep, expand, or roll back. The queue becomes a recovery mechanism again rather than an uncontrolled second write path.