Automation

Azure Event Grid: diagnose dead-lettered events before replay

A production runbook for qualifying Azure Event Grid delivery failures with subscription filters, endpoint health, dead-letter storage, diagnostics, replay scope, validation and rollback before reprocessing events.

05 Jul 2026 azureevent-griddead-letterevent-drivenautomationobservabilitykqlstoragerunbookrollbackproduction

Dead-lettered Event Grid events are not a queue to drain blindly. They are evidence that delivery failed, filters changed, the subscriber rejected requests, authentication broke, the endpoint became unreachable or the payload no longer matched the consumer contract. Replaying them too quickly can duplicate actions, reopen an incident or push stale data into a system that has already recovered.

The use case is an Azure platform where Event Grid routes production events to Functions, Logic Apps, webhooks, internal APIs or automation workers. A dead-letter container starts filling after a deployment or a dependency outage. The runbook goal is to decide whether to replay, discard, hold or roll back the subscription change, with enough proof to avoid processing the wrong events twice.

Freeze the subscription and the consumer

Start with one event subscription. Do not diagnose every topic and every consumer at once. The first job is to capture the delivery contract: event source, filters, endpoint, retry behavior and dead-letter destination.

text event-grid-delivery-contract.txt

Event Grid delivery contract
Topic or system topic: eg-prod-orders
Event subscription: sub-order-automation-prod
Event types: Microsoft.Storage.BlobCreated, OrderAccepted
Subject filter: /orders/prod/
Endpoint: orders-worker.internal/api/events
Endpoint type: webhook, Function, Logic App or internal API
Authentication: managed identity, key, webhook validation or APIM policy
Dead-letter destination: storage account, container and path
Recent change: deployment, filter update, endpoint release, auth rotation or network rule

Evidence required before replay
Event subscription configuration
Delivery failure reason and time window
Dead-letter blob sample and count
Endpoint logs for matching event IDs
Consumer idempotency proof
Replay scope and exclusion rules
Rollback or hold decision

If the team cannot name the consumer side effect, it should not replay yet. A notification event, a billing event and a provisioning event do not have the same blast radius.

Check whether the subscription changed

A dead-letter spike after a deployment often comes from a small subscription drift: a narrower subject filter, a new advanced filter, an endpoint URL change, an expired delivery identity or a dead-letter destination that was moved without operators noticing.

bash 01-event-subscription-snapshot.sh

RESOURCE_GROUP="rg-prod-events"
TOPIC_NAME="eg-prod-orders"
SUBSCRIPTION_NAME="sub-order-automation-prod"

az eventgrid event-subscription show --source-resource-id "$(az eventgrid topic show   --resource-group "$RESOURCE_GROUP"   --name "$TOPIC_NAME"   --query id -o tsv)" --name "$SUBSCRIPTION_NAME" --output json

az monitor activity-log list --resource-group "$RESOURCE_GROUP" --offset 24h --query "[?contains(operationName.value, 'eventSubscriptions')].{time:eventTimestamp, operation:operationName.value, caller:caller, status:status.value}" --output table

Treat filters as production code. A replay is not useful if the subscription still rejects the same shape of events or points to the wrong endpoint.

Read dead-letter blobs as evidence

Dead-letter storage should answer three questions: what failed, when it failed, and whether the event is still safe to process. Sample before acting.

bash 02-dead-letter-sample.sh

STORAGE_ACCOUNT="stprodeventdeadletter"
CONTAINER="eventgrid-deadletter"
PREFIX="sub-order-automation-prod/"

az storage blob list --account-name "$STORAGE_ACCOUNT" --container-name "$CONTAINER" --prefix "$PREFIX" --auth-mode login --query "[0:20].{name:name,lastModified:properties.lastModified,size:properties.contentLength}" --output table

# Download only a small sample first.
az storage blob download-batch --account-name "$STORAGE_ACCOUNT" --destination ./deadletter-sample --source "$CONTAINER" --pattern "${PREFIX}*" --auth-mode login

For each sample, keep the event ID, event type, subject, event time, dead-letter reason if present, delivery attempt metadata and any correlation ID carried in the payload. Do not assume every blob belongs to the same failure mode.

Correlate delivery failures and endpoint logs

The useful question is not “is Event Grid broken?”. It is “which control rejected delivery?”. Separate endpoint errors, authentication failures, network reachability, throttling and consumer-side validation.

kusto 03-event-grid-delivery-evidence.kql

let StartTime = datetime(2026-07-05T06:00:00Z);
let EndTime = datetime(2026-07-05T07:00:00Z);
let SubscriptionName = "sub-order-automation-prod";
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where ResourceProvider has "MICROSOFT.EVENTGRID"
| where tostring(eventSubscriptionName_s) == SubscriptionName
| project TimeGenerated,
        eventSubscriptionName_s,
        eventType_s,
        subject_s,
        deliveryStatus_s,
        deliveryResponseCode_s,
        deliveryResponseMessage_s,
        deadLetterReason_s,
        endpointUrl_s
| order by TimeGenerated asc

Then correlate with the consumer logs using event ID, request ID, operation ID or payload correlation. If Event Grid reports 401 or 403, replaying before fixing identity only creates more dead letters. If the endpoint returns 400, the consumer contract may have changed. If no request reaches the consumer, inspect DNS, firewall, APIM, Function networking or private endpoint dependencies according to the endpoint architecture.

Prove idempotency before replay

Replay is safe only when the consumer can tolerate duplicates or when the replay scope excludes events already processed. This is where many incident fixes become data incidents.

text idempotency-checklist.txt

Replay safety checks
Consumer stores processed event IDs or business keys
Side effects are idempotent or compensating action is documented
Event time is still inside the business validity window
Downstream dependency can accept historical events
Partial processing can be detected from logs or state
Replay batch can be limited by event type, subject and time window
Operators know how to stop replay quickly

Block replay when
Events trigger irreversible external actions
Consumer has no duplicate detection
The payload schema changed and old events are not compatible
The target state has already been rebuilt by another process
Dead-letter samples mix multiple failure causes

For event-driven automation, idempotency is not a nice-to-have. It is the condition that makes replay an operational tool instead of a second incident.

Build a bounded replay plan

Replay should be a controlled batch, not a storage-container loop. Define the inclusion rule, the maximum batch size, the target endpoint, the expected logs and the stop condition.

yaml event-grid-replay-plan.yml

replay:
incident_id: eg-deadletter-20260705-01
subscription: sub-order-automation-prod
source_container: eventgrid-deadletter
include:
  event_types:
    - OrderAccepted
  subject_prefix: /orders/prod/
  event_time:
    from: 2026-07-05T06:00:00Z
    to: 2026-07-05T06:30:00Z
exclude:
  already_processed_event_ids: true
  failed_schema_version: v1-preview
batch:
  max_events: 100
  pause_between_batches: 5m
validation:
  - endpoint returns expected 2xx
  - consumer logs processed event IDs
  - downstream state changes match event count
  - no new dead-letter spike appears
stop:
  - duplicate side effect detected
  - endpoint errors exceed threshold
  - unknown event type appears
  - operator cannot correlate replayed events

If the platform has no replay tooling, first build a dry-run report from dead-letter blobs. The report should list the events that would be replayed and the reason each one is included.

Decide replay, discard, hold or rollback

Keep the decision explicit. A dead-letter backlog can represent valid work to recover, poison events to discard, or a configuration change to roll back.

text event-grid-decision.txt

Replay
Root cause is fixed
Events are still valid
Consumer idempotency is proven
Replay scope is narrow and observable
Stop condition is assigned

Discard or archive
Events are obsolete
Consumer already rebuilt the target state
Payload schema is no longer accepted
Business owner confirms no action is needed

Hold
Failure causes are mixed
Endpoint still returns errors
Duplicate detection is missing
Event validity cannot be confirmed

Rollback
Dead letters started after subscription, endpoint, auth or filter change
Previous configuration restores delivery in test
The new filter excludes required production events
Replay would compensate for a change that is still active

The cleanest recovery may be rollback first, then replay only the events that remain valid after the route, identity, filter or endpoint behavior is stable again.

Validate after replay

A replay is not finished when the script exits. It is finished when the subscription stays healthy, the consumer state is coherent and the dead-letter container stops growing for the same reason.

text post-replay-validation.txt

Post-replay validation
Event subscription delivery succeeds for new events
Dead-letter count no longer increases for the same reason
Replayed event IDs are visible in consumer logs
Downstream state matches expected business count
Duplicate detection records are populated
Temporary replay identity or access is removed
Incident note stores sample events, KQL, replay scope and decision

Rollback replay tooling when
It uses broad storage permissions
It can publish arbitrary events without review
It bypasses normal authentication
It has no dry-run or batch limit

Keep the evidence pack. Event replay decisions are hard to review after the fact if the team only keeps the final success message.

Conclusion

Dead-lettered Event Grid events are a recovery opportunity only after diagnosis. Before replaying, prove the subscription contract, the failure reason, the endpoint behavior, the event validity and the consumer idempotency.

The production decision is then concrete: replay a narrow batch, discard obsolete events, hold mixed failures, or roll back the subscription change. That discipline keeps event-driven automation recoverable without turning replay into an uncontrolled second write path.