Azure Service Bus: repair a broken distributed trace before redeploying

An API accepts a command, publishes it to Azure Service Bus and returns 202. Seconds later, a worker fails on a downstream dependency. Application Insights shows a successful HTTP request and a worker exception, but no trace connects them. The incident turns into a timestamp hunt even though the system already carries enough identifiers to reconstruct the path.

This runbook covers an asynchronous flow instrumented with Application Insights or OpenTelemetry: producer API, Service Bus queue or topic, consumer and downstream dependencies. Its purpose is to separate an actual context break from missing ingestion or sampling, then repair propagation without changing the business message contract or redeploying the entire platform.

Freeze one asynchronous path

Start with one identifiable test message. A time window and queue name are not enough when several producers share the same entity. Capture the correlation contract before changing instrumentation.

Path under investigation
Producer: orders-api
Entry point: POST /orders
Service Bus entity: orders.commands
Consumer: orders-worker
Downstream dependencies: SQL, payment API, storage
Canary MessageId: trace-canary-20260801-001
Business CorrelationId: order-test-8472
UTC publish and processing timestamps
Producer and consumer versions

Expected evidence
One W3C trace with a stable trace ID
A Service Bus send span
A worker processing span
MessageId visible in producer and consumer logs
Downstream dependencies attached to processing
A repair, hold or rollback decision

Do not use a real customer command as the canary. Use a payload with no business side effect, or stop the scenario before any irreversible write.

Separate a trace break from missing telemetry

Three failures look similar in the portal: context is not propagated, the worker emits no telemetry, or spans exist but are sampled out or sent to another workspace. First verify that each segment exists on its own.

let StartTime = datetime(2026-08-01T06:00:00Z);
let EndTime = datetime(2026-08-01T07:00:00Z);
let Canary = "trace-canary-20260801-001";
union withsource=TelemetryTable
(AppRequests
  | where TimeGenerated between (StartTime .. EndTime)
  | project TimeGenerated, OperationId, ParentId, Name, ResultCode, Success, Properties),
(AppDependencies
  | where TimeGenerated between (StartTime .. EndTime)
  | project TimeGenerated, OperationId, ParentId, Name, ResultCode, Success, Properties),
(AppTraces
  | where TimeGenerated between (StartTime .. EndTime)
  | project TimeGenerated, OperationId, ParentId, Name=Message, ResultCode="", Success=true, Properties),
(AppExceptions
  | where TimeGenerated between (StartTime .. EndTime)
  | project TimeGenerated, OperationId, ParentId, Name=OuterMessage, ResultCode="", Success=false, Properties)
| where tostring(Properties) has Canary
| order by TimeGenerated asc

If producer and consumer telemetry exists under two different OperationId values, the break is likely at the messaging boundary. If the whole worker segment is missing, inspect its SDK, Application Insights connection string, OpenTelemetry exporter and destination workspace before touching Service Bus. If data appears late, measure ingestion delay before drawing a conclusion.

Verify W3C context at the message boundary

W3C trace context relies on traceparent and optionally tracestate. Auto-instrumentation may inject and extract them, but an internal wrapper, DTO mapping, SDK bridge or publisher that rebuilds a message from its body alone can break the chain.

Capture the application properties of a canary message in a controlled environment. Do not log the full body or sensitive data.

Producer side
An Activity or span is active when Send executes
Every service uses W3C propagation
Service Bus instrumentation is enabled
The publishing wrapper preserves application properties
The send span identifies the expected namespace and entity

Message
MessageId is stable and safe to log
traceparent is present and syntactically valid
tracestate is preserved when used
Business CorrelationId does not replace the technical trace ID
A retry does not silently create a new logical message

Consumer side
Context is extracted before the processing span starts
The processing span has the expected parent or explicit link
Handler logs observe the active Activity
Downstream dependencies inherit that context

Do not turn the business CorrelationId into a substitute for traceparent. The former finds a domain command. The latter records technical causality between spans. Both matter, and they have different lifecycles.

Rebuild the break with KQL

Once the canary is located, compare the operations around its MessageId. The following query uses application-enriched properties; adapt dimension names to the schema your instrumentation actually emits.

let StartTime = datetime(2026-08-01T06:00:00Z);
let EndTime = datetime(2026-08-01T07:00:00Z);
let Canary = "trace-canary-20260801-001";
let Telemetry = union
(AppRequests | project TimeGenerated, ItemType="request", OperationId, ParentId, Name, Success, Properties),
(AppDependencies | project TimeGenerated, ItemType="dependency", OperationId, ParentId, Name, Success, Properties),
(AppTraces | project TimeGenerated, ItemType="trace", OperationId, ParentId, Name=Message, Success=true, Properties),
(AppExceptions | project TimeGenerated, ItemType="exception", OperationId, ParentId, Name=OuterMessage, Success=false, Properties);
Telemetry
| where TimeGenerated between (StartTime .. EndTime)
| extend MessageId = tostring(Properties["messaging.message.id"]),
       Entity = tostring(Properties["messaging.destination.name"]),
       Role = tostring(Properties["cloud.role"])
| where MessageId == Canary or tostring(Properties) has Canary
| project TimeGenerated, Role, ItemType, Name, OperationId, ParentId, MessageId, Entity, Success
| order by TimeGenerated asc

The first point where OperationId changes without a documented link locates the break. Account for messaging semantics, however: processing may use a direct parent or an explicit span link depending on the consumption model. The goal is not one particular portal shape. It is navigable causality from publish to delivery, processing and downstream effects.

Check sampling before changing code

Two services can propagate context correctly and still retain incomplete traces when their sampling policies differ. Compare producer and consumer settings: ratio, adaptive sampling, span-type rules, exporter, processor and OpenTelemetry resource.

For a canary, apply a temporary narrow rule that retains the test trace using a non-sensitive attribute. Do not increase global production sampling to find one incident. The cost and volume obscure the root cause, and the change can overload the telemetry pipeline.

canary:
message_id: trace-canary-20260801-001
business_side_effects: disabled
expected_segments:
  - api request
  - service bus send
  - service bus process
  - downstream dependency
temporary_controls:
  keep_canary_trace: true
  log_message_body: false
  expiry: 30m
stop_when:
  - payload data appears in telemetry
  - telemetry volume exceeds budget
  - worker processes an unexpected command
  - correlation still breaks after one controlled attempt

Choose the narrowest repair

The first missing evidence determines the fix. Keep the change reversible and scoped to one component.

Repair propagation
traceparent is missing from message properties
The producer has an active span
The publishing wrapper drops metadata

Repair extraction
traceparent is present and valid
The worker starts a new trace with no parent or link
Context is extracted after the handler begins

Repair telemetry
Correlation exists inside the runtime
Spans or logs are not exported
The worker targets a different workspace or exporter

Repair sampling
OperationId matches on retained telemetry
Gaps follow a different sampling policy
A retained canary confirms the complete chain

Roll back
The break starts with an SDK or instrumentation upgrade
The fix changes the message body or business contract
Telemetry volume increases without restoring causality
Publish or processing performance regresses

Avoid copying headers manually in several places when the chosen SDK and instrumentation already support propagation. Local duplication creates two authorities and makes the next upgrade harder to reason about.

Validate and remove temporary controls

Deploy to one producer or consumer first. Send a canary, then verify the end-to-end trace, processing time, retries, telemetry volume and absence of sensitive data. A complete trace that doubles worker latency is not a validated fix.

Production validation must prove three things: new messages remain correlated, messages already queued still process correctly, and rollback restores the previous instrumentation without touching business data. Then remove the canary sampling rule and temporary logs.

Conclusion

A broken trace at the Service Bus boundary does not require a full-chain redeployment. Start from a side-effect-free message, prove every segment, inspect traceparent, separate propagation from export and sampling, then change the first faulty component.

Keep the repair only when a canary connects request, send, process and downstream dependencies at an acceptable cost. Otherwise roll back instrumentation, preserve the evidence identifiers and resume diagnosis from the last confirmed span.