Networking
Azure route asymmetry: diagnose before changing a UDR
A production runbook for qualifying Azure asymmetric routing with effective routes, UDRs, Azure Firewall, NAT Gateway, flow logs, validation and rollback before changing route tables.
Asymmetric routing is one of the fastest ways to make an Azure network incident confusing. The request leaves a workload subnet through the expected next hop, but the response comes back through a different path, bypasses a firewall, hits a different appliance, returns through a NAT Gateway, or disappears behind a route that only exists on one side. The symptom looks like a timeout, an intermittent 502, a firewall denial, a missing log or an application retry storm.
The risky reflex is to edit the UDR immediately: add a broader prefix, point everything to the firewall, remove a route, or open an NSG rule to make the test pass. That can hide the real fault and create a second outage. The use case here is a hub-and-spoke Azure environment where a workload calls an internal API, a private service or an on-premises dependency through controlled routing. The runbook goal is to prove the forward path and return path before changing route tables.
Freeze the flow and the expected path
Start by naming one flow. Do not diagnose the whole VNet at once. A route table change is only defensible when the exact source, destination, protocol, port and expected next hops are known.
Flow to qualify
Source workload: app-prod-api
Source subnet: snet-app-prod / 10.42.12.0/24
Source IP: 10.42.12.18
Destination: orders.internal.contoso.net / 10.55.4.20
Protocol and port: TCP 443
Expected forward path: spoke subnet -> Azure Firewall -> hub peering -> service subnet
Expected return path: service subnet -> Azure Firewall -> spoke subnet
Recent change: route table update on snet-app-prod or service subnet
Evidence required before changing UDR
DNS resolution from source
Effective route on the source NIC
Effective route on the destination NIC or subnet
Firewall or appliance logs for both directions
NSG flow evidence when available
Application replay with a correlation ID
Rollback route table or previous associationIf the team cannot describe the return path, it should not edit the forward route yet. Many asymmetric incidents are caused by assuming that route tables are symmetrical when they are not.
Prove DNS is not changing the target
Before comparing routes, prove that the caller resolves the intended address. A route investigation becomes meaningless if one side is calling a public endpoint, an old private IP, or a different regional hostname.
DNS checks from the workload path
Resolve the FQDN from the source subnet or runner
Compare resolved IP with the expected dependency IP
Check private zone links and forwarding rules when private DNS is involved
Confirm no local override or stale cache points to an old endpoint
Repeat from the destination side if the response path uses a named callback
Block UDR changes when
Source resolves a public address unexpectedly
Different callers resolve different private IPs without an owner
The failing test used an IP while production uses an FQDN
DNS changed during the incident windowPrivate Endpoint may appear in this check when the dependency uses a private address, but it is not the main diagnosis. The question is whether both sides agree on the address being routed.
Compare effective routes on both sides
A route table shows intent. Effective routes show what the NIC actually uses after system routes, peering, BGP, UDRs and propagated routes are combined. Capture effective routes on the source and on the destination side before changing either table.
SOURCE_NIC="nic-app-prod-api-01"
SOURCE_RG="rg-prod-app"
DEST_NIC="nic-orders-prod-01"
DEST_RG="rg-prod-orders"
DEST_IP="10.55.4.20"
SOURCE_IP="10.42.12.18"
az network nic show-effective-route-table --resource-group "$SOURCE_RG" --name "$SOURCE_NIC" --output table
az network nic show-effective-route-table --resource-group "$DEST_RG" --name "$DEST_NIC" --output table
# Then inspect the route covering DEST_IP on the source side
# and the route covering SOURCE_IP on the destination side.Look for different next-hop types: VirtualAppliance, VnetLocal, VNetPeering, Internet, VirtualNetworkGateway or None. The problem is not only a missing route. It may be a more specific prefix that wins on one side, a propagated route that overrides expectation, or a subnet associated with the wrong route table.
Inspect route table associations and recent drift
A correct UDR in the wrong route table does not help. A route table attached to only one subnet can create exactly the asymmetry operators are trying to fix.
VNET_RG="rg-network-prod"
VNET="vnet-spoke-prod"
az network vnet subnet list --resource-group "$VNET_RG" --vnet-name "$VNET" --query "[].{subnet:name,addressPrefix:addressPrefix,routeTable:routeTable.id,nsg:networkSecurityGroup.id,natGateway:natGateway.id}" --output table
az network route-table list --resource-group "$VNET_RG" --query "[].{name:name,routes:routes[].{name:name,prefix:addressPrefix,nextHop:nextHopType,nextHopIp:nextHopIpAddress}}" --output jsonAlso check recent writes on route tables, peerings, firewall policy, NAT Gateway association and subnet configuration. A route may have changed because an IaC deployment re-associated a subnet, not because someone edited the route itself.
Correlate firewall and flow evidence
Asymmetry often appears as missing evidence. The firewall sees the request but not the response, or the response but not the request. NSG flow logs may show denies on one side while the firewall looks clean.
let StartTime = datetime(2026-07-04T10:00:00Z);
let EndTime = datetime(2026-07-04T10:30:00Z);
let SourceIp = "10.42.12.18";
let DestinationIp = "10.55.4.20";
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where Category has_any ("AzureFirewallNetworkRule", "AzureFirewallApplicationRule")
| where msg_s has_any (SourceIp, DestinationIp)
| project TimeGenerated, Category, msg_s, action_s, protocol_s, source_s, destination_s, destinationPort_s
| order by TimeGenerated ascA single allowed entry is not enough. For stateful inspection, confirm that the expected device observes the complete conversation. If a central firewall is supposed to inspect the path but never sees the return traffic, adding an application rule will not fix the route.
Check NAT Gateway and outbound identity
NAT Gateway can make the symptom look like firewall or allowlist drift. A workload may leave through NAT on one path and through Azure Firewall or an appliance on another. The destination then sees a different source IP than expected.
Check NAT impact
Is NAT Gateway associated with the source subnet?
Is the destination allowlisting the firewall public IP, NAT public IP or private source IP?
Did a route change move traffic away from the NAT path?
Are SNAT metrics or connection counts abnormal?
Does the response path expect the original private source or the translated address?
Block route edits when
The observed source IP at the destination is unknown
NAT Gateway and Azure Firewall both appear in possible paths
The team cannot explain which public IP should be allowlisted
A rollback would change outbound identity without noticeRouting and identity meet at egress. The route may be technically valid while the dependency rejects the new source identity.
Validate with a controlled replay
Do not validate route changes with a vague application retry. Replay one flow with a timestamp, source host, destination, port and correlation ID. Capture logs from the source, firewall or appliance, destination and monitoring workspace.
replay:
flow_id: route-asymmetry-20260704-01
source: app-prod-api / 10.42.12.18
destination: orders.internal.contoso.net / 10.55.4.20
protocol: tcp
port: 443
method:
- resolve destination from source
- open one controlled TCP or HTTPS request
- include correlation header when HTTP is available
- capture effective route snapshots before and after change
- query firewall and flow logs for both IPs
success:
- source route points to expected next hop
- destination route back to source points to expected next hop
- firewall or appliance observes expected direction
- application receives a valid response
cleanup:
- remove temporary diagnostic rule or host route
- restore previous route table if validation failsA useful replay makes the route decision auditable. It also prevents the team from declaring success because another retry happened to pass through a different instance.
Decide change, rollback or block
A UDR change should be the result of the diagnosis, not the diagnostic method. Keep the decision explicit.
Change the UDR
Effective route proves the wrong next hop for the exact prefix
Return path is understood and stays symmetrical
Firewall or appliance policy already allows the intended flow
NAT or outbound identity impact is documented
Route table rollback is ready
Rollback the route table
Incident started after a route table association or route update
Previous route restored the expected forward and return path in test
The new route sends only one side through inspection
Application recovery is proven after rollback
Fix firewall, NSG or DNS instead
Routes are correct on both sides
Firewall denies the expected flow
NSG flow evidence shows a subnet or NIC deny
DNS resolves the wrong target
Block change
Only the forward path is known
Destination-side effective route is missing
No logs can prove where the return path goes
The proposed route uses a broad prefix to mask uncertainty
Rollback would affect unrelated workloadsThe safest correction is often a narrower prefix, a route table association fix, a peering propagation correction, or a rollback to the last known route table. Broad prefixes and emergency bypasses should expire quickly and stay attached to evidence.
Keep the rollback observable
Routing rollback is not complete when the command succeeds. It is complete when the original path is visible again and no unrelated workload has been moved.
Rollback validation
Previous route table or route entry restored
Effective routes checked on source and destination NICs
Firewall or appliance logs show the expected conversation
NAT or outbound source identity matches the documented path
Application probe succeeds with the same FQDN and port
Monitoring shows no new denies for adjacent workloads
Incident record includes route diff, decision and cleanup owner
Rollback is incomplete when
Only the source route was checked
A temporary broad route remains without expiry
The destination sees a different source identity
Firewall logs are still missing one direction
The route table is fixed manually but IaC will reintroduce driftIf infrastructure as code manages the route table, the rollback must include the IaC state or pull request. Otherwise the next deployment can recreate the asymmetry.
Conclusion
Asymmetric routing is not solved by adding a bigger UDR. It is solved by comparing the path in both directions: DNS target, effective routes, route table association, firewall or appliance evidence, NAT identity and controlled replay.
The production decision is simple only after that evidence exists: change a narrow route, fix another layer, rollback the route table, or block the change until the return path is known. That discipline keeps private Azure paths explainable instead of turning routing into trial and error.