Cloud

Azure NAT Gateway: diagnose SNAT exhaustion and outbound IP drift

An operational runbook for qualifying Azure outbound incidents involving NAT Gateway, SNAT ports, UDR, NSG, firewalls, allowlists and rollback.

17 Jun 2026 azurenat-gatewaysnatoutbound-networkingudrnsgfirewallapp-serviceaksobservabilitykqlrunbookrollback

A NAT Gateway incident rarely announces itself as a NAT incident. An external API becomes intermittent, a partner rejects an unexpected source IP, workers see timeouts, or a SaaS endpoint starts rejecting requests that were valid yesterday. The easy reaction is to widen the destination allowlist or redeploy the application.

The safer approach is to read Azure outbound access as an operable chain: source subnet, routes, filtering, NAT Gateway, SNAT ports, source IP observed by the destination, logs and rollback. The use case is an Azure workload calling the Internet, a partner API, a SaaS endpoint or any public service protected by source IP allowlisting.

Read outbound as a chain

NAT Gateway works at subnet level. It does not fix a route that sends traffic to a firewall, wrong DNS resolution, a blocking NSG rule or a destination that rejects the source IP it actually sees.

text azure-outbound-nat-chain.txt

Workload
App Service through VNet Integration, AKS node subnet, VM, job, runner or appliance
Source subnet documented
Expected destination hostname, port and protocol

Resolution
FQDN resolved from the workload path
Public or private address expected
No conclusion based only on the admin workstation

Routing
Effective route from the source subnet
UDR toward Internet, NAT Gateway path, Azure Firewall or NVA
No accidental default route overriding the expected egress

Filtering
NSG on the source subnet or NIC
Azure Firewall or NVA rule when traffic is routed there
Destination-side allowlist or policy

NAT and evidence
NAT Gateway attached to the source subnet
Expected public IP or prefix
SNAT capacity and connection churn
Destination logs, firewall logs and application traces aligned by timestamp

This prevents two common mistakes: diagnosing SNAT exhaustion while traffic is bypassing NAT Gateway, or changing the application when the destination is simply seeing a different IP than the one documented.

Classify the symptom before changing the allowlist

The same timeout can come from SNAT pressure, firewall filtering, routing, destination policy or DNS. Classify the symptom first, with the test location attached.

text nat-gateway-symptoms.txt

Observed symptom
Intermittent timeouts under load
  Check connection reuse, TCP churn, SNAT capacity and NAT Gateway metrics

403 or partner-side rejection
  Compare the observed outbound IP with the allowlist currently active

Works from a VM but not from the application
  Replay from the real source subnet or an equivalent probe

No NAT Gateway or firewall log
  Go back to DNS, effective route, NSG or a wrong test point

Incident after a network change
  Compare UDR, NAT Gateway association, public IP, prefix, firewall and NSG

Incident after traffic growth
  Check SNAT ports, connection pooling, keep-alive and fan-out toward the same destination

A broader allowlist may hide the problem for a few hours. It does not prove that egress is controlled.

Prove the subnet and outbound IP

Start by proving that the workload exits from the subnet you are diagnosing. Then verify the NAT Gateway association and the IP observed by the destination.

bash 01-nat-gateway-subnet-egress-check.sh

VNET_RG=rg-prod-network
VNET=vnet-prod-spoke
SUBNET=snet-prod-workers

az network vnet subnet show -g "$VNET_RG" --vnet-name "$VNET" -n "$SUBNET" --query "{name:name,addressPrefix:addressPrefix,natGateway:natGateway.id,routeTable:routeTable.id,nsg:networkSecurityGroup.id}" -o jsonc

az network nat gateway show --ids "$(az network vnet subnet show -g "$VNET_RG" --vnet-name "$VNET" -n "$SUBNET" --query natGateway.id -o tsv)" --query "{name:name,publicIps:publicIpAddresses[].id,publicPrefixes:publicIpPrefixes[].id,sku:sku.name,idleTimeout:idleTimeoutInMinutes}" -o jsonc

# Run from the workload, a diagnostic container or an equivalent probe.
curl -s https://ifconfig.me
curl -vk --connect-timeout 5 https://api.partner.example/health

For App Service or Functions with VNet Integration, the test should run from the application context or a truly equivalent point. For AKS, separate node subnet egress from pods, ingress paths and appliances.

Read routes, NSG and firewall together

NAT Gateway is not always the last component before the Internet. A default route to Azure Firewall or an NVA may be intentional. In that case, the destination may see the firewall public IP, not the NAT Gateway IP.

bash 02-effective-route-and-filter-check.sh

NIC_ID=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod/providers/Microsoft.Network/networkInterfaces/nic-prod-worker-01

az network nic show-effective-route-table --ids "$NIC_ID" -o table
az network nic list-effective-nsg --ids "$NIC_ID" -o table

# If the subnet is routed to Azure Firewall, diagnose the firewall path before concluding on NAT Gateway.
FIREWALL_RG=rg-prod-sec
FIREWALL_NAME=afw-prod

az network firewall show -g "$FIREWALL_RG" -n "$FIREWALL_NAME" --query "{name:name,ipConfigurations:ipConfigurations[].publicIpAddress.id,sku:sku.tier}" -o jsonc

The diagnostic question is therefore simple: which component actually carries the source IP for this exact flow?

Look for SNAT exhaustion with evidence

SNAT exhaustion usually appears when many outbound connections target the same endpoint, when connections are short-lived, when pooling is weak or when traffic suddenly grows. The typical symptom is intermittent: low traffic works, then timeouts start under load.

text snat-evidence-pack.txt

Minimal evidence pack
Exact incident window
Source subnet and attached NAT Gateway
Destination, port and protocol
Connection volume and application error rate
Outbound IP observed by the destination
NAT Gateway metrics on the same window
Firewall or destination logs when an intermediary exists
Recent change in route, NAT, public IP, code or traffic

Available metrics depend on configuration and collection. The principle stays the same: correlate SNAT capacity, connection attempts and application errors on the same time window.

kusto 03-egress-errors-correlation.kql

let Window = 2h;
let TargetHost = "api.partner.example";
AppTraces
| where TimeGenerated > ago(Window)
| where Message has_any (TargetHost, "timeout", "SocketException", "connection reset", "connection refused", "403")
| project TimeGenerated, AppRoleName, SeverityLevel, Message, OperationId
| order by TimeGenerated desc

If the destination can provide logs, request source IP, timestamp, path and status. A 403 with an unexpected IP is not the same incident as a timeout with no destination trace.

Correct without opening too broadly

The correction must target the proven layer. Adding an IP to an allowlist can be correct when the observed IP is legitimate and documented. Adding a wide range or bypassing the firewall without evidence creates an operations debt.

text nat-gateway-correction-matrix.txt

Proven cause
NAT Gateway missing from the source subnet
  Correction: attach NAT Gateway to the right subnet or fix the egress point
  Validation: destination sees the expected IP

Default route to firewall or NVA
  Correction: document real egress or adjust UDR and firewall rules
  Validation: firewall and destination logs match

Outbound IP not aligned with the allowlist
  Correction: align public IP, prefix or targeted allowlist entry
  Validation: timestamped test from the workload

Likely SNAT exhaustion
  Correction: reduce churn, reuse connections, distribute load or increase outbound capacity
  Validation: timeouts decrease and metrics stabilize

NSG or firewall blocks the flow
  Correction: targeted source, destination, port and protocol rule
  Validation: deny disappears without broad opening

In many incidents, the best correction is not a network change: enabling HTTP pooling, reducing fan-out, setting a sane timeout or avoiding a new client per request can sharply reduce SNAT pressure.

Prepare rollback and decision

Any egress change needs an explicit rollback. Otherwise the team discovers too late that nobody knows which IP a partner should authorize.

text nat-gateway-rollback.txt

Change
NAT Gateway association
  Rollback: reattach the previous NAT Gateway or remove the faulty association
  Evidence: before and after outbound IP plus destination test

Public IP or prefix
  Rollback: restore the previously authorized IP or prefix
  Evidence: partner allowlist and rejection logs

UDR toward firewall or Internet
  Rollback: restore the previous route
  Evidence: effective route and egress component logs

NSG or firewall rule
  Rollback: return to the previous rule or targeted temporary exception
  Evidence: deny identified and scope limited

Application pooling change
  Rollback: return to the previous version if functional errors appear
  Evidence: SNAT timeouts reduced or returned

Return to service should be evidence-based: the outbound IP is expected, destination logs are clean, timeouts have stopped, and the network path is documented.

Conclusion

A NAT Gateway incident is rarely solved by looking at NAT Gateway alone. Prove the source subnet, effective route, filtering layer, observed IP, SNAT pressure and destination logs.

The decision then becomes defensible: fix the NAT association when the subnet is wrong, adjust UDR when traffic exits elsewhere, target NSG or firewall when a deny is proven, align the allowlist when the IP drifted, or reduce application churn when SNAT exhaustion is the real signal.