Networking
Azure Load Balancer: diagnose a health probe before changing the backend pool
A production runbook for qualifying an unhealthy Azure Load Balancer backend with probe behavior, NSGs, routing, logs, pool configuration, validation and rollback before changing the rule or redeploying.
An Azure Load Balancer backend marked unhealthy often triggers the wrong reflexes: remove a VM from the pool, change the probe port, open an NSG broadly, redeploy the service or move traffic to another instance. Those actions may make a test pass, but they can also hide a simpler break: a probe sent to the wrong port, a service listening only on localhost, an asymmetric return path, a host firewall, an overly strict NSG rule or an instance that no longer serves the same application health path.
The use case is an internal application exposed through Azure Standard Load Balancer, with several VMs or appliances in a backend pool, subnets controlled by NSGs and sometimes UDRs toward Azure Firewall or an NVA. The symptom is clear: one or more instances leave the pool, connections become intermittent or a failover no longer receives traffic. The runbook goal is to prove whether the fault sits in the probe, the application, the network or the pool before changing the load-balancing rule.
Freeze the health contract before correcting anything
Start by naming the health contract. A health probe is not a minor setting. It is the signal that decides whether an instance is allowed to receive traffic.
Load Balancer health contract
Load balancer: lb-prod-internal
Frontend IP: 10.20.4.10
Backend pool: pool-orders-api
Instances: vm-orders-01, vm-orders-02, vm-orders-03
Rule: tcp-443-orders
Probe: tcp-8080-health or http-/healthz
Interval and threshold: 5s / 2 failures
Backend subnet: snet-app-prod
Expected path: LB -> backend private IP -> local service -> probe response
Recent change: application deployment, NSG, route table, VM image, extension or host firewall
Evidence required before change
Full LB, rule, probe and backend pool configuration
Backend state per instance
Local service listener on the probe port
Effective NSGs and effective routes on the backend NIC
Application or system logs during the probe window
Test from a VM in the same subnet or an operations runner
Rollback plan for the probe, pool or deploymentIf the team cannot explain why the probe uses that port and path, it should not relax the probe yet. A probe that is too permissive can put an instance back into the pool even though it accepts TCP but no longer serves the application correctly.
Capture the real Load Balancer configuration
The portal is useful, but diagnosis needs a snapshot that can be reviewed later. Capture the rule, probe, backend pool and NIC associations before any modification.
RESOURCE_GROUP="rg-network-prod"
LOAD_BALANCER="lb-prod-internal"
az network lb show --resource-group "$RESOURCE_GROUP" --name "$LOAD_BALANCER" --query "{frontend:frontendIPConfigurations[].{name:name,ip:privateIPAddress},rules:loadBalancingRules[].{name:name,frontendPort:frontendPort,backendPort:backendPort,protocol:protocol,probe:probe.id,backendPool:backendAddressPool.id},probes:probes[].{name:name,protocol:protocol,port:port,requestPath:requestPath,interval:intervalInSeconds,threshold:numberOfProbes},pools:backendAddressPools[].{name:name,backendIPConfigurations:backendIPConfigurations[].id}}" --output json
az network lb address-pool show --resource-group "$RESOURCE_GROUP" --lb-name "$LOAD_BALANCER" --name "pool-orders-api" --output jsonThen verify that every expected instance is actually associated with the pool. An unhealthy backend is not always a failing probe. It can also be a removed NIC, an association moved by IaC, or a different pool from the one used by the active rule.
Separate TCP and HTTP probe semantics
A TCP probe only proves that a socket accepts a connection. An HTTP probe proves that a response is returned on a path. They do not carry the same level of confidence.
TCP probe
Validates: open port and TCP handshake
Does not validate: application dependencies, business readiness, HTTP code, authentication
Risk: putting a blocked service back into the pool because the port is open
HTTP probe
Validates: HTTP endpoint returns a status accepted by the probe
Does not always validate: deep dependencies, message queue, downstream database
Risk: removing a healthy instance if /healthz depends on an unstable external service
Questions before change
Does the probe measure readiness or only connectivity?
Did the /healthz path change with the last deployment?
Does the service listen on all interfaces or only on 127.0.0.1?
Is the probe port the same as the rule backendPort?
Does authentication, redirect or host header behavior block the probe?Changing an HTTP probe to TCP can calm the incident while sending traffic to an instance that cannot process requests. That change should be a documented workaround decision, not an automatic fix.
Prove the service listens locally
Before opening the network, verify the instance. Many probes fail because the process no longer listens, listens on the wrong port, or exposes the health endpoint only on localhost.
# On each affected backend
sudo ss -lntp | grep -E ':8080|:443'
curl -sv --max-time 3 http://127.0.0.1:8080/healthz
curl -sv --max-time 3 http://$(hostname -I | awk '{print $1}'):8080/healthz
systemctl status orders-api --no-pager
journalctl -u orders-api --since "30 minutes ago" --no-pager | tail -100If 127.0.0.1 responds but the VM private address does not, the fault is not the Load Balancer. It is in the application binding, host firewall or service configuration. If the service responds locally but not from the subnet, move to the network path.
Check NSGs, routes and host firewall
Azure Load Balancer probes come from Azure infrastructure and must be allowed to reach the backend. With Standard Load Balancer, NSGs must explicitly allow the expected traffic, including the probe from the AzureLoadBalancer service tag when that pattern is used in policy.
BACKEND_NIC="nic-vm-orders-01"
BACKEND_RG="rg-prod-app"
PROBE_PORT="8080"
az network nic list-effective-nsg --resource-group "$BACKEND_RG" --name "$BACKEND_NIC" --output table
az network nic show-effective-route-table --resource-group "$BACKEND_RG" --name "$BACKEND_NIC" --output table
az network watcher test-ip-flow --resource-group "$BACKEND_RG" --vm "vm-orders-01" --direction Inbound --protocol TCP --local 10.20.5.14 "$PROBE_PORT" --remote 168.63.129.16 65503 --output jsonTreat 168.63.129.16 as a useful operational signal for testing some Azure paths, not as the only proof. The decision should combine effective NSGs, effective routes, host firewall behavior and application response. If a UDR forces an unexpected return path toward an NVA, the probe may fail even though the port is open.
Read metrics and logs as a timeline
An unhealthy health probe must be placed on a timeline. The important question is not only the current state, but when it changed relative to a deployment, NSG rule, image rotation or system update.
let StartTime = datetime(2026-07-09T08:00:00Z);
let EndTime = datetime(2026-07-09T09:00:00Z);
AzureMetrics
| where TimeGenerated between (StartTime .. EndTime)
| where ResourceProvider =~ "MICROSOFT.NETWORK"
| where MetricName in ("DipAvailability", "VipAvailability", "HealthProbeStatus")
| project TimeGenerated, Resource, MetricName, Average, Minimum, Maximum, backendIPAddress_s=tostring(Tags["BackendIPAddress"]), frontendIPAddress_s=tostring(Tags["FrontendIPAddress"])
| order by TimeGenerated ascAdd application logs, system logs, Azure Activity Log changes and any appliance logs. A probe that drops exactly during an application deployment does not call for the same fix as a probe that drops when an NSG is associated.
Test from the closest useful path
The Load Balancer path is not always testable from outside production. Use an operations VM in the same VNet, or as close to the backend subnet as possible, to reproduce what the probe expects: port, protocol, HTTP path and backend private address.
BACKEND_IP="10.20.5.14"
PROBE_PORT="8080"
FRONTEND_IP="10.20.4.10"
# Test the backend directly from an operations VM in the VNet.
nc -vz "$BACKEND_IP" "$PROBE_PORT"
curl -sv --max-time 3 "http://${BACKEND_IP}:${PROBE_PORT}/healthz"
# Then test the frontend on the application port.
nc -vz "$FRONTEND_IP" 443
curl -skv --max-time 5 "https://orders.internal.example/healthz" -H "x-correlation-id: lb-health-20260709-01"If the backend responds from a VM in the same subnet but remains unhealthy in the Load Balancer view, go back to probe, pool and rule configuration. If the backend does not answer from the VNet, the Load Balancer is only revealing a lower-level fault.
Decide fix, removal, rollback or block
The correction must follow the evidence. Keep the decision explicit to avoid stacking changes during the incident.
Fix the probe
The service exposes a new valid health path
The historical probe port no longer matches the intended service
The current probe depends on an unstable external dependency
The change is tested on one instance before rollout
Probe rollback is documented
Fix the backend
The service does not listen on the private address
The host firewall blocks the probe port
The process is failing or returns an unsupported code
The last deployment broke /healthz
Fix the network
Effective NSG blocks probe or backend traffic
UDR or appliance breaks the return path
NIC, subnet or pool association drifted
Effective routes no longer match the expected path
Temporarily remove an instance
Only one VM is unhealthy
Other backends are healthy and sized for the load
Removal is tracked and reversible
The application owner accepts the capacity impact
Rollback
The incident starts after a probe, rule, NSG, UDR, pool or application release change
Returning to the previous state restores probe health in test
The proposed correction increases risk for other flows
Block the change
No local service evidence exists
Backends are not identified individually
The proposed plan opens an NSG too broadly
The probe would be weakened without application validationThis decision prevents a health-check incident from becoming a permanent security or routing change.
Validate after the fix and prepare the return path
Validation does not stop when the backend turns healthy. Confirm that application traffic returns, capacity is coherent and the probe has not become silent.
Post-fix validation
Every expected backend is healthy with its identified IP address
The frontend answers on the application port
Application logs show real requests on all healthy instances
DipAvailability and VipAvailability remain stable
No temporary NSG or host firewall opening remains without expiry
The matching IaC change is aligned with the corrected state
Incident notes keep initial config, evidence, decision and rollback
Rollback is incomplete when
An instance was removed from the pool without a return ticket
The probe was weakened without application testing
A diagnostic port remains allowed
Effective routes were not rechecked after correction
IaC will recreate the old configuration on the next deploymentIf Terraform, Bicep or a pipeline manages the infrastructure, the manual correction must be reflected in code or explicitly rolled back. Otherwise the next delivery can put the probe or pool back into the incident state.
Conclusion
An unhealthy Load Balancer backend is not an invitation to edit the balancing rule by trial and error. It is a signal to qualify: pool configuration, probe semantics, local service state, NSGs, routes, logs and controlled replay.
The correct incident exit is a traceable decision: fix the probe, fix the backend, fix the network, temporarily remove an instance, roll back the recent change or block the action because evidence is missing. That discipline keeps the Load Balancer as a reliable operating mechanism instead of a black box adjusted during the outage.