Azure: make private paths verifiable with synthetic probes

An Azure private path can look correct on paper and remain fragile in operations. The Private Endpoint exists, the private DNS zone is linked, Application Gateway has a backend, and WAF logs requests, but the team may still not know quickly whether the complete path works from the right network, with the right name, certificate and application behavior.

The use case is intentionally concrete: an internal application is published behind Application Gateway, reaches a private backend through Private Endpoint and depends on hybrid DNS resolution. The team wants to detect drift before the incident, not only discover a 502, timeout or wave of WAF blocks after deployment. Synthetic probes become an operational control, not just a monitoring extra.

Test the real path, not only the resource

A useful probe does not only check that an Azure resource exists. It crosses the same path that the service or user actually uses. For a private backend, this means starting from a representative test point: CI runner in the private network, administration VM, diagnostic container or authorized monitoring worker.

The control should separate several questions:

• does the application name resolve to the expected private address?
• does the CNAME chain go through the expected privatelink domain?
• does the port answer from the consumer network?
• does the presented certificate match the requested name?
• does Application Gateway see the backend as healthy?
• does WAF block a legitimate probe request?
• does the application return the expected status on a lightweight endpoint?

Keep those questions separate. A probe that only returns down forces the team to rebuild the diagnosis from scratch.

Build a minimal probe matrix

The right starting point is a short matrix. It connects each check to evidence, threshold and expected action. The goal is not perfect observability; it is avoiding alerts that cannot be used.

Check
Expected public DNS closure
Private DNS from workload network
TCP connection to backend
TLS with application hostname
Application Gateway health probe
Lightweight application request
WAF logs on the same time window

Expected evidence
Private address or controlled failure
Consistent privatelink CNAME
Port open from the right network
Certificate valid for the name
Backend healthy
HTTP 200, 204 or expected business status
No unexpected block on the probe

Version this matrix with the runbook. When an endpoint changes, the team knows which check must move and which behavior is still acceptable.

Do not mix DNS probes and application probes

A DNS failure does not mean the same thing as an application failure. If the name still resolves publicly, the issue is likely in the private zone, VNet link, hybrid forwarding or resolver cache. If DNS is correct but HTTP fails, diagnosis moves to Application Gateway, TLS, health probe, WAF or the application.

The DNS probe should therefore be explicit:

HOST=api.internal.example.com

dig +short "$HOST"
dig +short CNAME "$HOST"

resolved_ip="$(dig +short "$HOST" | tail -n 1)"
case "$resolved_ip" in
10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.2*|172.30.*|172.31.*|192.168.*)
  echo "private_dns_ok=$resolved_ip"
  ;;
*)
  echo "private_dns_unexpected=$resolved_ip"
  exit 2
  ;;
esac

This probe can run from an authorized runner. It does not prove that the application works, but it prevents a naming issue from being confused with an application issue.

Verify TLS with the real hostname

Many Application Gateway diagnostics become unclear when the backend is queried with an IP address, a technical name or a hostname different from the configured one. The TLS probe must use the name expected by the backend or by Application Gateway.

HOST=api.internal.example.com
PORT=443

echo | openssl s_client -connect "$HOST:$PORT" -servername "$HOST" 2>/dev/null | openssl x509 -noout -subject -issuer -dates

Read the result in path context: expired certificate, wrong SAN, missing SNI, incorrect backend setting or Application Gateway probe using the wrong hostname. The value is producing fast evidence before changing routing.

Connect the probe to WAF and Application Gateway logs

A synthetic probe should leave enough trace to be found later. A dedicated user-agent or correlation header makes the logs filterable without polluting user traffic analysis.

HOST=https://api.internal.example.com/healthz
RUN_ID="synthetic-private-path-$(date -u +%Y%m%d%H%M)"

curl -fsS "$HOST" -H "User-Agent: naxaya-private-path-probe" -H "X-Probe-Run: $RUN_ID" --max-time 5

In logs, this marker helps separate a real outage from user traffic noise. It also checks whether WAF started blocking the probe after a rule change.

Define the runbook before the alert

A failing probe should point to a short reading path. The runbook should not start with “open the Azure portal”. It should state the first split to make.

If the DNS probe fails
Check privatelink CNAME
Check Private DNS Zone / VNet link
Check hybrid forwarding and cache

If DNS is OK but TLS fails
Check SNI, certificate SAN, backend setting and expiration

If TLS is OK but HTTP fails
Check Application Gateway health probe, backend pool and application logs

If HTTP is expected but WAF blocks
Read ruleId, match field, URI and user-agent in KQL

If it works from a VM but not from CI
Compare identity, DNS resolver, route and runner network

This structure reduces incident time. It also prevents premature fixes: changing WAF while DNS drifted, or changing the application while the health probe uses the wrong hostname.

Automate without making diagnosis opaque

Synthetic probes can run from a scheduled job, pipeline, AWX script or monitoring function. Automation is useful if it keeps results readable: DNS output, TLS status, HTTP code, run identifier, log link and last known change.

A good alert signal should stay precise:

• private_dns_unexpected
• tls_hostname_mismatch
• appgw_backend_unhealthy
• waf_blocked_synthetic_probe
• http_status_unexpected
• ci_runner_path_differs_from_workload

These names are more useful than a single “probe failed” alert. They feed tags, paths and the Failure Atlas better because they describe an actionable symptom.

Conclusion

A synthetic probe is useful only if it crosses the real path and produces readable evidence. For Azure private paths, it should separate DNS, network, TLS, Application Gateway, WAF and application behavior, then connect each failure to a first check.

The value is not only knowing that an endpoint answers. It is turning a private path that is hard to explain into a measurable runbook: where to test, what to observe, which signal to correlate and which decision to make before touching configuration.