Azure Private DNS Resolver: diagnose split-brain DNS before changing zones

A hybrid DNS incident rarely announces itself as DNS. The application sees a timeout. A private API works from one subnet but not from a runner. An on-premises administrator resolves a public address while an Azure VM resolves a private one. Someone proposes to add a record, link another private zone or change a forwarder. That can fix the symptom, but it can also create a split-brain model that nobody can explain later.

The use case is an Azure environment with hub and spoke VNets, Azure DNS Private Resolver, on-premises DNS forwarders, private DNS zones and workloads that use private Azure services. The incident is simple to state: the same FQDN does not resolve to the same expected answer depending on the source network. The runbook goal is to decide whether to fix forwarding, zone links, records, caches or rollback a recent DNS change before editing production zones.

Name the failing resolution path

Start by writing the DNS path as a production flow. Do not begin with the zone. Begin with the resolver used by the caller.

Question to answer
Which source network receives the wrong DNS answer?
Which resolver does that source use?
Which suffix or FQDN is queried?
Which answer is expected: private IP, public IP, NXDOMAIN, CNAME chain?
Which recent change touched zones, VNet links, resolver rulesets or forwarders?

Sources to compare
Azure workload subnet
Azure private runner subnet
hub tools subnet
on-premises application server
operator workstation
monitoring probe location

This prevents a common shortcut: fixing a private DNS zone because one test fails, while the failing source is actually using an on-premises resolver, a custom DNS server or a stale forwarding ruleset.

Capture answers from every relevant source

A split-brain incident is proven by comparison. Use the same FQDN, same record type and same timestamp from each source. Keep the raw answer, resolver address, CNAME chain and TTL.

FQDN="myvault.vault.azure.net"
EXPECTED_SUFFIX="privatelink.vaultcore.azure.net"

echo "source=$(hostname)"
echo "resolvers"
cat /etc/resolv.conf || true

echo "short answer"
dig +short "$FQDN"

echo "trace with cname and ttl"
dig "$FQDN" A +noall +answer

echo "expected privatelink suffix"
dig "$EXPECTED_SUFFIX" A +noall +answer || true

For Windows hosts, capture the server used by the query instead of only the final IP.

$Name = "myvault.vault.azure.net"
Resolve-DnsName $Name -Type A | Select-Object Name, Type, IPAddress, NameHost, TTL
Get-DnsClientServerAddress -AddressFamily IPv4 | Select-Object InterfaceAlias, ServerAddresses

The important evidence is not that one command failed. It is that the same name crosses different resolver paths and produces different operational meaning.

Separate private zones from forwarding rules

Azure Private DNS zones and DNS forwarding rulesets solve different problems. A wrong private zone link affects Azure networks consuming that zone. A wrong outbound ruleset affects Azure workloads resolving non-Azure or on-premises suffixes. A wrong on-premises conditional forwarder affects queries entering Azure through the inbound endpoint.

If Azure workload resolves wrong private Azure answer
Check private DNS zone record
Check VNet link to the consuming VNet
Check custom DNS server configuration on the VNet
Check whether the query leaves Azure unexpectedly

If Azure workload cannot resolve on-premises name
Check DNS forwarding ruleset
Check ruleset link to the VNet
Check outbound endpoint health and target DNS servers
Check firewall path to UDP/TCP 53

If on-premises source resolves wrong Azure private answer
Check conditional forwarder suffix
Check target Azure inbound endpoint IP
Check private zone and VNet link visible to the resolver path
Check competing local zone or cached record

The fix should stay on the failing boundary. Adding a private zone link will not repair an on-premises conditional forwarder that points to the wrong inbound endpoint. Changing an outbound ruleset will not help a workload that uses a custom DNS appliance and bypasses Azure-provided DNS behavior.

Inspect Azure DNS Resolver objects

The Private Resolver configuration must be read as a route table for names: inbound endpoint for queries entering Azure, outbound endpoint and rulesets for queries leaving Azure toward other DNS servers.

RG="rg-network-prod"
RESOLVER="pdnsr-hub-prod"
RULESET="pdnsrs-hub-prod"

az network dns-resolver show --resource-group "$RG" --name "$RESOLVER" --output table

az network dns-resolver inbound-endpoint list --dns-resolver-name "$RESOLVER" --resource-group "$RG" --output table

az network dns-resolver outbound-endpoint list --dns-resolver-name "$RESOLVER" --resource-group "$RG" --output table

az network dns-resolver forwarding-ruleset rule list --resource-group "$RG" --ruleset-name "$RULESET" --output table

az network dns-resolver forwarding-ruleset virtual-network-link list --resource-group "$RG" --ruleset-name "$RULESET" --output table

Look for the absence of a link as much as for a wrong rule. A spoke created recently may simply not be linked to the ruleset that all older spokes use.

Verify private DNS zone ownership and records

A private zone can be correct and still not used by the failing source. Inventory the record, then the VNet links. Avoid creating duplicate zones with the same name in different resource groups unless there is a deliberate ownership model.

ZONE="privatelink.vaultcore.azure.net"
RG="rg-private-dns-prod"

az network private-dns record-set a list --resource-group "$RG" --zone-name "$ZONE" --output table

az network private-dns link vnet list --resource-group "$RG" --zone-name "$ZONE" --output table

If the same private zone exists twice, stop before editing records. The incident may be an ownership conflict rather than a missing DNS record.

Treat caches as evidence, not excuses

DNS caches can extend an incident after the control plane is fixed. They should be measured and flushed only after the source of truth is understood.

Capture before flush
Resolver used by the source
Current answer and TTL
Expected answer from authoritative or Azure path
First time the wrong answer was observed
Recent change time

Flush only after deciding
Local application host cache
DNS service cache on custom resolver
On-premises DNS forwarder cache
CI runner or container base image cache
Application connection pool if it pins resolved IPs

Flushing every cache before capturing answers destroys the evidence that tells you whether the wrong path came from forwarding, zone links or stale data.

Decide the smallest correction

The decision should describe the failing boundary and the rollback. DNS changes are deceptively small; their blast radius can be large when they affect shared suffixes.

Fix a private zone record
Record is missing or wrong in the authoritative private zone
VNet links and resolver path are correct
Change affects only the intended service name
Rollback: restore previous record value and TTL

Fix a VNet link
Record exists but consuming VNet cannot see the zone
The VNet should consume that private answer by design
No duplicate zone should be preferred instead
Rollback: remove the new VNet link

Fix a forwarding rule or conditional forwarder
Source uses a resolver that forwards the suffix to the wrong place
Inbound endpoint or target DNS server is known
UDP and TCP 53 are validated
Rollback: restore previous forwarder target or ruleset rule

Rollback the DNS change
Multiple suffixes changed at once
Answers diverge across critical sources
Ownership of the authoritative zone is unclear
Previous behavior is known and can be retested

Do not fix split-brain DNS by adding the same record everywhere. That creates a second source of truth and makes the next incident harder.

Automate a DNS evidence pack

The useful automation is a read-only evidence pack that runs from representative networks. It should not edit zones. It should tell the reviewer what answer each source sees and which resolver path produced it.

dns_evidence_pack:
fqdn: myvault.vault.azure.net
expected:
  cname_suffix: privatelink.vaultcore.azure.net
  answer_type: private_ip
sources:
  - azure_workload_subnet
  - private_runner_subnet
  - hub_tools_subnet
  - onprem_app_server
collect:
  - resolver_configuration
  - answer_with_ttl
  - cname_chain
  - private_zone_record
  - vnet_zone_links
  - ruleset_links
  - forwarder_target
block_change_when:
  - authoritative_zone_unknown
  - duplicate_private_zone_detected
  - resolver_path_not_captured
  - rollback_not_defined

This kind of pack fits well in a pipeline, AWX job or incident console. The operator still decides; the automation makes the decision reviewable.

Validate and roll back cleanly

After the correction, replay the same query matrix. Validation must include a source that was failing and a source that was already healthy, because DNS fixes can move the problem.

Validation
Same FQDN tested from every source in the incident scope
Resolver address captured for each source
CNAME chain matches expected private path
TTL is understood and caches are refreshed deliberately
Application probe succeeds from the original failing network
Monitoring probe still resolves the expected answer

Rollback
Revert the private zone record or VNet link
Restore previous forwarding rule or conditional forwarder
Flush only the caches affected by the reverted path
Replay the same query matrix
Keep before and after answers in the incident record

If the application still fails while DNS is now correct, stop changing DNS and continue with routing, NSG, firewall, TLS or identity. DNS should leave the incident with evidence, not with a pile of emergency records.

Conclusion

Azure hybrid DNS is operable when every answer can be traced to a resolver path, a private zone, a forwarding rule and a source network. A split-brain incident should therefore be handled as a path diagnosis, not as a race to add records.

The right fix may be a zone link, a forwarding rule, a cache flush, a record correction or a rollback. The decision is defensible only when the team can show the query matrix before and after the change. That is what keeps Private DNS Resolver and private zones useful in production instead of becoming another opaque shared dependency.