Azure APIM self-hosted gateway: diagnose sync drift before rerouting traffic

An Azure API Management self-hosted gateway can keep serving traffic while no longer matching the APIM service that manages it. That is the operational trap: requests still flow, but a new policy is not enforced, an API is missing, a route returns an old response, or a secondary cluster looks ready while serving stale configuration.

The use case is a central APIM instance in Azure with self-hosted gateways deployed on Kubernetes, close to internal or hybrid backends. The team wants to move part of the traffic to another cluster after a network incident or during maintenance. Before changing DNS, Front Door, Application Gateway or an internal load balancer, the runbook must prove that the target gateway is synchronized, authenticated, observable and able to reach the expected backends.

Freeze the risk before shifting traffic

Start by writing what the traffic shift is supposed to change. A routing change is not neutral when the target gateway does not carry the same configuration as the active gateway.

change:
apim_service: apim-prod-core
gateway_active: shgw-aks-west
gateway_target: shgw-aks-east
api: payments-api
operation: POST /payments/authorize
traffic_shift: 25_percent_canary
reason: regional maintenance on the active Kubernetes cluster

evidence_required:
- target gateway pods ready
- configuration downloaded after the last APIM policy change
- gateway authentication still valid
- backend route works from the target cluster
- logs or local traces available for the canary
- rollback restores previous traffic path

This note avoids a shallow decision: a Ready pod does not prove that the gateway applies the right configuration or can reach the real backend.

Separate control plane, runtime and backend

The diagnosis must split three planes. The control plane synchronizes configuration from Azure API Management. The runtime receives requests and enforces policies. The backend plane reaches the services exposed behind the gateway.

Control plane
The gateway must authenticate to APIM and download configuration
Risks: expired token, broken Workload Identity binding, blocked DNS or outbound 443, old cached configuration

Gateway runtime
Pods must receive traffic and enforce policies, products, subscriptions and transformations
Risks: different version, unready replica, missing secret, inconsistent local configuration

Backend plane
The gateway must reach the backend with the right hostname, TLS, route, identity and timeout
Risks: cluster DNS, NetworkPolicy, firewall, certificate, UDR, NAT or source allowlist

An application symptom can come from any of these planes. Rerouting traffic before separating them can move the incident instead of resolving it.

Check the real Kubernetes state

The first check is local to the cluster. It should confirm the deployed version, restarts, mounted secrets, probes and latest synchronization logs.

NAMESPACE="apim-gateway"
APP_LABEL="app.kubernetes.io/name=azure-api-management-gateway"

kubectl get pods -n "$NAMESPACE" -l "$APP_LABEL" -o wide
kubectl describe deployment -n "$NAMESPACE" -l "$APP_LABEL"
kubectl get events -n "$NAMESPACE" --sort-by=.lastTimestamp | tail -40

kubectl logs -n "$NAMESPACE" -l "$APP_LABEL" --tail=200 | egrep -i "config|sync|token|auth|error|warning|backend|certificate"

A recent restart may be normal after token rotation or a Helm upgrade. It becomes suspicious when the gateway starts with an old configuration, repeats authentication errors or no longer downloads APIM updates.

Qualify gateway authentication

The self-hosted gateway must authenticate to the associated APIM instance to retrieve configuration. Depending on the deployment, the team may use a gateway token, Microsoft Entra ID or Workload Identity on AKS. The risk is not only expiration: it is also a rotation that did not reach every pod, a secret mounted in the wrong namespace or a federated identity binding that no longer matches the service account.

Token or identity check
The expected authentication method is documented
The secret or service account used by pods is the production one
Recent rotation has reached every replica
Logs contain no auth denial or expired token
The last pod start is later than the secret change when rotation occurred

Hold traffic shift when
One pod still uses the old token
The target gateway no longer downloads configuration
Events show a secret mount error
Federated identity does not match the namespace or service account

When authentication is doubtful, rollback is not a routing tweak. Remove the gateway from the target path, fix the secret or identity, then wait for proven synchronization.

Prove configuration sync

The key point is linking the configuration served by the gateway to the last relevant APIM change: policy, API, product, subscription, backend or certificate. A gateway can survive temporary loss of Azure connectivity by serving local configuration, but that behavior must not be confused with healthy synchronization.

GATEWAY_URL="https://shgw-east.internal.example.com"
API_PATH="/payments/authorize"
CORRELATION_ID="diag-$(date +%Y%m%d%H%M%S)"

curl -sk -D - -H "x-correlation-id: $CORRELATION_ID" -H "Ocp-Apim-Subscription-Key: $SUBSCRIPTION_KEY" "$GATEWAY_URL$API_PATH?diagnostic=true"

echo "Correlation: $CORRELATION_ID"

The evidence request must cross the same API, policy and backend as the traffic you plan to shift. A generic health endpoint only proves that the container answers.

Check the backend path from the target gateway

A synchronized gateway can still fail when the target cluster cannot resolve the backend, exits through a non-allowed IP address or uses a different root certificate. Before rerouting, run checks from the same namespace and, where possible, from a diagnostic image subject to the same NetworkPolicies.

NAMESPACE="apim-gateway"
BACKEND_HOST="payments.internal.example.com"
BACKEND_URL="https://payments.internal.example.com/health"

kubectl run apim-path-check -n "$NAMESPACE" --rm -i --restart=Never --image=curlimages/curl:latest -- sh -c "nslookup $BACKEND_HOST && curl -sk -w '
http_code=%{http_code}
remote_ip=%{remote_ip}
' $BACKEND_URL"

If this test fails, avoid changing APIM policies. The likely cause is DNS, routing, firewall, backend TLS, NetworkPolicy or source allowlisting.

Read the available traces

A self-hosted gateway is not diagnosed exactly like the managed APIM gateway. Depending on configuration, Azure Monitor resource logs may not cover the full self-hosted runtime. Plan for local logs, Kubernetes collection, cloud metrics when enabled, and correlation IDs propagated to the backend.

let CorrelationId = "diag-20260726143000";
AppRequests
| where TimeGenerated > ago(30m)
| where tostring(CustomDimensions["x-correlation-id"]) == CorrelationId
| project TimeGenerated, AppRoleName, Name, ResultCode, DurationMs, OperationId, CloudRoleInstance
| order by TimeGenerated asc

No backend log for the correlation probably means the request did not reach the service. A correlated 401, 403, 502 or 504 gives a more precise correction layer.

Decide reroute, fix or rollback

Keep the decision bounded. A target gateway is not safe to expose only because its cluster is available.

Allow canary
Stable pods and expected version
Valid gateway authentication
Configuration synchronized after the last APIM change
Evidence request on the real API succeeds
Backend reachable from the target cluster
Local logs or backend correlation usable
Routing rollback tested

Fix before rerouting
Expired token or unpropagated secret
Workload Identity not bound to the right service account
DNS or outbound 443 to APIM blocked
Backend resolves differently from the target cluster
Logs too weak to qualify the canary

Rollback
The canary serves an old policy
Errors increase without usable traces
The target gateway loses synchronization during the window
The backend denies the new network source
Returning to the previous path restores the expected response

The clean rollback is to restore the previous traffic weight, preserve canary correlations, then fix the gateway outside the user path. Do not blindly regenerate every token when only one cluster is drifting.

Validate after recovery

After the fix or rollback, replay the same evidence request, verify synchronization after a controlled APIM change and document the propagation time. That measure becomes runbook data for the next shift.

Keep for the runbook
Last APIM change before the incident
Time when sync drift was detected
Authentication method used by the gateway
Exact cause: token, identity, outbound 443, DNS, backend or logs
Propagation time observed after correction
Validated rollback command or procedure
Signal to add to monitoring

Conclusion

An APIM self-hosted gateway is a distributed execution boundary. It can be healthy from a Kubernetes point of view and still be unsafe to expose if configuration, authentication or backend reachability is not proven.

The practical reflex before rerouting is straightforward: check the pod, prove authentication, link configuration to the last APIM change, test the real backend from the target cluster, keep readable correlation and prepare traffic rollback. The shift becomes a production decision, not a bet on an endpoint that happens to answer.