Azure APIM: diagnose backend TLS before disabling certificate validation

An API published through Azure API Management starts returning 500 or 502 responses while the backend still works when called directly. Its certificate was renewed a few hours earlier. The proposed fix comes quickly: disable chain or name validation in the APIM backend configuration.

That switch can restore traffic, but it removes the control that proves APIM is talking to the intended server. The running case is an internal API whose HTTPS backend uses an enterprise certificate. This runbook separates an incomplete chain, untrusted authority, wrong hostname, expired certificate, protocol mismatch and application error, then drives a decision: fix the certificate, make a bounded backend change, restore the previous certificate, or roll back APIM configuration.

Freeze one request and the change window

Start with a reproducible request that carries no sensitive data. Preserve its path, APIM operation, CorrelationId, gateway region, client response, backend response when available, and UTC timestamps. Add a healthy control request against another operation on the same backend, or the same operation in an unaffected region.

incident: INC-APIM-214
first_seen_utc: 2026-09-05T13:20:00Z
last_known_success_utc: 2026-09-05T12:48:00Z

apim:
service: apim-platform-prod
api: orders-v2
operation: GET /orders/{id}
backend_id: orders-api-prod
gateway_region: westeurope

backend:
configured_url: https://orders-api.internal.example.net
expected_hostname: orders-api.internal.example.net
recent_change: backend-certificate-renewal

evidence:
- APIM gateway log and CorrelationId
- backend entity before and after the change
- certificate leaf and presented chain
- DNS answer from an approved probe path
- backend access log for the same timestamp
- last known APIM configuration and certificate version

Do not change skipCertificateChainValidation or skipCertificateNameValidation yet. Capture their current values. An exception that is already active changes the diagnosis: APIM may reach the backend while no longer verifying part of its identity.

Prove the failure happens during the handshake

A client-side 502 is not evidence of a TLS failure. APIM can fail before it calls the backend, during connection setup, inside a policy, on timeout, or after an invalid response. Gateway logs separate those stages.

let incidentStart = datetime(2026-09-05T13:10:00Z);
let incidentEnd = datetime(2026-09-05T14:00:00Z);
ApiManagementGatewayLogs
| where TimeGenerated between (incidentStart .. incidentEnd)
| where ApiId == "orders-v2" or BackendId == "orders-api-prod"
| project TimeGenerated, CorrelationId, Region, OperationId,
        ResponseCode, BackendResponseCode, BackendTime, BackendUrl,
        LastErrorSource, LastErrorReason, LastErrorMessage
| order by TimeGenerated desc

Look for a backend connection error that begins with the renewal and compare it with server logs. If the backend has a request at the same time or with the same identifier, the handshake probably completed; inspect the HTTP response, policy and application instead. If there is no BackendResponseCode and the gateway error points to connection or certificate validation, TLS becomes the leading hypothesis.

Avoid filtering only on an exact error string. Messages can change. The combination of LastErrorSource, LastErrorReason, missing backend response, region and incident start is more durable evidence.

Read the backend configuration that is actually deployed

The effective backend can be an APIM backend entity selected by set-backend-service, a URL defined directly on the API, or a conditional policy target. Export the entity and policies at global, product, API and operation scope before drawing a conclusion.

SUBSCRIPTION_ID="<subscription-id>"
RG="rg-api-platform-prod"
APIM="apim-platform-prod"
BACKEND_ID="orders-api-prod"
API_VERSION="2024-05-01"

BACKEND_URI="https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG}/providers/Microsoft.ApiManagement/service/${APIM}/backends/${BACKEND_ID}?api-version=${API_VERSION}"

az rest --method get --url "$BACKEND_URI" --query '{id:id,url:properties.url,protocol:properties.protocol,tls:properties.tls,credentials:properties.credentials}' --output json

az apim api policy show --resource-group "$RG" --service-name "$APIM" --api-id "orders-v2" --output json

Check the complete URL, not only the expected certificate. A backend configured with an IP address, different DNS alias or retired hostname can receive a valid certificate whose name does not match. Also verify that a policy does not send selected operations to another backend entity.

Separate name, chain, validity and protocol

Treat the handshake as four independent checks:

• the requested name must match a SAN in the presented certificate;
• the presented chain must lead to an authority trusted by the gateway;
• every required certificate must be within its validity period and suitable for server use;
• APIM and the backend must share compatible TLS protocol settings.

From an approved diagnostic host with a comparable network path, inspect DNS and the certificate presented for the real hostname. This does not reproduce the managed APIM runtime, but it quickly exposes a wrong endpoint, missing SAN or incomplete chain.

HOST="orders-api.internal.example.net"
PORT="443"

getent ahosts "$HOST"

openssl s_client -connect "${HOST}:${PORT}" -servername "$HOST" -showcerts -verify_return_error </dev/null

openssl s_client -connect "${HOST}:${PORT}" -servername "$HOST" </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -serial -dates -ext subjectAltName

The -servername argument matters. Without SNI, a reverse proxy can present its default certificate and create a false diagnosis. If the certificate uses private trust, repeat the check with the expected CA bundle. A correct leaf does not compensate for a missing intermediate; the server generally needs to present the required intermediates.

Correlate renewal with the certificate being served

A certificate can be renewed in Key Vault, a load balancer or a server without becoming the certificate that APIM sees. Compare the fingerprint and serial presented by every endpoint or region with the intended object. Check cutover time, propagation and backend pool behavior as well.

Use a short matrix:

Expected certificate
hostname and SAN
issuer and chain
serial or fingerprint
notBefore / notAfter
deployment target and version

Observed from approved probe
resolved IP
hostname sent as SNI
leaf serial or fingerprint
presented intermediates
verify return code

Observed by APIM
gateway region
backend URL and BackendId
first failure UTC
LastErrorReason and LastErrorMessage
backend response present: yes/no

If one APIM region fails, do not immediately call it a global trust problem. Check DNS and the certificate served by that region’s backends. If every region fails immediately after an APIM change, compare the backend entity and policies first.

Choose the narrowest correction

The failing control determines the action:

Hostname missing from SAN
Correct the backend URL or issue a certificate for the real hostname
Do not disable name validation to hide a poorly designed alias

Incomplete public chain
Configure the server to present the required intermediates
Revalidate from multiple paths before restoring traffic

Expected private authority
Configure the custom CA supported by the actual APIM tier and gateway type
Prove the trust scope and preserve name validation

Expired certificate or wrong version served
Restore the previous valid certificate or complete renewal
Verify every endpoint before retiring the old version

Incompatible TLS setting
Restore the last known combination or correct the backend
Test certificate-authenticated clients separately before enabling TLS 1.3

HTTP error after handshake
Stop changing certificates
Diagnose policy, identity, timeout and application behavior

Custom CA mechanics differ across managed gateways, self-hosted gateways and APIM tiers. Check support for the tier in use before preparing the change. Adding a CA to a managed service does not prove that the same trust exists on a self-hosted gateway.

Bound any temporary exception

Disabling validation is acceptable only when the architecture explicitly relies on a self-signed certificate, the tier requires it, and the risk is understood. Scope it to one backend entity, preserve as many checks as possible, and assign an expiry, owner and rollback. Do not disable both chain and name checks for convenience.

Before the exception, preserve the complete backend object. After the change, replay a positive and a negative request: the legitimate hostname must work, while a neighboring backend or unexpected name must not become a valid target. Never use the exception to compensate for a broken public chain or expired certificate.

Validate rollout and rollback

Deploy first to a canary API or revision that follows the same path. Correlate the request end to end, then watch TLS errors and backend responses by region before widening the rollout.

Before rollout
BackendId, URL and policies exported
Expected and served certificates match
SAN, chain, dates and protocol verified
Cause visible in APIM logs
Positive request and negative test defined
Rollback object ready

Success
APIM request returns the expected application response
Backend receives the same CorrelationId
No new TLS error group appears by region
Chain and name validation remain enabled unless explicitly approved

Immediate rollback
One region still serves an unexpected certificate
Negative test becomes reachable
Effective backend differs from the validated object
TLS exception affects more APIs than intended
Errors change shape without proof of a successful handshake

Rollback means restoring the known backend entity, policy or certificate version and replaying the same evidence. Disabling a whole API may contain risk, but it is not the rollback of the TLS change.

Conclusion

An APIM error after certificate renewal does not justify removing a TLS control. First prove which stage fails, read the effective backend, test the hostname with SNI, rebuild the chain and correlate the certificate actually served with gateway logs.

The operational decision then becomes straightforward: fix the SAN or chain, configure a supported private CA, restore the previous certificate, revert a TLS setting, or address an application error that was never TLS. Production is ready when the positive call succeeds, the negative test stays blocked and certificate validation remains explainable.