Cloud

Azure APIM: validate a Named Values rotation before breaking a private API

A production runbook for qualifying an APIM secret rotation with Named Values, Key Vault, managed identity, private backend, logs, application validation and rollback.

02 Jul 2026 azureapimapi-managementkey-vaultnamed-valuesmanaged-identityprivate-apiobservabilitykqlrunbookrollbacksecurityautomation

An APIM secret rotation often looks harmless: change a Named Value, update a Key Vault reference, renew a backend key or replace a certificate. In production, that change can break a private API even when the network path, backend and APIM policy did not visibly change. The symptom reaches callers as 401, 403, 500, 502 or timeout, and the team may start chasing WAF, Private Endpoint or application code too early.

The use case is an API exposed through internal Azure API Management, sometimes behind Application Gateway WAF, with a private Azure Function, App Service or internal API as backend. APIM uses Named Values to inject a key, header, URL, client secret or Key Vault reference into policies. This runbook helps decide whether the rotation is ready, whether APIM can still read Key Vault, whether the policy consumes the old value or whether the safest action is rollback without opening the backend or weakening controls.

Scope the value that really changes

Start by describing the rotation as an application dependency change, not as a simple secret update. One APIM Named Value can be consumed by several APIs, policies and environments.

text apim-rotation-scope.txt

Requested rotation
APIM: apim-prod-internal
API: orders-private-api
Named Value: backend-orders-api-key
Source: Key Vault reference or protected APIM value
Backend: https://func-orders-prod.azurewebsites.net/api
Path: private client -> Application Gateway WAF -> internal APIM -> private backend

Questions before change
Which APIs and policies read this Named Value?
Is the value stored in APIM or referenced from Key Vault?
Which APIM identity reads the secret?
Does the backend accept both old and new values during the window?
Which test proves the API works without opening the backend?

This prevents a common mistake: fixing the private API path while the failing component is a shared configuration dependency used by multiple policies.

Inventory APIM usage

Before rotation, identify every policy that references the Named Value. The risk is not only breaking the target API. It is also changing a shared fragment that another API depends on.

bash 01-apim-named-value-inventory.sh

APIM_RG="rg-apim-prod"
APIM_NAME="apim-prod-internal"
NAMED_VALUE="backend-orders-api-key"

az apim nv show --resource-group "$APIM_RG" --service-name "$APIM_NAME" --named-value-id "$NAMED_VALUE" --query "{name:name,displayName:displayName,secret:secret,keyVault:keyVault,resourceId:id}" --output json

az apim api list --resource-group "$APIM_RG" --service-name "$APIM_NAME" --query "[].{name:name,path:path,apiRevision:apiRevision,isCurrent:isCurrent}" --output table

Complete the check with a repository search when policies are versioned. A Named Value can appear as {{backend-orders-api-key}} in inbound, backend or outbound policy sections, or in a shared fragment.

bash 02-policy-reference-search.sh

rg "\{\{backend-orders-api-key\}\}" apim/policies apim/fragments infrastructure || true
rg "set-header|set-query-parameter|set-backend-service|authentication-managed-identity" apim/policies || true

If policies are not versioned, the rotation needs a stricter change record: export the APIM state before the change and write down the exact rollback.

Verify the identity reading Key Vault

When the Named Value points to Key Vault, the failure can come from the value, the secret version, the Key Vault network boundary or the APIM managed identity. Do not change the policy until that path is qualified.

bash 03-key-vault-reference-check.sh

APIM_RG="rg-apim-prod"
APIM_NAME="apim-prod-internal"
KV_NAME="kv-prod-shared"
SECRET_NAME="backend-orders-api-key"

az apim show --resource-group "$APIM_RG" --name "$APIM_NAME" --query "{name:name,identity:identity}" --output json

az keyvault secret show --vault-name "$KV_NAME" --name "$SECRET_NAME" --query "{id:id,enabled:attributes.enabled,expires:attributes.expires,updated:attributes.updated,contentType:contentType}" --output json

az keyvault show --name "$KV_NAME" --query "{name:name,networkAcls:properties.networkAcls,publicNetworkAccess:properties.publicNetworkAccess}" --output json

If Key Vault is private, also validate DNS resolution and the network path from the APIM perspective when the architecture allows it. A secret rotation should not become a temporary public opening for Key Vault.

Read logs as an evidence chain

APIM errors must be tied to a request, a policy, a backend and a change window. HTTP status codes alone are not enough.

kusto 04-apim-rotation-evidence.kql

let ApiName = "orders-private-api";
let WindowStart = ago(2h);
ApiManagementGatewayLogs
| where TimeGenerated > WindowStart
| where ApiId has ApiName or OperationId has ApiName
| project TimeGenerated,
        RequestId,
        ApiId,
        OperationId,
        Method,
        Url,
        ResponseCode,
        BackendResponseCode,
        LastErrorReason,
        LastErrorMessage,
        BackendUrl,
        CallerIpAddress
| order by TimeGenerated desc

Add Key Vault events when the value is referenced as a secret.

kusto 05-key-vault-secret-access.kql

let SecretName = "backend-orders-api-key";
let WindowStart = ago(2h);
AzureDiagnostics
| where TimeGenerated > WindowStart
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName has "SecretGet"
| where id_s has SecretName or requestUri_s has SecretName
| project TimeGenerated,
        OperationName,
        ResultType,
        ResultSignature,
        identity_claim_appid_g,
        identity_claim_oid_g,
        CallerIPAddress,
        requestUri_s
| order by TimeGenerated desc

The evidence should show whether APIM reads the secret, Key Vault denies access, APIM executes a policy with a missing value, or the backend rejects the rotated value.

Prepare a production-compatible rotation

The strongest rotation avoids a hard cutover. When the backend supports it, keep a window where both old and new values are accepted. Otherwise, prepare an explicit APIM rollback and an immediate backend test.

yaml apim-secret-rotation-guardrails.yml

before_rotation:
- export_apim_named_value_and_policy_state
- verify_named_value_references
- verify_apim_managed_identity_can_read_key_vault
- verify_backend_accepts_new_secret_or_has_rollback_window
- run_private_api_probe_before_change

block_rotation_when:
- named_value_used_by_unknown_policy
- key_vault_secret_disabled_or_expired
- apim_identity_not_visible_in_key_vault_logs
- backend_has_no_validation_endpoint
- rollback_value_not_captured

after_rotation:
- force_policy_refresh_or_wait_for_expected_cache_window
- run_private_api_probe_from_client_network
- compare_apim_gateway_logs_before_after
- confirm_key_vault_secret_get_success
- attach_evidence_to_change_ticket

The key point is consumption evidence. Updating Key Vault does not prove that APIM consumes the expected version at the expected time.

Validate from the private path

Validation must start from a representative client: application subnet, private runner, synthetic probe or operations host using the same private path as consumers.

bash 06-private-api-probe.sh

APIM_HOST="api.internal.example"
API_PATH="/orders/health"
CORRELATION_ID="rotation-$(date +%Y%m%d%H%M%S)"

curl -sS -o /tmp/apim-health.out -w "%{http_code}\n" "https://$APIM_HOST$API_PATH" -H "x-correlation-id: $CORRELATION_ID" -H "Accept: application/json"

echo "correlation_id=$CORRELATION_ID"
cat /tmp/apim-health.out

Then search for the same identifier in APIM and backend logs. A valid test should show APIM gateway handling, the expected policy path, the private backend and a consistent application response.

Decide: deploy, fix or roll back

Keep the decision operational. Do not mix secret, policy, network and backend corrections into one change.

text apim-rotation-decision.txt

Deploy
The target Named Value is identified
Consuming policies are known
APIM reads Key Vault with its managed identity
The backend accepts the rotated value
The private probe succeeds and logs confirm the path

Fix before rotation
The Named Value is shared by an unexpected API
Key Vault denies the APIM identity
The policy reads a different value than documented
The backend returns 401 or 403 with the new value

Rollback
APIM errors increase after rotation
Key Vault reads fail or become intermittent
The private backend does not accept the new value
The previous value is known and retestable
Recovery can be validated from the same private path

Rollback is not only restoring the old value. It must prove that the private API responds again, Key Vault errors stop and APIM logs return to the expected behavior.

Conclusion

An APIM secret rotation is a full production change: Named Value, policy, managed identity, Key Vault, private backend, logs and real client path. Treating it as a simple configuration update pushes teams toward the wrong network diagnosis or overly broad exceptions.

The decision should be simple: deploy when the consumption chain is proven, fix when the gap is localized, roll back when the value or identity is no longer controlled. That is how teams rotate secrets without turning a private API into an opaque APIM incident.