Azure APIM: diagnose a subscription key before rotating it in production

Incident
Hostname: api.internal.example.com
API: orders-api
Product: partner-standard
Subscription: partner-fulfillment-prod
Consumer: partner-orders-worker
Expected header: Ocp-Apim-Subscription-Key
Symptom: 401 Access denied or 403 Forbidden
Window: after planned rotation or emergency regeneration
Backend: healthy on controlled probe

Questions before action
Which key is in use: primary or secondary?
Is the subscription active?
Does the product still include the called API?
Has the consumer deployed the new value?
Does a policy change the header name or require JWT, mTLS or IP allowlist?
Does rollback restore a key or only application configuration?

Layered reading
APIM 401
  Missing key, wrong key, wrong header name, inactive subscription or API outside the product

APIM 403
  Valid subscription but policy denial, closed product, quota, rate limit, IP restriction or JWT/mTLS validation

Backend 401 or 403
  APIM accepted the request, but the backend rejects identity, token, application key or source address

WAF or gateway 403
  The request may never reach APIM: check Application Gateway or Front Door logs before APIM

No APIM log
  Wrong hostname, route, DNS, listener, certificate, WAF or client path

RG="rg-shared-api-prod"
APIM="apim-prod-core"
SUBSCRIPTION_ID="partner-fulfillment-prod"
PRODUCT_ID="partner-standard"
API_ID="orders-api"

az apim subscription show --resource-group "$RG" --service-name "$APIM" --sid "$SUBSCRIPTION_ID" --query "{state:state,scope:scope,ownerId:ownerId,displayName:displayName,createdDate:createdDate,expirationDate:expirationDate}" --output json

az apim product api list --resource-group "$RG" --service-name "$APIM" --product-id "$PRODUCT_ID" --query "[].{apiId:name,displayName:displayName,path:path}" --output table

az apim api show --resource-group "$RG" --service-name "$APIM" --api-id "$API_ID" --query "{path:path,protocols:protocols,subscriptionRequired:subscriptionRequired}" --output json

let Window = 2h;
let ApiPath = "/orders";
AzureDiagnostics
| where TimeGenerated > ago(Window)
| where ResourceProvider == "MICROSOFT.APIMANAGEMENT"
| where tostring(Url) has ApiPath or tostring(RequestUri) has ApiPath
| project TimeGenerated,
        ServiceName=tostring(Resource),
        ApiId=tostring(ApiId),
        OperationId=tostring(OperationId),
        ProductId=tostring(ProductId),
        SubscriptionId=tostring(SubscriptionId),
        CallerIp=tostring(CallerIPAddress),
        Status=toint(ResponseCode),
        Error=tostring(ErrorMessage),
        CorrelationId=tostring(CorrelationId),
        Url=tostring(Url)
| where Status in (401, 403, 429, 500)
| order by TimeGenerated desc

Controlled rotation sequence
1. Identify the key currently used by the consumer
2. Distribute the secondary key if primary is active, or the opposite
3. Replay a controlled request with x-correlation-id
4. Verify APIM logs: subscriptionId, productId, apiId, 2xx status or expected error
5. Move the consumer to the new value
6. Wait through the agreed observation window
7. Regenerate only the old key
8. Keep documented application rollback until validation is complete

Forbidden
Regenerating primary and secondary in the same window
Rotating a key without knowing which consumer uses it
Fixing a backend 403 by regenerating the subscription key
Removing the old value before APIM and application evidence exists

HOST="api.internal.example.com"
PATH_TO_TEST="/orders/health"
CORRELATION_ID="apim-key-$(date +%Y%m%d%H%M%S)"
SUBSCRIPTION_KEY="<candidate-key>"

curl -sS -D - "https://${HOST}${PATH_TO_TEST}" -H "Ocp-Apim-Subscription-Key: ${SUBSCRIPTION_KEY}" -H "x-correlation-id: ${CORRELATION_ID}" -o /tmp/apim-response.json

echo "correlation_id=${CORRELATION_ID}"
cat /tmp/apim-response.json

Rotate the key
The subscription is active
The product still exposes the API
The consumer validated the replacement key
APIM logs prove the expected subscriptionId
Application rollback is ready

Fix the consumer
Wrong header or old value
Wrong Key Vault or App Configuration variable
Partial deployment on only one worker
Application cache not refreshed

Fix APIM
API was removed from the product by mistake
Subscription is disabled or expired
Policy changed the authentication contract
Quota or rate limit blocks a legitimate consumer

Rollback or block
Both keys were invalidated
Real consumers cannot be identified
Logs do not separate APIM and backend
Rotation hides an unqualified security incident

apim_subscription_rotation:
api: orders-api
product: partner-standard
subscription: partner-fulfillment-prod
consumer: partner-orders-worker
active_key_after_change: secondary
old_key_regenerated: primary
evidence:
  - apim_logs_show_expected_subscription_id
  - controlled_replay_correlation_id_attached
  - consumer_deployment_version_identified
  - backend_errors_checked_separately
rollback:
  restore_consumer_secret_version: kv://kv-prod/api/apim-key/previous
  do_not_regenerate_secondary_until: 2026-07-15T18:00:00Z
  owner: platform-operations
follow_up:
  - document consumers for the subscription
  - alert on APIM 401/403 by product and subscription
  - avoid shared subscriptions for unrelated consumers