Azure API Management: validate a shared policy fragment before production rollout

A team needs to harden authentication across several APIs behind Azure API Management. The logic already lives in a shared policy fragment: validate the token, normalize selected headers and pass identity context to the backend. Editing that fragment in place looks cleaner than changing every API. It is also the fastest way to distribute one mistake to every consumer.

XML validation is only part of the problem. A fragment is inserted as-is wherever a policy references it. Its behavior therefore depends on the policy section, statement order, variables already initialized, named values and the gateway executing the policy. This runbook ends with an operational decision: promote a new fragment version, restrict its scope, or restore the previous reference without rewriting the entire policy estate.

Freeze the contract before editing the fragment

Describe what the change must achieve and what it must never alter. The XML diff is only an implementation of that contract.

change:
id: CHG-2941
fragment_current: auth-context-v1
fragment_candidate: auth-context-v2
intent: reject expired tokens and preserve backend context

must_preserve:
- valid-client-request-is-forwarded
- backend-receives-correlation-id
- existing-product-subscriptions-still-work
- error-body-does-not-leak-token-details

must_reject:
- expired-token
- wrong-audience
- missing-required-claim

rollback:
reference: auth-context-v1
owner: api-platform
trigger: unexplained-401-403-5xx-or-latency-regression

Freeze the exact scope too: APIM instance, workspace if applicable, products, APIs, operations and gateways. A passing test on one API says little about another policy with a different ordering of <base />, fragments or variables.

Inventory the actual consumers

The list of teams believed to use the fragment is not an inventory. Read references from deployed policies, then map each reference to its scope and production traffic.

APIM_ID="/subscriptions/<subscription>/resourceGroups/rg-api-prod/providers/Microsoft.ApiManagement/service/apim-prod"
API_VERSION="<supported-management-api-version>"

az rest --method get \
--url "https://management.azure.com/$APIM_ID/policyFragments?api-version=$API_VERSION" \
--output json > policy-fragments-before.json

# Export global, product, API and operation policies from IaC or the
# management API, then locate the exact reference.
rg -n 'include-fragment fragment-id="auth-context-v1"' exported-policies/ \
> auth-context-v1-references.txt

test -s auth-context-v1-references.txt

Retain the fragment content, references, named values and their scopes, each API’s current revision, and the expected IaC commit. Do not assume indirect references: a fragment cannot nest another fragment, and it cannot contain a policy section or <base /> element.

Version instead of editing in place

Updating a shared fragment affects its existing references. For a sensitive change, create a new identifier such as auth-context-v2, keep it immutable during the canary, and update one test policy to reference it. Temporary duplication is a rollout control, not permanent architecture.

Structure
Valid XML containing one or more policy statements
No inbound, backend, outbound or on-error section
No base element
No nested include-fragment

Dependencies
Named values exist at the required scope
Variables are read after initialization
Secrets never appear in traces or responses
Statements support the target section and gateway

Change
New fragment ID used for the canary
Diff reviewed as behavior, not only XML
v1 reference retained for rollback
No unrelated policy change in the same batch

Execution order is the sharp edge. Moving validation before variable initialization, placing a transformation after backend routing, or changing its position relative to <base /> can alter behavior while leaving the document syntactically valid.

Build a representative canary revision

Create a non-current revision of a representative API and replace only the v1 reference with v2. Keep the same parameters, backend, product and test subscription. An explicit ;rev=<number> URL lets the team call the candidate without moving normal traffic.

The canary needs a happy path, expected denials and a backend non-regression control.

BASE_URL="https://api.example.net/orders"
REVISION="<candidate-revision>"

# Control: current revision.
curl --fail-with-body --silent --show-error \
-H "Authorization: Bearer <valid-test-token>" \
-H "x-correlation-id: apim-fragment-current-001" \
"$BASE_URL/health"

# Candidate: explicitly address the new revision.
curl --fail-with-body --silent --show-error \
-H "Authorization: Bearer <valid-test-token>" \
-H "x-correlation-id: apim-fragment-v2-001" \
"https://api.example.net/orders;rev=$REVISION/health"

# Expected denial: verify status and a non-sensitive error body.
curl --silent --show-error --output candidate-deny.json --write-out '%{http_code}
' \
-H "Authorization: Bearer <expired-test-token>" \
-H "x-correlation-id: apim-fragment-v2-deny-001" \
"https://api.example.net/orders;rev=$REVISION/health"

Do not reuse a production token captured after expiry. Prepare test identities and tokens whose claims represent the expected scenarios. Check the status, body, headers, backend call and absence of sensitive output.

Read traces as chain evidence

An expected 401 is not sufficient: it may come from the fragment, another policy or the backend. Enable approved diagnostics on the canary and correlate each probe by ID. Adapt the fields to the table produced by the APIM diagnostic mode in use.

ApiManagementGatewayLogs
| where TimeGenerated > ago(30m)
| where CorrelationId startswith "apim-fragment-v2-"
| project TimeGenerated,
        CorrelationId,
        ApiId,
        OperationId,
        ResponseCode,
        BackendResponseCode,
        TotalTime,
        BackendTime,
        LastErrorReason
| order by TimeGenerated asc

The evidence must connect request, policy decision and backend. A valid token produces one backend call with the expected context. An invalid token produces no backend call. If latency rises, separate gateway time from backend time before blaming the new validation.

Test the blast radius before promotion

The first canary proves the logic, not compatibility with every consumer. Group references by policy shape: global or product scope, include section, <base /> placement, existing variables, gateway type and named-value dependencies. Test one representative from every class.

Class A: standard API on managed gateway
Happy path, invalid token, missing claim, backend timeout

Class B: product policy with base and quotas
Valid auth, subscription key, quota preserved, stable error

Class C: self-hosted gateway or workspace
Statement support, named-value access, diagnostics available

Cross-cutting controls
No secret in logs or responses
Correlation ID preserved
401, 403 and 5xx rates compared with control
Gateway latency compared under representative load
v1 reference remains deployable

If one class cannot be tested, do not generalize the rollout. Keep v2 limited to proven scopes or remove the fragment’s implicit dependency.

Promote without losing rollback

Promotion changes references in bounded batches toward one frozen candidate. Do not edit v2 while APIs are migrating. A moving target makes results impossible to compare.

promote:
when:
- every-policy-class-has-a-passing-canary
- valid-and-denied-paths-are-correlated
- gateway-latency-remains-within-change-budget
- backend-context-is-preserved
action: migrate-references-in-bounded-batches

hold:
when:
- consumer-inventory-is-incomplete
- named-value-scope-is-ambiguous
- one-gateway-class-is-untested
action: keep-v1-current-and-fix-evidence

rollback:
when:
- unexplained-authentication-regression
- backend-call-occurs-after-expected-deny
- sensitive-data-appears-in-output-or-traces
- gateway-error-or-latency-budget-is-breached
action:
- restore-v1-reference-for-last-batch
- redeploy-policy-only
- replay-positive-and-negative-probes
- reconcile-rejected-or-duplicated-requests

Rollback is not an emergency rewrite of v2. Restore the known v1 reference, redeploy only the policies in the affected batch, and replay the same probes. Keep v2 for analysis, then produce a corrected v3 if necessary.

Conclusion

An APIM policy fragment removes duplication but concentrates risk. The change unit is not the XML file alone. It includes the fragment, its references, named values, execution order, gateways and the requests that prove its behavior.

The final decision is straightforward to explain. Promote a frozen version after one canary per policy class, hold when inventory or traces are incomplete, and roll back by restoring the previous reference. Reuse remains an advantage only when its blast radius is observable and reversible.