Azure WAF: diagnose inconsistent policy enforcement after deployment

An Azure WAF policy has just been deployed to Application Gateway. The platform team’s test succeeds, yet some users still receive 403. Another route appears to ignore the new rule. The rollout is quickly labelled inconsistent, and reverting the entire policy starts to look safer than leaving production in an unknown state.

That conclusion often arrives too early. Requests may select different listeners, path rules or gateways. A higher-priority custom rule may terminate evaluation before the managed rule being inspected. ARM may also report a complete deployment while tests vary by hostname, path or payload. This runbook drives one explicit decision: wait within a bounded window, repair one association or rule, restore the previous policy, or keep the block because it is working as intended.

Freeze the change and one symptom pair

Capture one accepted request and one blocked request before changing the policy again. Preserve UTC time, hostname, path, method, source IP, correlation identifier and application outcome. Without that pair, “intermittent” may combine traffic that never crossed the same configuration.

change:
gateway: agw-api-prod
policy: waf-api-prod
intended_policy_version: git-8d2c1f4
deployment_operation: dep-waf-20261003-1605
started_utc: 2026-10-03T16:05:00Z
expected_scope:
  hostnames: [api.example.com]
  paths: [/partner/*]

symptom_pair:
accepted:
  utc: 2026-10-03T16:18:11Z
  method: POST
  path: /partner/orders
  correlation_id: waf-canary-accepted-01
blocked:
  utc: 2026-10-03T16:18:43Z
  method: POST
  path: /partner/orders
  correlation_id: waf-canary-blocked-01

decision_deadline_utc: 2026-10-03T16:35:00Z
rollback_artifact: last-known-good-policy-json

Keep the approved IaC diff and the last healthy policy export as well. Rollback should restore a known object, not rebuild an old rule manually during the incident.

Prove ARM state before choosing to wait

A successful pipeline step does not prove that Azure finished updating the resources. Read both the policy and gateway from Azure, then connect their state to the change operation.

RG="rg-edge-prod"
GATEWAY="agw-api-prod"
POLICY="waf-api-prod"

POLICY_ID=$(az network application-gateway waf-policy show --resource-group "$RG" --name "$POLICY" --query id -o tsv)

az network application-gateway waf-policy show --resource-group "$RG" --name "$POLICY" --query '{id:id,state:provisioningState,settings:policySettings,customRules:customRules[].{name:name,priority:priority,state:state,action:action}}' --output jsonc

az network application-gateway show --resource-group "$RG" --name "$GATEWAY" --query '{state:provisioningState,policy:firewallPolicy.id}' --output jsonc

az monitor activity-log list --resource-id "$POLICY_ID" --start-time "2026-10-03T15:45:00Z" --end-time "2026-10-03T16:35:00Z" --query '[].{time:eventTimestamp,status:status.localizedValue,operation:operationName.localizedValue,correlationId:correlationId}' --output table

When provisioningState is not Succeeded, do not stack a second modification onto the first. Bound the observation window, follow the operation and prepare rollback. When Azure does report Succeeded, continuing to invoke propagation without comparing associations and requests leaves the incident without evidence.

Inventory the associations that can affect the request

A policy can be associated at gateway, listener or path-rule scope. A more specific attachment changes which traffic receives which controls. List every attachment and compare its full resource ID with the approved architecture.

az network application-gateway show --resource-group "$RG" --name "$GATEWAY" --query '{
  gatewayPolicy:firewallPolicy.id,
  listeners:httpListeners[].{name:name,hostNames:hostNames,policy:firewallPolicy.id},
  pathMaps:urlPathMaps[].{
    name:name,
    rules:pathRules[].{name:name,paths:paths,policy:firewallPolicy.id}
  }
}' --output jsonc

Then inspect gateway routing: selected listener, hostnames, URL path map, path-rule order and backend. A probe sent to an IP with the wrong Host header is not representative of real traffic. Two gateways behind Traffic Manager or Front Door may also reference different policies or have separate rollout operations.

Do not flatten every association merely to make the output uniform. The hierarchy may be deliberate: a gateway baseline, a stricter administrative listener and a bounded path override. Repair only the difference between the approved design and deployed state.

Compare deployed content with the approved commit

A stable policy name does not imply stable contents. Export the resource, remove volatile fields and compare custom rules, priorities, actions, conditions, exclusions and managed rule versions with the approved artifact.

az network application-gateway waf-policy show --resource-group "$RG" --name "$POLICY" --query '{
  mode:policySettings.mode,
  state:policySettings.state,
  requestBodyCheck:policySettings.requestBodyCheck,
  maxRequestBodySizeInKb:policySettings.maxRequestBodySizeInKb,
  customRules:customRules,
  managedRules:managedRules
}' --output json > waf-policy-effective.json

jq -S . waf-policy-effective.json > waf-policy-effective.sorted.json
jq -S . waf-policy-approved.json > waf-policy-approved.sorted.json
diff -u waf-policy-approved.sorted.json waf-policy-effective.sorted.json

A numerically lower custom-rule priority runs earlier. A terminating action can therefore prevent the request from reaching the rule the team expects to observe. Check disabled rules, match variables, transformations, negations and exclusions too. One small difference can look like partial propagation from the client side.

Use WAF events as decision evidence

WAF logs prove that a rule evaluated a request at a point in time. They are not, by themselves, a policy version identifier. Use them to compare the request pair and identify the actual rule, action, hostname and path.

let StartTime = datetime(2026-10-03T16:05:00Z);
let EndTime = datetime(2026-10-03T16:35:00Z);
AGWFirewallLogs
| where TimeGenerated between (StartTime .. EndTime)
| where Hostname =~ "api.example.com"
| where RequestUri startswith "/partner/"
| summarize
  matches=count(),
  blocked=countif(Action =~ "Blocked"),
  transactions=dcount(TransactionId),
  rules=make_set(RuleId, 20),
  messages=make_set(Message, 10)
by bin(TimeGenerated, 2m), Hostname, RequestUri, ClientIp
| order by TimeGenerated asc

If the workspace still uses AzureDiagnostics, map the fields to hostname_s, requestUri_s, clientIp_s, action_s, ruleId_s and transactionId_g. Correlate the same identifiers with access and application logs. No WAF row may mean the request used another gateway, no rule emitted an event, or ingestion is late. It does not automatically prove an inactive policy.

Replay a matrix that varies one dimension

A useful canary keeps the same public DNS path, TLS hostname, method, route and request body. It changes exactly one controlled dimension: an expected header, the value intended to match a rule, or the client identity.

Canary A - allowed control
Same hostname, route, method and reference payload
Expected: application response without a WAF block

Canary B - targeted rule
Change one value so the custom rule must match
Expected: blocked with the intended ruleId in logs

Canary C - out-of-scope control
Another route on the same listener
Expected: behavior remains unchanged

Canary D - security refusal
Approved test payload that a managed rule must still block
Expected: block remains visible without a broad exclusion

Run the matrix from a known source at low volume, with a unique identifier per request. Repeat it for only a short interval when ARM has just moved to Succeeded. Continuous retries add noise and may trigger rate limiting without proving anything about policy rollout.

Decide whether to correct, wait or restore

Wait only while an Azure operation is demonstrably active, the change window permits it and no critical journey is exposed. Repair an association when the listener or path rule points to a different policy resource ID than the contract. Repair the rule when its scope, priority or condition differs from the approved commit.

Restore the last healthy policy when Azure reports a complete deployment but the allowed canary regresses, the change affects an out-of-scope hostname or route, the refusal control stops blocking, or effective state cannot be explained before the decision deadline. Roll back through the workflow that owns the resource, then recheck provisioning state, associations, exported diff and the same canary matrix.

A WAF rollback cannot undo a request already accepted by the backend. If the faulty policy opened traffic, retain transaction IDs, review application-side actions and invoke the relevant security runbook. If it only blocked legitimate traffic, measure recovery in gateway and application telemetry before closing the incident.

Conclusion

Apparently inconsistent WAF enforcement is not a reason to edit several rules until the probes turn green. The investigation must connect one request pair to ARM state, effective associations, exact policy contents and WAF events.

That evidence produces an operable choice: wait for an active operation, fix one bounded scope, preserve a legitimate block or restore the last healthy policy. Final validation must prove both the allowed journey and the security refusal that still protects it.