Choose a symptom. Leave with an action path.

[Incident] Internal APIM returns an error on a private API
Context: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Evidence to confirm: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Immediate checks: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID
Proposed action: Run the first checks in order: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Internal APIM returns an error on a private API
Initial hypothesis: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Evidence used: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Checks performed: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID
Decision / action: Run the first checks in order: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Private Endpoint name still resolves publicly
Context: Confirm the CNAME chain, Private DNS Zone association and hybrid forwarding from the consuming network.
Evidence to confirm: nslookup from the workload subnet, CNAME chain, private DNS zone links, resolver forwarding path and cached answers.
Immediate checks: Run nslookup from the workload network | Check privatelink CNAME | Verify Private DNS Zone links and forwarders
Proposed action: Fix zone association or forwarding first, then clear caches and retest from the consuming network.
Rollback: Restore the previous link or forwarder and document the public/private answer difference.

Handled symptom: Private Endpoint name still resolves publicly
Initial hypothesis: Confirm the CNAME chain, Private DNS Zone association and hybrid forwarding from the consuming network.
Evidence used: nslookup from the workload subnet, CNAME chain, private DNS zone links, resolver forwarding path and cached answers.
Checks performed: Run nslookup from the workload network | Check privatelink CNAME | Verify Private DNS Zone links and forwarders
Decision / action: Fix zone association or forwarding first, then clear caches and retest from the consuming network.
Rollback plan: Restore the previous link or forwarder and document the public/private answer difference.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A synthetic probe fails on an Azure private path
Context: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Evidence to confirm: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Immediate checks: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs
Proposed action: Run the first checks in order: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: A synthetic probe fails on an Azure private path
Initial hypothesis: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Evidence used: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Checks performed: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs
Decision / action: Run the first checks in order: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Container Apps private ingress fails or reaches the wrong revision
Context: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Evidence to confirm: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Immediate checks: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Container Apps private ingress fails or reaches the wrong revision
Initial hypothesis: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Evidence used: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Checks performed: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] AKS private ingress returns 502 or reaches no service endpoints
Context: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Evidence to confirm: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Immediate checks: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: AKS private ingress returns 502 or reaches no service endpoints
Initial hypothesis: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Evidence used: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Checks performed: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Functions private HTTP endpoint returns 403, 503 or no request logs
Context: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Evidence to confirm: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Immediate checks: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Functions private HTTP endpoint returns 403, 503 or no request logs
Initial hypothesis: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Evidence used: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Checks performed: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure WAF blocks a legitimate request
Context: Start from blocked requests, rule ID and URI before deciding between exclusion, custom rule or application fix.
Evidence to confirm: Blocked URI, ruleId, match variable, client IP, hostname, request ID and exact time window.
Immediate checks: List blocked URIs in KQL | Identify ruleId and match field | Validate false-positive scope
Proposed action: Create the smallest exclusion or custom rule that covers the false positive without disabling the rule globally.
Rollback: Remove the exclusion/custom rule and verify expected blocking returns for the same rule family.

Handled symptom: Azure WAF blocks a legitimate request
Initial hypothesis: Start from blocked requests, rule ID and URI before deciding between exclusion, custom rule or application fix.
Evidence used: Blocked URI, ruleId, match variable, client IP, hostname, request ID and exact time window.
Checks performed: List blocked URIs in KQL | Identify ruleId and match field | Validate false-positive scope
Decision / action: Create the smallest exclusion or custom rule that covers the false positive without disabling the rule globally.
Rollback plan: Remove the exclusion/custom rule and verify expected blocking returns for the same rule family.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Terraform state lock is stuck
Context: Prove that no apply is still running before using force-unlock, then restart with a clean plan.
Evidence to confirm: Lock ID, lock owner, CI run, backend target, pending plan and whether an apply is still active.
Immediate checks: Identify lock owner | Check CI job status | Run plan after unlock
Proposed action: Unlock only after proving no apply is running, then start with a fresh plan before any apply.
Rollback: Return to previous commit or restore the last validated state version if drift was introduced.

Handled symptom: Terraform state lock is stuck
Initial hypothesis: Prove that no apply is still running before using force-unlock, then restart with a clean plan.
Evidence used: Lock ID, lock owner, CI run, backend target, pending plan and whether an apply is still active.
Checks performed: Identify lock owner | Check CI job status | Run plan after unlock
Decision / action: Unlock only after proving no apply is running, then start with a fresh plan before any apply.
Rollback plan: Return to previous commit or restore the last validated state version if drift was introduced.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A secret rotation or managed identity change breaks an application or pipeline consumer
Context: Separate preparation, cutover, revocation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Evidence to confirm: Separate preparation, cutover, revocation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Immediate checks: List real consumers | Verify the runtime identity and vault read access | Check private DNS and source network | Watch 401/403/500 or Key Vault denials
Proposed action: Run the first checks in order: List real consumers | Verify the runtime identity and vault read access | Check private DNS and source network | Watch 401/403/500 or Key Vault denials. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: A secret rotation or managed identity change breaks an application or pipeline consumer
Initial hypothesis: Separate preparation, cutover, revocation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Evidence used: Separate preparation, cutover, revocation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Checks performed: List real consumers | Verify the runtime identity and vault read access | Check private DNS and source network | Watch 401/403/500 or Key Vault denials
Decision / action: Run the first checks in order: List real consumers | Verify the runtime identity and vault read access | Check private DNS and source network | Watch 401/403/500 or Key Vault denials. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] An automation entry point behaves like a remote console
Context: Bound inputs, templates and repository structure before exposing operations to more users.
Evidence to confirm: Inputs accepted by the template, permissions, inventory scope, repository branch and audit trail.
Immediate checks: List accepted inputs | Remove arbitrary command fields | Review job template permissions
Proposed action: Replace arbitrary inputs with bounded choices and isolate job templates by operational intent.
Rollback: Disable the exposed template or revert to the previous approved template version.

Handled symptom: An automation entry point behaves like a remote console
Initial hypothesis: Bound inputs, templates and repository structure before exposing operations to more users.
Evidence used: Inputs accepted by the template, permissions, inventory scope, repository branch and audit trail.
Checks performed: List accepted inputs | Remove arbitrary command fields | Review job template permissions
Decision / action: Replace arbitrary inputs with bounded choices and isolate job templates by operational intent.
Rollback plan: Disable the exposed template or revert to the previous approved template version.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A private AI agent can act but nobody can explain the action
Context: Tie sources, identities, tool calls, logs and human validation before increasing autonomy.
Evidence to confirm: Source documents, identity, tool calls, prompt context, logs and human approval point.
Immediate checks: List approved sources | Trace tool calls | Define human approval points
Proposed action: Reduce tool scope, require approval on sensitive actions and trace every tool call to a source.
Rollback: Disable the tool integration or force human validation until the action path is explainable.

Handled symptom: A private AI agent can act but nobody can explain the action
Initial hypothesis: Tie sources, identities, tool calls, logs and human validation before increasing autonomy.
Evidence used: Source documents, identity, tool calls, prompt context, logs and human approval point.
Checks performed: List approved sources | Trace tool calls | Define human approval points
Decision / action: Reduce tool scope, require approval on sensitive actions and trace every tool call to a source.
Rollback plan: Disable the tool integration or force human validation until the action path is explainable.
To improve: detection, runbook, guardrail, ownership and communication delay.