Choose a symptom. Leave with an action path.

[Incident] Internal APIM returns an error on a private API
Context: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Evidence to confirm: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Immediate checks: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID
Proposed action: Run the first checks in order: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Internal APIM returns an error on a private API
Initial hypothesis: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Evidence used: Correlate Application Gateway/WAF and APIM logs, then separate DNS, TLS, policy, identity and private backend reachability before changing policies or opening access.
Checks performed: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID
Decision / action: Run the first checks in order: Check whether WAF blocked the request | Confirm APIM received the same path | Validate backend DNS and TLS from the APIM path | Replay with a correlation ID. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Private Endpoint name still resolves publicly
Context: Confirm the CNAME chain, Private DNS Zone association and hybrid forwarding from the consuming network.
Evidence to confirm: nslookup from the workload subnet, CNAME chain, private DNS zone links, resolver forwarding path and cached answers.
Immediate checks: Run nslookup from the workload network | Check privatelink CNAME | Verify Private DNS Zone links and forwarders
Proposed action: Fix zone association or forwarding first, then clear caches and retest from the consuming network.
Rollback: Restore the previous link or forwarder and document the public/private answer difference.

Handled symptom: Private Endpoint name still resolves publicly
Initial hypothesis: Confirm the CNAME chain, Private DNS Zone association and hybrid forwarding from the consuming network.
Evidence used: nslookup from the workload subnet, CNAME chain, private DNS zone links, resolver forwarding path and cached answers.
Checks performed: Run nslookup from the workload network | Check privatelink CNAME | Verify Private DNS Zone links and forwarders
Decision / action: Fix zone association or forwarding first, then clear caches and retest from the consuming network.
Rollback plan: Restore the previous link or forwarder and document the public/private answer difference.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Storage private endpoint returns 403, times out or produces no request logs
Context: Separate Storage subresource DNS, Private Endpoint approval, firewall rules, runtime identity and Storage logs before opening public access or broadening RBAC.
Evidence to confirm: Separate Storage subresource DNS, Private Endpoint approval, firewall rules, runtime identity and Storage logs before opening public access or broadening RBAC.
Immediate checks: Resolve the exact Storage subresource from the workload network | Check Private Endpoint status and private DNS zone group | Replay with a client request ID | Correlate Storage logs for 403, caller IP and requester identity
Proposed action: Run the first checks in order: Resolve the exact Storage subresource from the workload network | Check Private Endpoint status and private DNS zone group | Replay with a client request ID | Correlate Storage logs for 403, caller IP and requester identity. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Storage private endpoint returns 403, times out or produces no request logs
Initial hypothesis: Separate Storage subresource DNS, Private Endpoint approval, firewall rules, runtime identity and Storage logs before opening public access or broadening RBAC.
Evidence used: Separate Storage subresource DNS, Private Endpoint approval, firewall rules, runtime identity and Storage logs before opening public access or broadening RBAC.
Checks performed: Resolve the exact Storage subresource from the workload network | Check Private Endpoint status and private DNS zone group | Replay with a client request ID | Correlate Storage logs for 403, caller IP and requester identity
Decision / action: Run the first checks in order: Resolve the exact Storage subresource from the workload network | Check Private Endpoint status and private DNS zone group | Replay with a client request ID | Correlate Storage logs for 403, caller IP and requester identity. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure SQL private endpoint returns timeouts, firewall errors or no SQL logs
Context: Separate SQL private DNS, Private Endpoint state, public access, firewall rules, runtime identity and SQL diagnostics before changing schema, code or broad permissions.
Evidence to confirm: Separate SQL private DNS, Private Endpoint state, public access, firewall rules, runtime identity and SQL diagnostics before changing schema, code or broad permissions.
Immediate checks: Resolve the SQL FQDN from the workload network | Check Private Endpoint status and privatelink.database.windows.net records | Replay with the real runtime identity | Correlate SQL diagnostics for firewall and login errors
Proposed action: Run the first checks in order: Resolve the SQL FQDN from the workload network | Check Private Endpoint status and privatelink.database.windows.net records | Replay with the real runtime identity | Correlate SQL diagnostics for firewall and login errors. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure SQL private endpoint returns timeouts, firewall errors or no SQL logs
Initial hypothesis: Separate SQL private DNS, Private Endpoint state, public access, firewall rules, runtime identity and SQL diagnostics before changing schema, code or broad permissions.
Evidence used: Separate SQL private DNS, Private Endpoint state, public access, firewall rules, runtime identity and SQL diagnostics before changing schema, code or broad permissions.
Checks performed: Resolve the SQL FQDN from the workload network | Check Private Endpoint status and privatelink.database.windows.net records | Replay with the real runtime identity | Correlate SQL diagnostics for firewall and login errors
Decision / action: Run the first checks in order: Resolve the SQL FQDN from the workload network | Check Private Endpoint status and privatelink.database.windows.net records | Replay with the real runtime identity | Correlate SQL diagnostics for firewall and login errors. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Service Bus private endpoint times out, denies access or lets backlog grow
Context: Separate Service Bus private DNS, Private Endpoint state, public access, managed identity or SAS, queue metrics and processing logs before touching queues or redeploying consumers.
Evidence to confirm: Separate Service Bus private DNS, Private Endpoint state, public access, managed identity or SAS, queue metrics and processing logs before touching queues or redeploying consumers.
Immediate checks: Resolve the Service Bus FQDN from the workload network | Check Private Endpoint approval and public network access | Verify the real sender or receiver identity | Correlate backlog, dead-letter and Service Bus errors
Proposed action: Run the first checks in order: Resolve the Service Bus FQDN from the workload network | Check Private Endpoint approval and public network access | Verify the real sender or receiver identity | Correlate backlog, dead-letter and Service Bus errors. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Service Bus private endpoint times out, denies access or lets backlog grow
Initial hypothesis: Separate Service Bus private DNS, Private Endpoint state, public access, managed identity or SAS, queue metrics and processing logs before touching queues or redeploying consumers.
Evidence used: Separate Service Bus private DNS, Private Endpoint state, public access, managed identity or SAS, queue metrics and processing logs before touching queues or redeploying consumers.
Checks performed: Resolve the Service Bus FQDN from the workload network | Check Private Endpoint approval and public network access | Verify the real sender or receiver identity | Correlate backlog, dead-letter and Service Bus errors
Decision / action: Run the first checks in order: Resolve the Service Bus FQDN from the workload network | Check Private Endpoint approval and public network access | Verify the real sender or receiver identity | Correlate backlog, dead-letter and Service Bus errors. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A synthetic probe fails on an Azure private path
Context: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Evidence to confirm: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Immediate checks: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs
Proposed action: Run the first checks in order: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: A synthetic probe fails on an Azure private path
Initial hypothesis: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Evidence used: Separate DNS, TLS, Application Gateway health, WAF blocks and runner network before changing routing or application code.
Checks performed: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs
Decision / action: Run the first checks in order: Resolve the hostname from the probe network | Check TLS/SNI with the real hostname | Correlate probe run with WAF and gateway logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure traffic leaves through one path and returns through another
Context: Compare DNS target, effective routes, route table associations, firewall evidence, NAT identity and return path before changing UDRs or bypassing inspection.
Evidence to confirm: Compare DNS target, effective routes, route table associations, firewall evidence, NAT identity and return path before changing UDRs or bypassing inspection.
Immediate checks: Resolve the destination from the source path | Compare effective routes on source and destination NICs | Check firewall or appliance logs for both directions | Confirm NAT or outbound source identity
Proposed action: Run the first checks in order: Resolve the destination from the source path | Compare effective routes on source and destination NICs | Check firewall or appliance logs for both directions | Confirm NAT or outbound source identity. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure traffic leaves through one path and returns through another
Initial hypothesis: Compare DNS target, effective routes, route table associations, firewall evidence, NAT identity and return path before changing UDRs or bypassing inspection.
Evidence used: Compare DNS target, effective routes, route table associations, firewall evidence, NAT identity and return path before changing UDRs or bypassing inspection.
Checks performed: Resolve the destination from the source path | Compare effective routes on source and destination NICs | Check firewall or appliance logs for both directions | Confirm NAT or outbound source identity
Decision / action: Run the first checks in order: Resolve the destination from the source path | Compare effective routes on source and destination NICs | Check firewall or appliance logs for both directions | Confirm NAT or outbound source identity. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Container Apps private ingress fails or reaches the wrong revision
Context: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Evidence to confirm: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Immediate checks: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Container Apps private ingress fails or reaches the wrong revision
Initial hypothesis: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Evidence used: Separate private DNS, Application Gateway handoff, Container Apps ingress mode, revision traffic and console logs before rolling back or changing traffic weights.
Checks performed: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Check ingress target port and active revisions | Correlate system and console logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] AKS private ingress returns 502 or reaches no service endpoints
Context: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Evidence to confirm: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Immediate checks: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: AKS private ingress returns 502 or reaches no service endpoints
Initial hypothesis: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Evidence used: Separate private DNS, Application Gateway health, ingress controller routing, Kubernetes service selectors, endpoint slices and pod readiness before rolling back a deployment.
Checks performed: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Check Application Gateway backend health and host header | Verify ingress, service and endpoint slices | Correlate controller and application logs. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] An AKS node pool upgrade is blocked while draining a pod protected by a PDB
Context: Separate unavailable replicas, readiness failures, pod placement, surge capacity and an over-constrained PodDisruptionBudget before deleting the safeguard or forcing the upgrade.
Evidence to confirm: Separate unavailable replicas, readiness failures, pod placement, surge capacity and an over-constrained PodDisruptionBudget before deleting the safeguard or forcing the upgrade.
Immediate checks: Capture the blocked node and eviction events | Inspect PDB disruptionsAllowed and selectors | Verify replacement capacity, replica readiness and topology | Resume only with positive disruption headroom
Proposed action: Run the first checks in order: Capture the blocked node and eviction events | Inspect PDB disruptionsAllowed and selectors | Verify replacement capacity, replica readiness and topology | Resume only with positive disruption headroom. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: An AKS node pool upgrade is blocked while draining a pod protected by a PDB
Initial hypothesis: Separate unavailable replicas, readiness failures, pod placement, surge capacity and an over-constrained PodDisruptionBudget before deleting the safeguard or forcing the upgrade.
Evidence used: Separate unavailable replicas, readiness failures, pod placement, surge capacity and an over-constrained PodDisruptionBudget before deleting the safeguard or forcing the upgrade.
Checks performed: Capture the blocked node and eviction events | Inspect PDB disruptionsAllowed and selectors | Verify replacement capacity, replica readiness and topology | Resume only with positive disruption headroom
Decision / action: Run the first checks in order: Capture the blocked node and eviction events | Inspect PDB disruptionsAllowed and selectors | Verify replacement capacity, replica readiness and topology | Resume only with positive disruption headroom. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Functions private HTTP endpoint returns 403, 503 or no request logs
Context: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Evidence to confirm: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Immediate checks: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions
Proposed action: Run the first checks in order: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Functions private HTTP endpoint returns 403, 503 or no request logs
Initial hypothesis: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Evidence used: Separate private DNS, Private Endpoint reachability, access restrictions, Functions runtime state, private storage and Application Insights evidence before redeploying code or opening public access.
Checks performed: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions
Decision / action: Run the first checks in order: Resolve the hostname from the caller network | Replay with a correlation ID | Check Function App access restrictions and Private Endpoint status | Correlate requests, traces and exceptions. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure WAF blocks a legitimate request
Context: Start from blocked requests, rule ID and URI before deciding between exclusion, custom rule or application fix.
Evidence to confirm: Blocked URI, ruleId, match variable, client IP, hostname, request ID and exact time window.
Immediate checks: List blocked URIs in KQL | Identify ruleId and match field | Validate false-positive scope
Proposed action: Create the smallest exclusion or custom rule that covers the false positive without disabling the rule globally.
Rollback: Remove the exclusion/custom rule and verify expected blocking returns for the same rule family.

Handled symptom: Azure WAF blocks a legitimate request
Initial hypothesis: Start from blocked requests, rule ID and URI before deciding between exclusion, custom rule or application fix.
Evidence used: Blocked URI, ruleId, match variable, client IP, hostname, request ID and exact time window.
Checks performed: List blocked URIs in KQL | Identify ruleId and match field | Validate false-positive scope
Decision / action: Create the smallest exclusion or custom rule that covers the false positive without disabling the rule globally.
Rollback plan: Remove the exclusion/custom rule and verify expected blocking returns for the same rule family.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Terraform state lock is stuck
Context: Prove that no apply is still running before using force-unlock, then restart with a clean plan.
Evidence to confirm: Lock ID, lock owner, CI run, backend target, pending plan and whether an apply is still active.
Immediate checks: Identify lock owner | Check CI job status | Run plan after unlock
Proposed action: Unlock only after proving no apply is running, then start with a fresh plan before any apply.
Rollback: Return to previous commit or restore the last validated state version if drift was introduced.

Handled symptom: Terraform state lock is stuck
Initial hypothesis: Prove that no apply is still running before using force-unlock, then restart with a clean plan.
Evidence used: Lock ID, lock owner, CI run, backend target, pending plan and whether an apply is still active.
Checks performed: Identify lock owner | Check CI job status | Run plan after unlock
Decision / action: Unlock only after proving no apply is running, then start with a fresh plan before any apply.
Rollback plan: Return to previous commit or restore the last validated state version if drift was introduced.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] Azure Monitor fires an alert storm after deployment
Context: Separate real service impact, noisy dimensions, threshold drift, action group behavior and rollback before silencing notifications or changing rules.
Evidence to confirm: Separate real service impact, noisy dimensions, threshold drift, action group behavior and rollback before silencing notifications or changing rules.
Immediate checks: Group fired alerts by rule and target | Compare first alert time with the deployment window | Check service logs before changing thresholds | Keep one user-symptom alert active
Proposed action: Run the first checks in order: Group fired alerts by rule and target | Compare first alert time with the deployment window | Check service logs before changing thresholds | Keep one user-symptom alert active. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: Azure Monitor fires an alert storm after deployment
Initial hypothesis: Separate real service impact, noisy dimensions, threshold drift, action group behavior and rollback before silencing notifications or changing rules.
Evidence used: Separate real service impact, noisy dimensions, threshold drift, action group behavior and rollback before silencing notifications or changing rules.
Checks performed: Group fired alerts by rule and target | Compare first alert time with the deployment window | Check service logs before changing thresholds | Keep one user-symptom alert active
Decision / action: Run the first checks in order: Group fired alerts by rule and target | Compare first alert time with the deployment window | Check service logs before changing thresholds | Keep one user-symptom alert active. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A secret rotation, federated identity or managed identity change breaks an application or pipeline consumer
Context: Separate preparation, cutover, revocation, workload identity federation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Evidence to confirm: Separate preparation, cutover, revocation, workload identity federation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Immediate checks: List real consumers | Verify the runtime identity and vault read access | Check OIDC claims or private DNS depending on the path | Watch 401/403/500, sign-in failures or Key Vault denials
Proposed action: Run the first checks in order: List real consumers | Verify the runtime identity and vault read access | Check OIDC claims or private DNS depending on the path | Watch 401/403/500, sign-in failures or Key Vault denials. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: A secret rotation, federated identity or managed identity change breaks an application or pipeline consumer
Initial hypothesis: Separate preparation, cutover, revocation, workload identity federation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Evidence used: Separate preparation, cutover, revocation, workload identity federation and managed identity diagnostics; validate the real execution identity, private path and authentication errors before deleting the old value or broadening access.
Checks performed: List real consumers | Verify the runtime identity and vault read access | Check OIDC claims or private DNS depending on the path | Watch 401/403/500, sign-in failures or Key Vault denials
Decision / action: Run the first checks in order: List real consumers | Verify the runtime identity and vault read access | Check OIDC claims or private DNS depending on the path | Watch 401/403/500, sign-in failures or Key Vault denials. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] An Azure Functions poison queue keeps growing
Context: Separate contract failures, transient dependencies, identity, resource pressure and partial side effects before replaying Storage Queue messages.
Evidence to confirm: Separate contract failures, transient dependencies, identity, resource pressure and partial side effects before replaying Storage Queue messages.
Immediate checks: Peek a redacted sample without consuming messages | Correlate message IDs with invocation and dependency logs | Verify downstream state and idempotency before replay
Proposed action: Run the first checks in order: Peek a redacted sample without consuming messages | Correlate message IDs with invocation and dependency logs | Verify downstream state and idempotency before replay. Open the linked notes before changing production.
Rollback: Stop the change, restore the last known safe state and keep the captured evidence for comparison.

Handled symptom: An Azure Functions poison queue keeps growing
Initial hypothesis: Separate contract failures, transient dependencies, identity, resource pressure and partial side effects before replaying Storage Queue messages.
Evidence used: Separate contract failures, transient dependencies, identity, resource pressure and partial side effects before replaying Storage Queue messages.
Checks performed: Peek a redacted sample without consuming messages | Correlate message IDs with invocation and dependency logs | Verify downstream state and idempotency before replay
Decision / action: Run the first checks in order: Peek a redacted sample without consuming messages | Correlate message IDs with invocation and dependency logs | Verify downstream state and idempotency before replay. Open the linked notes before changing production.
Rollback plan: Stop the change, restore the last known safe state and keep the captured evidence for comparison.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] An automation entry point behaves like a remote console
Context: Bound inputs, templates and repository structure before exposing operations to more users.
Evidence to confirm: Inputs accepted by the template, permissions, inventory scope, repository branch and audit trail.
Immediate checks: List accepted inputs | Remove arbitrary command fields | Review job template permissions
Proposed action: Replace arbitrary inputs with bounded choices and isolate job templates by operational intent.
Rollback: Disable the exposed template or revert to the previous approved template version.

Handled symptom: An automation entry point behaves like a remote console
Initial hypothesis: Bound inputs, templates and repository structure before exposing operations to more users.
Evidence used: Inputs accepted by the template, permissions, inventory scope, repository branch and audit trail.
Checks performed: List accepted inputs | Remove arbitrary command fields | Review job template permissions
Decision / action: Replace arbitrary inputs with bounded choices and isolate job templates by operational intent.
Rollback plan: Disable the exposed template or revert to the previous approved template version.
To improve: detection, runbook, guardrail, ownership and communication delay.

[Incident] A private AI agent can act but nobody can explain the action
Context: Tie sources, identities, tool calls, logs and human validation before increasing autonomy.
Evidence to confirm: Source documents, identity, tool calls, prompt context, logs and human approval point.
Immediate checks: List approved sources | Trace tool calls | Define human approval points
Proposed action: Reduce tool scope, require approval on sensitive actions and trace every tool call to a source.
Rollback: Disable the tool integration or force human validation until the action path is explainable.

Handled symptom: A private AI agent can act but nobody can explain the action
Initial hypothesis: Tie sources, identities, tool calls, logs and human validation before increasing autonomy.
Evidence used: Source documents, identity, tool calls, prompt context, logs and human approval point.
Checks performed: List approved sources | Trace tool calls | Define human approval points
Decision / action: Reduce tool scope, require approval on sensitive actions and trace every tool call to a source.
Rollback plan: Disable the tool integration or force human validation until the action path is explainable.
To improve: detection, runbook, guardrail, ownership and communication delay.