Azure Chaos Studio: bound a resilience test before disrupting production
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read article
Tag
123 articles connected to this technical signal.
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read articleA production runbook for finding an Azure change with Resource Graph Change Analysis, connecting it to its actor and impact, then validating or rolling it back without reverting blindly.
Read articleA production runbook for mapping an ACR vulnerability signal to AKS pods, freezing promotion, returning to a known digest, validating the rollout, then retaining or deleting the suspect image.
Read articleA production runbook for running an Azure Site Recovery test failover in an isolated network, validating dependencies, measuring application recovery and cleaning up unambiguously.
Read articleA production runbook for comparing a candidate agent with the active runtime on the same requests without duplicating actions, then deciding promotion, canary or rollback.
Read articleA production runbook for qualifying an Azure Key Vault secret consumed by Azure Pipelines through a variable group, isolating version, mapping, identity, network and precedence, then validating or rolling back without exposing the value.
Read articleA production runbook for proving which Azure Firewall rule handles a flow, separating routing from policy order, and applying a targeted fix with validation and rollback.
Read articleA production runbook for building and qualifying an Azure Monitor burn-rate alert by separating the SLI, error budget, short and long windows, telemetry quality, notification, validation and rollback.
Read articleA production runbook for qualifying poisoned Queue trigger messages, separating deterministic and transient failures, proving idempotency, then replaying or rolling back without duplicating business side effects.
Read articleA production runbook for locating telemetry loss across receivers, processors, queues and exporters, then validating capacity changes or rolling back safely.
Read articleA production runbook for attributing token and tool-call growth, isolating amplification, enforcing an execution budget, and validating or rolling back an AI agent release.
Read articleA production runbook for diagnosing concurrent GitHub Actions deployments, serializing writes, validating the active version, and resuming or rolling back without adding another race.
Read articleA production runbook for canarying a KQL transformation in an Azure Monitor Data Collection Rule, comparing volume and schema, detecting rejected records, then validating or rolling back without an observability gap.
Read articleA production runbook for qualifying a Terraform provider upgrade with a lockfile, baseline plan, canary, operational evidence and an explicit rollback.
Read articleA production runbook for defining latency budgets, idempotency, retries, circuit breakers, traces and rollback for an MCP tool called by an AI agent.
Read articleA production runbook for bounding a BGP change, comparing learned and advertised routes, checking effective routes, canarying one prefix, then validating or withdrawing it without destabilizing the Azure hub.
Read articleA production runbook for diagnosing an AKS drain blocked by a PodDisruptionBudget, recovering disruption headroom, fixing capacity or readiness, then validating or postponing the upgrade without removing safeguards.
Read articleA production runbook for recovering Application Insights and OpenTelemetry correlation across Azure Service Bus, qualifying W3C propagation, sampling and message processing, then validating or rolling back the fix.
Read articleA production runbook for qualifying structured AI agent tool output with schema, sources, diff, idempotence, policy, traces, human validation and rollback before writing to production.
Read articleA production runbook for reviewing managed resources, unmanage behavior, deny settings, excluded identities, validation and rollback before updating an Azure Deployment Stack.
Read articleA production runbook for qualifying AI agent permission drift with the real identity, RBAC, tool scopes, traces, expected denials, human validation and rollback before widening access.
Read articleA production runbook for qualifying an Azure Container Apps Job that no longer consumes correctly with KEDA scale rule, backlog, managed identity, secrets, logs, idempotency, validation and rollback before rerunning workers.
Read articleA production runbook for qualifying an Azure DevOps self-hosted agent that no longer picks up jobs by separating runner service, identity, network, disk, cache, pool, logs and rollback.
Read articleA production runbook for qualifying a blocked Azure flow with Network Watcher, NSG Flow Logs, effective security rules, routes, firewall, KQL evidence, validation and rollback before opening a broad rule.
Read articleA production runbook for qualifying the handoff from a Microsoft Foundry agent to Azure Automation, AWX, Azure DevOps or an MCP tool with contract, identity, traces, dry run, human validation and rollback.
Read articleA production runbook for qualifying Azure Firewall Threat Intelligence with logs, false positives, critical flows, targeted exceptions, deny decision and rollback before enabling Deny.
Read articleA production runbook for qualifying an Azure API Management regression with APIM traces, policy diff, headers, cache, backend, identity, logs, validation and rollback before changing the API or target service.
Read articleA production runbook for qualifying an Azure WAF OWASP/CRS managed rule upgrade with logs, false positives, Detection mode, policy diff, application validation, Prevention decision and rollback.
Read articleA production runbook for qualifying an Azure API Management self-hosted gateway with configuration sync, token or Workload Identity, Kubernetes, logs, backend reachability, validation and rollback before moving traffic.
Read articleA production runbook for qualifying a stuck Azure Durable Functions orchestration with instance history, activity state, storage, idempotency, KQL, validation and rollback before replay, terminate or purge.
Read articleA production runbook for qualifying Azure Container Apps 401/403 failures with Managed Identity, token evidence, RBAC scope, Key Vault, ACR, logs, validation and rollback before widening permissions.
Read articleA production runbook for qualifying an Azure Storage lifecycle policy with Blob inventory, versions, snapshots, delete or tiering rules, evidence, validation and rollback before removing data.
Read articleA production runbook for qualifying an Azure Container Apps incident after deployment by separating active revisions, traffic weights, labels, logs, probes, dependencies, validation and rollback.
Read articleA production runbook for qualifying a Conditional Access block on a workload identity with service principal logs, policy scope, risk, CI identity, KQL evidence, bounded exception and rollback.
Read articleA production runbook for qualifying MCP server drift with tool manifests, schemas, identity, secrets, network path, traces, evaluations, validation and rollback before reauthorizing an AI agent.
Read articleA production runbook for qualifying a failed Azure Functions Timer Trigger with execution history, schedule locks, storage, Application Insights, idempotency, validation and rollback before manual replay.
Read articleA production runbook for qualifying Azure DevOps Variable Group drift with variables, secrets, Key Vault, scopes, logs, validation and rollback before rerunning a pipeline.
Read articleA production runbook for qualifying Azure Workbooks drift with KQL, Log Analytics, Application Insights, dimensions, ingestion latency, validation and rollback before changing alerts or dashboards.
Read articleA production runbook for qualifying Azure Front Door origin health with probes, routing, DNS, TLS, WAF evidence, backend logs, validation and rollback before forcing failover.
Read articleA production runbook for qualifying Azure WAF rate limiting with logs, counting key, real clients, false positives, load checks, threshold decision and rollback.
Read articleA production runbook for qualifying an internal MCP server with tool inventory, scopes, identities, secrets, audit, dry runs, evaluations, human validation and rollback before agent access.
Read articleA production runbook for qualifying Azure API Management mTLS failures with client certificate chain, hostname, policy, gateway logs, backend identity, validation and rollback before changing API policies.
Read articleA production runbook for qualifying apparent Application Insights telemetry loss with sampling, ingestion, SDK configuration, KQL, alerts, validation and rollback before changing thresholds.
Read articleA production runbook for qualifying an Azure Service Health or Resource Health signal with user impact, dependencies, routing, DNS, observability, failover decision and rollback.
Read articleA production runbook for qualifying a failed AI agent tool call with trace evidence, idempotence, identity, backend state, approvals, validation and rollback before retrying.
Read articleA production runbook for qualifying AI agent memory with sources, traces, aging, permissions, evaluations, guardrails, human validation and rollback before it influences real actions.
Read articleA production runbook for qualifying an Azure Backup restore point with application consistency, dependencies, identity, networking, test restore, evidence, decision and rollback before cutover.
Read articleA production runbook for qualifying an Azure Monitor Action Group by separating rules, receivers, webhooks, escalation paths, KQL evidence, validation and rollback before reducing notifications.
Read articleA production runbook for qualifying an AI agent evaluation set with business cases, retrieval, tool calls, traces, thresholds, human validation and rollback before promotion.
Read articleA production runbook for qualifying an Azure App Service Managed Identity failure with IMDS endpoint, Entra ID, RBAC, Key Vault or target API evidence, logs, validation and rollback before reintroducing a client secret.
Read articleA production runbook for qualifying an Azure DevOps failure on a self-hosted agent with disk, workspace, cache, local services, identity, logs, controlled cleanup, validation and rollback.
Read articleA production runbook for tightening an Azure Firewall rule without breaking useful traffic, with inventory, KQL evidence, dependencies, controlled validation and rollback.
Read articleA production runbook for qualifying Azure API Management 401/403 errors with subscription key, product, API, consumer, logs, validation and rollback before rotation.
Read articleA production runbook for qualifying a delayed Azure Monitor alert by separating application timestamps, Log Analytics ingestion, KQL query windows, evaluation frequency, action groups, validation and rollback.
Read articleA production runbook for qualifying App Service egress IP drift with VNet Integration, NAT Gateway, DNS, UDRs, destination logs, validation and rollback before changing a partner allowlist.
Read articleA production runbook for qualifying a failed Azure Logic Apps execution with trigger, connectors, managed identity, payload, logs, validation and rollback before resubmit.
Read articleA production runbook for qualifying Cosmos DB 429 throttling with RU consumption, hot partitions, query shape, SDK retries, indexing, KQL evidence, scaling decision and rollback.
Read articleA production runbook for qualifying Microsoft Foundry agent guardrails with sources, refusals, tools, identity, traces, canary, human validation and rollback before user exposure.
Read articleA production runbook for qualifying suspected prompt injection in an AI agent retrieval corpus with sources, traces, evaluation, tools, guardrails, validation and rollback.
Read articleA production runbook for qualifying Key Vault degradation by separating latency, throttling, identity, network path, application cache, logs and rollback before starting a secret rotation.
Read articleA production runbook for qualifying an Azure DevOps deployment blocked by approvals or checks with environment scope, identity, logs, audit evidence, validation and rollback before bypassing the guardrail.
Read articleA production runbook for qualifying an Azure Policy deployment deny with assignment, initiative, deny effect, compliance, scoped exemption, validation and rollback.
Read articleA production runbook for qualifying an agent approval policy change with action scope, identity, traces, evaluation cases, guardrails, human validation and rollback.
Read articleA production runbook for qualifying an unhealthy Azure Load Balancer backend with probe behavior, NSGs, routing, logs, pool configuration, validation and rollback before changing the rule or redeploying.
Read articleA production runbook for qualifying an Application Gateway TLS certificate rotation from Key Vault with managed identity, listener checks, WAF evidence, probes, logs, validation and rollback.
Read articleA production runbook for qualifying a Terraform, Bicep or ARM plan before apply with drift, CI identity, Azure Policy, destructive changes, validation and rollback.
Read articleA production runbook for qualifying Azure OpenAI or Microsoft Foundry throttling with quota, deployment capacity, agent traces, retries, fallback, validation and rollback before changing models.
Read articleA production runbook for qualifying an Azure Managed Grafana dashboard or alert with Azure Monitor datasource, managed identity, Log Analytics permissions, variables, traces, validation and rollback before changing KQL queries.
Read articleA production runbook for rotating or revoking an AI agent runtime identity with scoped permissions, dry-run tool calls, traces, approvals, validation and rollback before breaking production actions.
Read articleA production runbook for qualifying an Azure Front Door failure with origin groups, health probes, TLS, Private Link, DNS, WAF, logs, validation and rollback before changing routing.
Read articleA production runbook for qualifying Azure Event Grid delivery failures with subscription filters, endpoint health, dead-letter storage, diagnostics, replay scope, validation and rollback before reprocessing events.
Read articleA production runbook for qualifying a Key Vault reference failure with managed identity, RBAC, secret version, diagnostics, KQL, validation and rollback before rotating or broadening access.
Read articleA production runbook for qualifying an Azure Monitor alert that fired but did not notify anyone, with action groups, receivers, processing rules, webhooks, evidence, validation and rollback.
Read articleA production runbook for qualifying Azure asymmetric routing with effective routes, UDRs, Azure Firewall, NAT Gateway, flow logs, validation and rollback before changing route tables.
Read articleA production runbook for qualifying an Azure Automation failure with managed identity, parameters, modules, Hybrid Worker, webhooks, logs, validation and rollback before rerun.
Read articleA production runbook for qualifying a scheduled Azure Automation job with trigger, managed identity, parameters, dry-run, logs, validation and rollback before allowing real action.
Read articleA production runbook for qualifying a new AI agent tool with contract review, scoped identity, dry run, traces, approvals, evaluation cases and rollback before enabling real actions.
Read articleA production runbook for qualifying an APIM secret rotation with Named Values, Key Vault, managed identity, private backend, logs, application validation and rollback.
Read articleA production runbook for qualifying a retrieval index update with source diff, metadata, chunking, evaluations, traces, human validation and rollback before changing an AI agent's answers.
Read articleA production runbook for qualifying an Azure App Configuration or feature flag regression with labels, identities, refresh, Key Vault references, logs, validation and rollback.
Read articleA production runbook for qualifying an Azure App Service slot swap with configuration drift, warmup, identity, dependencies, logs, validation and rollback before promotion.
Read articleA production runbook for qualifying an Azure egress failure with Firewall DNS proxy, FQDN resolution, UDRs, application rules, logs, validation and rollback before broadening traffic.
Read articleA production runbook for qualifying incomplete AI agent traces with conversation events, sources, tool calls, identities, approvals, evaluations and rollback before restoring an action.
Read articleA production runbook for qualifying an AWX dynamic inventory change with source, host diff, variables, identities, limits, validation and rollback before execution.
Read articleA production runbook for qualifying an ACR image pull failure with identity, network, DNS, firewall, logs, node cache, validation and rollback before rerunning deployment.
Read articleA production runbook for qualifying missing Azure Monitor logs with Diagnostic Settings, DCRs, ingestion, KQL, cost controls, validation and rollback before changing alerts.
Read articleA production runbook for qualifying Azure RBAC drift with the real identity, scope, PIM, role assignments, deny assignments, logs, validation and rollback before adding permissions.
Read articleA production runbook for exposing Azure DevOps to an AI agent through MCP with project scope, permissions, allowed actions, evidence, logs, human validation and rollback.
Read articleA production runbook for qualifying an Azure Private DNS change with workload-side resolution, VNet links, Private Resolver, TTL, logs, application validation and rollback.
Read articleA production runbook for qualifying a failed AWX job with Ansible events, changed tasks, affected hosts, variables, limited rerun, validation and rollback.
Read articleA production runbook for qualifying an Azure OIDC authentication failure in a CI pipeline with federated credentials, claims, roles, logs, validation and rollback.
Read articleA production runbook for qualifying AI agent contract drift with prompts, tool manifests, sources, evaluations, traces, human validation and rollback.
Read articleA production runbook for qualifying Azure hybrid DNS incidents with Private DNS Resolver, forwarding rulesets, private zones, caches, evidence, validation and rollback.
Read articleA production runbook for qualifying Azure egress failures with UDR, NAT Gateway, NSG, firewall, DNS, KQL, validation and rollback before adding an overbroad rule.
Read articleA production runbook for documenting an Azure WAF change with KQL evidence, policy diff, application probes, security validation, merge decision and rollback.
Read articleA production runbook for qualifying an AI agent action with traces, sources, tools, identity, KQL, human validation and rollback without disabling the whole assistant.
Read articleA production runbook for qualifying an Azure deployment failure from a private runner by separating identity, DNS, routing, NSG, NAT, dependency access, logs and rollback.
Read articleA production runbook for qualifying Azure Container Apps outbound failures by separating DNS, UDR, NSG, NAT Gateway, firewall, identity, logs and rollback before changing the application.
Read articleA production runbook for deciding an Azure rollback after deployment with Azure Monitor, KQL, impact correlation, regression evidence, validation and controlled recovery.
Read articleA production runbook for running Azure Policy remediation with impact preview, managed identity checks, controlled batches, exemptions, logs and rollback.
Read articleA production runbook for qualifying an Azure WAF policy before switching to Prevention with KQL evidence, change scope, application validation, rollback window and an operable decision.
Read articleA production runbook for qualifying blocked Azure traffic by separating NSG, UDR, Azure Firewall, DNS, effective routes, logs and rollback before opening access too broadly.
Read articleA runbook for validating an AI agent before real action by separating sources, tools, identity, evaluation cases, human approvals, logs and rollback.
Read articleA production runbook for exposing MCP tools to an AI agent while keeping action scope, approvals, identities, logs, evaluation and rollback under control.
Read articleA production runbook for qualifying an Azure Monitor alert storm after deployment by separating real signal, noise, regression, threshold drift, action group behavior and rollback.
Read articleAn operational runbook for qualifying Azure outbound incidents involving NAT Gateway, SNAT ports, UDR, NSG, firewalls, allowlists and rollback.
Read articleAn operational runbook for qualifying Azure App Service or Functions outbound failures by separating VNet Integration, DNS, UDR, NSG, NAT, logs and rollback.
Read articleA production runbook for qualifying an AI agent that selects the wrong tool, acts without evidence or hides an action behind a plausible answer.
Read articleAn operational runbook for Azure Service Bus private access incidents by separating DNS, Private Endpoint, identity, firewall, metrics, logs and rollback.
Read articleAn operational runbook for Azure SQL private access failures by separating DNS, Private Endpoint, firewall, identity, SQL logs and rollback evidence.
Read articleAn operational runbook for Azure Storage private access failures by separating DNS, Private Endpoint, firewall, identity, logs and rollback evidence.
Read articleBuild an operational runbook for App Service private access failures by separating DNS, Private Endpoint, access restrictions, Application Gateway, application logs and rollback evidence.
Read articleBuild an operational runbook for private Azure Functions failures by separating DNS, Private Endpoint, access restrictions, private storage, Application Insights logs and rollback evidence.
Read articleBuild an operational runbook for AKS private ingress failures by separating DNS, Application Gateway, ingress controller, Kubernetes service endpoints, pod readiness and rollback evidence.
Read articleA short query to read ingress controller and application logs together when a private AKS route returns 502, timeouts or no endpoints.
Read articleBuild an operational runbook for Azure Container Apps private ingress failures by separating DNS, ingress mode, revision routing, application logs and rollback evidence.
Read articleA short query to correlate Azure Container Apps system and console logs when private ingress, probes or revision traffic fail.
Read articleQualify a failure across Application Gateway, WAF, internal APIM and a private backend by separating DNS, routing, policy, identity and logs before any fix.
Read articleBuild a runbook for Key Vault, Storage or private API access failures with managed identity, RBAC, private DNS, logs and real execution evidence.
Read articleApply a temporary Azure WAF custom rule with priority, KQL evidence, business validation and rollback, without permanently hiding managed-rule signals.
Read articleA short command to list custom rules in an Azure WAF policy with priority, action and type before an urgent change.
Read articleBuild operable rotation for secrets, certificates, and application identities with dependency inventory, evidence, change windows, monitoring, and rollback.
Read articleA short query to watch 401, 403, and 500 errors after rotating an application secret or service identity.
Read article