Azure Chaos Studio: bound a resilience test before disrupting production
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read article
Tag
129 articles connected to this technical signal.
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read articleA production runbook for finding an Azure change with Resource Graph Change Analysis, connecting it to its actor and impact, then validating or rolling it back without reverting blindly.
Read articleA production runbook for mapping an ACR vulnerability signal to AKS pods, freezing promotion, returning to a known digest, validating the rollout, then retaining or deleting the suspect image.
Read articleA production runbook for running an Azure Site Recovery test failover in an isolated network, validating dependencies, measuring application recovery and cleaning up unambiguously.
Read articleA production runbook for qualifying an Azure Key Vault secret consumed by Azure Pipelines through a variable group, isolating version, mapping, identity, network and precedence, then validating or rolling back without exposing the value.
Read articleA production runbook for proving which Azure Firewall rule handles a flow, separating routing from policy order, and applying a targeted fix with validation and rollback.
Read articleA production runbook for building and qualifying an Azure Monitor burn-rate alert by separating the SLI, error budget, short and long windows, telemetry quality, notification, validation and rollback.
Read articleA production runbook for qualifying poisoned Queue trigger messages, separating deterministic and transient failures, proving idempotency, then replaying or rolling back without duplicating business side effects.
Read articleA production runbook for canarying a KQL transformation in an Azure Monitor Data Collection Rule, comparing volume and schema, detecting rejected records, then validating or rolling back without an observability gap.
Read articleA production runbook for bounding a BGP change, comparing learned and advertised routes, checking effective routes, canarying one prefix, then validating or withdrawing it without destabilizing the Azure hub.
Read articleA production runbook for diagnosing an AKS drain blocked by a PodDisruptionBudget, recovering disruption headroom, fixing capacity or readiness, then validating or postponing the upgrade without removing safeguards.
Read articleA production runbook for recovering Application Insights and OpenTelemetry correlation across Azure Service Bus, qualifying W3C propagation, sampling and message processing, then validating or rolling back the fix.
Read articleA production runbook for reviewing managed resources, unmanage behavior, deny settings, excluded identities, validation and rollback before updating an Azure Deployment Stack.
Read articleA production runbook for qualifying an Azure Container Apps Job that no longer consumes correctly with KEDA scale rule, backlog, managed identity, secrets, logs, idempotency, validation and rollback before rerunning workers.
Read articleA production runbook for qualifying a blocked Azure flow with Network Watcher, NSG Flow Logs, effective security rules, routes, firewall, KQL evidence, validation and rollback before opening a broad rule.
Read articleA production runbook for qualifying Azure Firewall Threat Intelligence with logs, false positives, critical flows, targeted exceptions, deny decision and rollback before enabling Deny.
Read articleA production runbook for qualifying an Azure API Management regression with APIM traces, policy diff, headers, cache, backend, identity, logs, validation and rollback before changing the API or target service.
Read articleA production runbook for qualifying an Azure WAF OWASP/CRS managed rule upgrade with logs, false positives, Detection mode, policy diff, application validation, Prevention decision and rollback.
Read articleA production runbook for qualifying an Azure API Management self-hosted gateway with configuration sync, token or Workload Identity, Kubernetes, logs, backend reachability, validation and rollback before moving traffic.
Read articleA production runbook for qualifying a stuck Azure Durable Functions orchestration with instance history, activity state, storage, idempotency, KQL, validation and rollback before replay, terminate or purge.
Read articleA production runbook for qualifying Azure Container Apps 401/403 failures with Managed Identity, token evidence, RBAC scope, Key Vault, ACR, logs, validation and rollback before widening permissions.
Read articleA production runbook for qualifying an Azure Storage lifecycle policy with Blob inventory, versions, snapshots, delete or tiering rules, evidence, validation and rollback before removing data.
Read articleA production runbook for qualifying an Azure Container Apps incident after deployment by separating active revisions, traffic weights, labels, logs, probes, dependencies, validation and rollback.
Read articleA production runbook for qualifying a Conditional Access block on a workload identity with service principal logs, policy scope, risk, CI identity, KQL evidence, bounded exception and rollback.
Read articleA production runbook for qualifying a failed Azure Functions Timer Trigger with execution history, schedule locks, storage, Application Insights, idempotency, validation and rollback before manual replay.
Read articleA production runbook for qualifying Azure DevOps Variable Group drift with variables, secrets, Key Vault, scopes, logs, validation and rollback before rerunning a pipeline.
Read articleA production runbook for qualifying Azure Workbooks drift with KQL, Log Analytics, Application Insights, dimensions, ingestion latency, validation and rollback before changing alerts or dashboards.
Read articleA production runbook for qualifying Azure Front Door origin health with probes, routing, DNS, TLS, WAF evidence, backend logs, validation and rollback before forcing failover.
Read articleA production runbook for qualifying Azure WAF rate limiting with logs, counting key, real clients, false positives, load checks, threshold decision and rollback.
Read articleA production runbook for qualifying Azure API Management mTLS failures with client certificate chain, hostname, policy, gateway logs, backend identity, validation and rollback before changing API policies.
Read articleA production runbook for qualifying apparent Application Insights telemetry loss with sampling, ingestion, SDK configuration, KQL, alerts, validation and rollback before changing thresholds.
Read articleA production runbook for qualifying an Azure Service Health or Resource Health signal with user impact, dependencies, routing, DNS, observability, failover decision and rollback.
Read articleA production runbook for qualifying an Azure Backup restore point with application consistency, dependencies, identity, networking, test restore, evidence, decision and rollback before cutover.
Read articleA production runbook for qualifying an Azure Monitor Action Group by separating rules, receivers, webhooks, escalation paths, KQL evidence, validation and rollback before reducing notifications.
Read articleA production runbook for qualifying an Azure App Service Managed Identity failure with IMDS endpoint, Entra ID, RBAC, Key Vault or target API evidence, logs, validation and rollback before reintroducing a client secret.
Read articleA production runbook for tightening an Azure Firewall rule without breaking useful traffic, with inventory, KQL evidence, dependencies, controlled validation and rollback.
Read articleA production runbook for qualifying Azure API Management 401/403 errors with subscription key, product, API, consumer, logs, validation and rollback before rotation.
Read articleA production runbook for qualifying a delayed Azure Monitor alert by separating application timestamps, Log Analytics ingestion, KQL query windows, evaluation frequency, action groups, validation and rollback.
Read articleA production runbook for qualifying App Service egress IP drift with VNet Integration, NAT Gateway, DNS, UDRs, destination logs, validation and rollback before changing a partner allowlist.
Read articleA production runbook for qualifying a failed Azure Logic Apps execution with trigger, connectors, managed identity, payload, logs, validation and rollback before resubmit.
Read articleA production runbook for qualifying Cosmos DB 429 throttling with RU consumption, hot partitions, query shape, SDK retries, indexing, KQL evidence, scaling decision and rollback.
Read articleA production runbook for qualifying Key Vault degradation by separating latency, throttling, identity, network path, application cache, logs and rollback before starting a secret rotation.
Read articleA production runbook for qualifying an Azure Policy deployment deny with assignment, initiative, deny effect, compliance, scoped exemption, validation and rollback.
Read articleA production runbook for qualifying an unhealthy Azure Load Balancer backend with probe behavior, NSGs, routing, logs, pool configuration, validation and rollback before changing the rule or redeploying.
Read articleA production runbook for qualifying an Application Gateway TLS certificate rotation from Key Vault with managed identity, listener checks, WAF evidence, probes, logs, validation and rollback.
Read articleA production runbook for qualifying a Terraform, Bicep or ARM plan before apply with drift, CI identity, Azure Policy, destructive changes, validation and rollback.
Read articleA production runbook for qualifying Azure OpenAI or Microsoft Foundry throttling with quota, deployment capacity, agent traces, retries, fallback, validation and rollback before changing models.
Read articleA production runbook for qualifying an Azure Managed Grafana dashboard or alert with Azure Monitor datasource, managed identity, Log Analytics permissions, variables, traces, validation and rollback before changing KQL queries.
Read articleA production runbook for qualifying an Azure Front Door failure with origin groups, health probes, TLS, Private Link, DNS, WAF, logs, validation and rollback before changing routing.
Read articleA production runbook for qualifying Azure Event Grid delivery failures with subscription filters, endpoint health, dead-letter storage, diagnostics, replay scope, validation and rollback before reprocessing events.
Read articleA production runbook for qualifying a Key Vault reference failure with managed identity, RBAC, secret version, diagnostics, KQL, validation and rollback before rotating or broadening access.
Read articleA production runbook for qualifying an Azure Monitor alert that fired but did not notify anyone, with action groups, receivers, processing rules, webhooks, evidence, validation and rollback.
Read articleA production runbook for qualifying Azure asymmetric routing with effective routes, UDRs, Azure Firewall, NAT Gateway, flow logs, validation and rollback before changing route tables.
Read articleA production runbook for qualifying an Azure Automation failure with managed identity, parameters, modules, Hybrid Worker, webhooks, logs, validation and rollback before rerun.
Read articleA production runbook for qualifying a scheduled Azure Automation job with trigger, managed identity, parameters, dry-run, logs, validation and rollback before allowing real action.
Read articleA production runbook for qualifying an APIM secret rotation with Named Values, Key Vault, managed identity, private backend, logs, application validation and rollback.
Read articleA production runbook for qualifying an Azure App Configuration or feature flag regression with labels, identities, refresh, Key Vault references, logs, validation and rollback.
Read articleA production runbook for qualifying an Azure App Service slot swap with configuration drift, warmup, identity, dependencies, logs, validation and rollback before promotion.
Read articleA production runbook for qualifying an Azure egress failure with Firewall DNS proxy, FQDN resolution, UDRs, application rules, logs, validation and rollback before broadening traffic.
Read articleA production runbook for qualifying an ACR image pull failure with identity, network, DNS, firewall, logs, node cache, validation and rollback before rerunning deployment.
Read articleA production runbook for qualifying missing Azure Monitor logs with Diagnostic Settings, DCRs, ingestion, KQL, cost controls, validation and rollback before changing alerts.
Read articleA production runbook for qualifying Azure RBAC drift with the real identity, scope, PIM, role assignments, deny assignments, logs, validation and rollback before adding permissions.
Read articleA production runbook for qualifying an Azure Private DNS change with workload-side resolution, VNet links, Private Resolver, TTL, logs, application validation and rollback.
Read articleA production runbook for qualifying an Azure OIDC authentication failure in a CI pipeline with federated credentials, claims, roles, logs, validation and rollback.
Read articleA production runbook for qualifying Azure hybrid DNS incidents with Private DNS Resolver, forwarding rulesets, private zones, caches, evidence, validation and rollback.
Read articleA production runbook for qualifying Azure egress failures with UDR, NAT Gateway, NSG, firewall, DNS, KQL, validation and rollback before adding an overbroad rule.
Read articleA production runbook for documenting an Azure WAF change with KQL evidence, policy diff, application probes, security validation, merge decision and rollback.
Read articleA short query to isolate rule, field, URI and examples before proposing a targeted WAF exclusion.
Read articleA production runbook for qualifying an Azure deployment failure from a private runner by separating identity, DNS, routing, NSG, NAT, dependency access, logs and rollback.
Read articleA short query to separate missing consumption, retries, poison messages and access errors on a private Storage Queue.
Read articleA production runbook for qualifying Azure Container Apps outbound failures by separating DNS, UDR, NSG, NAT Gateway, firewall, identity, logs and rollback before changing the application.
Read articleA production runbook for deciding an Azure rollback after deployment with Azure Monitor, KQL, impact correlation, regression evidence, validation and controlled recovery.
Read articleA short query to find federated principal sign-in failures before falling back to a client secret.
Read articleA production runbook for running Azure Policy remediation with impact preview, managed identity checks, controlled batches, exemptions, logs and rollback.
Read articleA production runbook for qualifying an Azure WAF policy before switching to Prevention with KQL evidence, change scope, application validation, rollback window and an operable decision.
Read articleA short query to find writes, deletes and destination changes that explain a sudden disappearance of logs.
Read articleA production runbook for qualifying blocked Azure traffic by separating NSG, UDR, Azure Firewall, DNS, effective routes, logs and rollback before opening access too broadly.
Read articleA short query to separate subscription or key issues, private backend routing, APIM policy failure and application-side API errors.
Read articleA production runbook for qualifying an Azure Monitor alert storm after deployment by separating real signal, noise, regression, threshold drift, action group behavior and rollback.
Read articleA short query to separate application rule denial, network rule denial and DNS proxy resolution before adding an overbroad rule.
Read articleAn operational runbook for qualifying Azure outbound incidents involving NAT Gateway, SNAT ports, UDR, NSG, firewalls, allowlists and rollback.
Read articleAn operational runbook for qualifying Azure App Service or Functions outbound failures by separating VNet Integration, DNS, UDR, NSG, NAT, logs and rollback.
Read articleA short sequence to confirm private zone, VNet link, ruleset and forwarding rule before touching DNS records.
Read articleA short query to separate unavailable backend, application failure and WAF blocking when Application Gateway returns 502s on a private path.
Read articleAn operational runbook for Azure Service Bus private access incidents by separating DNS, Private Endpoint, identity, firewall, metrics, logs and rollback.
Read articleA short query to separate missing connection evidence, identity denial, expired SAS and processing errors during a private Service Bus incident.
Read articleAn operational runbook for Azure SQL private access failures by separating DNS, Private Endpoint, firewall, identity, SQL logs and rollback evidence.
Read articleA short query to separate firewall, authentication, public endpoint and missing connection evidence during a private Azure SQL incident.
Read articleAn operational runbook for Azure Storage private access failures by separating DNS, Private Endpoint, firewall, identity, logs and rollback evidence.
Read articleA short query to separate identity, firewall, public endpoint and wrong subresource when private Azure Storage access is denied.
Read articleBuild an operational runbook for App Service private access failures by separating DNS, Private Endpoint, access restrictions, Application Gateway, application logs and rollback evidence.
Read articleA short query to separate WAF, gateway, private DNS, access restrictions and App Service logs during a private access incident.
Read articleBuild an operational runbook for private Azure Functions failures by separating DNS, Private Endpoint, access restrictions, private storage, Application Insights logs and rollback evidence.
Read articleA short query to separate DNS, private access, Functions runtime and application exceptions during a private HTTP incident.
Read articleBuild an operational runbook for AKS private ingress failures by separating DNS, Application Gateway, ingress controller, Kubernetes service endpoints, pod readiness and rollback evidence.
Read articleA short query to read ingress controller and application logs together when a private AKS route returns 502, timeouts or no endpoints.
Read articleBuild an operational runbook for Azure Container Apps private ingress failures by separating DNS, ingress mode, revision routing, application logs and rollback evidence.
Read articleA short query to correlate Azure Container Apps system and console logs when private ingress, probes or revision traffic fail.
Read articleQualify a failure across Application Gateway, WAF, internal APIM and a private backend by separating DNS, routing, policy, identity and logs before any fix.
Read articleA short query to see whether a private API request is blocked by Application Gateway WAF, received by APIM or missing from the expected path.
Read articleBuild a runbook for Key Vault, Storage or private API access failures with managed identity, RBAC, private DNS, logs and real execution evidence.
Read articleA short query to separate identity denial, network path and source address when an Azure workload can no longer access Key Vault.
Read articleBuild useful synthetic probes for DNS, TLS, Application Gateway, WAF and Private Endpoint so private Azure paths fail with evidence before production incidents.
Read articleA short query to track synthetic probe failures and separate DNS, TLS, WAF or Application Gateway symptoms on an Azure private path.
Read articleApply a temporary Azure WAF custom rule with priority, KQL evidence, business validation and rollback, without permanently hiding managed-rule signals.
Read articleA short command to list custom rules in an Azure WAF policy with priority, action and type before an urgent change.
Read articleBuild operable rotation for secrets, certificates, and application identities with dependency inventory, evidence, change windows, monitoring, and rollback.
Read articleBuild an operational drift reading across Terraform, Private Endpoint, private DNS, CI runners, and validation evidence before a private Azure path breaks in production.
Read articleA short check to compare the expected DNS chain, returned private address, and test path from a workload or CI runner.
Read articleA short sequence to confirm that an Azure service exposed through Private Endpoint resolves to a private address from the right network.
Read articleA short KQL query to identify the most blocked URIs by Azure Web Application Firewall on Application Gateway.
Read articleDesign an Azure Terraform backend based on a private Storage Account with CI identity, controlled network access, locking, separate bootstrap, and a diagnostic runbook when init or plan fails.
Read articleAn operational method to analyze access failures to Azure Key Vault behind Private Endpoint by separating DNS resolution, network path, managed identity, RBAC, and application configuration.
Read articleA concrete method to operate Azure Key Vault with private endpoint, private DNS, managed identities, secret rotation, and application-side validation controls.
Read articleA practical method to analyze an Azure WAF block, isolate the rule involved, compare application evidence, and decide between a fix, a targeted exclusion, or a custom rule.
Read articleKnow when to add an Azure WAF custom rule to block or allow precise traffic before managed OWASP/CRS rules, without hiding useful security signals.
Read articleMove from a qualified WAF block to a targeted OWASP/CRS exclusion in an Azure Application Gateway policy, with scope, variable, rule, validation and rollback.
Read articleA KQL analysis method to qualify an Azure WAF block, distinguish attack, noise and application false positive, then document the decision before any exclusion.
Read articleBuild useful KQL queries to identify requests blocked by Azure Web Application Firewall on Application Gateway, with action, ruleId, URI, client IP, hostname and time window.
Read articleA diagnostic method for Azure Application Gateway 502 errors that separates DNS resolution, probes, backend settings, TLS, hostnames, certificates and real application behavior.
Read articleClarify the roles of Private Endpoint, VNet Integration, Application Gateway, API Management, DNS, routing and application authentication in a private Azure architecture.
Read articlePrepare an Azure Private Endpoint production rollout with a validation matrix that separates DNS, routing, public access closure, TLS, application dependencies and tests from Azure and on premises.
Read articleDesign a private API flow where API Management stays internal and calls an Azure Function exposed through Private Endpoint, with private DNS, clear network boundaries and operational checks.
Read articleAn operational walkthrough for publishing a controlled business application behind Azure Application Gateway with dedicated HTTPS listeners, mutual TLS, a scoped WAF policy, HTTPS backends, and network validation points.
Read articleBuilding an Azure AI Foundry agent that reads internal data, triggers controlled actions, and remains operable inside a private network architecture.
Read articleA practical diagnostic procedure for an Azure Linux VM that fails to join Active Directory, with DNS, routing, ports, Kerberos, realmd, and SSSD checks.
Read articleOrganizing Azure Private DNS Zones in a hub and spoke architecture without multiplying links, exceptions, and configurations that become difficult to operate.
Read articleA practical note on Azure App Service and Functions VNet Integration, focused on outbound reachability, DNS, routing, NSGs, UDRs, NAT, and the design mistakes that appear when teams assume the app itself becomes privately exposed.
Read articleA practical runbook-style article on Private Endpoints, private DNS zones, hybrid resolution, validation commands, and the failure patterns that create NXDOMAIN and misleading network diagnostics.
Read article