Azure Chaos Studio: bound a resilience test before disrupting production
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read article
Tag
observability
98 articles connected to this technical signal.
Azure Resource Graph: reconstruct configuration drift before rollback
A production runbook for finding an Azure change with Resource Graph Change Analysis, connecting it to its actor and impact, then validating or rolling it back without reverting blindly.
Read articleAzure Site Recovery: test a recovery plan without touching production
A production runbook for running an Azure Site Recovery test failover in an isolated network, validating dependencies, measuring application recovery and cleaning up unambiguously.
Read articleAgentOps: shadow a prompt or model change before production cutover
A production runbook for comparing a candidate agent with the active runtime on the same requests without duplicating actions, then deciding promotion, canary or rollback.
Read articleAzure DevOps: diagnose a Key Vault secret after rotation before rerunning production
A production runbook for qualifying an Azure Key Vault secret consumed by Azure Pipelines through a variable group, isolating version, mapping, identity, network and precedence, then validating or rolling back without exposing the value.
Read articleAzure Firewall: diagnose a shadowed rule before opening traffic
A production runbook for proving which Azure Firewall rule handles a flow, separating routing from policy order, and applying a targeted fix with validation and rollback.
Read articleAzure Monitor: validate an SLO burn-rate alert before paging on-call
A production runbook for building and qualifying an Azure Monitor burn-rate alert by separating the SLI, error budget, short and long windows, telemetry quality, notification, validation and rollback.
Read articleAzure Functions: diagnose a poison queue before replaying messages
A production runbook for qualifying poisoned Queue trigger messages, separating deterministic and transient failures, proving idempotency, then replaying or rolling back without duplicating business side effects.
Read articleOpenTelemetry Collector: diagnose telemetry drops before adding memory
A production runbook for locating telemetry loss across receivers, processors, queues and exporters, then validating capacity changes or rolling back safely.
Read articleAgentOps: contain an AI agent consumption runaway before shutting down production
A production runbook for attributing token and tool-call growth, isolating amplification, enforcing an execution budget, and validating or rolling back an AI agent release.
Read articleGitHub Actions: prevent overlapping production deployments
A production runbook for diagnosing concurrent GitHub Actions deployments, serializing writes, validating the active version, and resuming or rolling back without adding another race.
Read articleAzure Monitor: roll out a DCR transformation without losing telemetry
A production runbook for canarying a KQL transformation in an Azure Monitor Data Collection Rule, comparing volume and schema, detecting rejected records, then validating or rolling back without an observability gap.
Read articleTerraform: upgrade a provider without uncontrolled drift
A production runbook for qualifying a Terraform provider upgrade with a lockfile, baseline plan, canary, operational evidence and an explicit rollback.
Read articleAgentOps: bound MCP tool timeouts, retries and circuit breakers before production
A production runbook for defining latency budgets, idempotency, retries, circuit breakers, traces and rollback for an MCP tool called by an AI agent.
Read articleAzure Service Bus: repair a broken distributed trace before redeploying
A production runbook for recovering Application Insights and OpenTelemetry correlation across Azure Service Bus, qualifying W3C propagation, sampling and message processing, then validating or rolling back the fix.
Read articleAgentOps: validate tool output before a production write
A production runbook for qualifying structured AI agent tool output with schema, sources, diff, idempotence, policy, traces, human validation and rollback before writing to production.
Read articleAgentOps: diagnose permission drift before widening an agent's rights
A production runbook for qualifying AI agent permission drift with the real identity, RBAC, tool scopes, traces, expected denials, human validation and rollback before widening access.
Read articleAzure Container Apps Jobs: diagnose a KEDA scale rule before rerunning workers
A production runbook for qualifying an Azure Container Apps Job that no longer consumes correctly with KEDA scale rule, backlog, managed identity, secrets, logs, idempotency, validation and rollback before rerunning workers.
Read articleAzure DevOps: diagnose an offline self-hosted agent before recreating it
A production runbook for qualifying an Azure DevOps self-hosted agent that no longer picks up jobs by separating runner service, identity, network, disk, cache, pool, logs and rollback.
Read articleAzure Network Watcher: validate Flow Logs before opening an NSG rule
A production runbook for qualifying a blocked Azure flow with Network Watcher, NSG Flow Logs, effective security rules, routes, firewall, KQL evidence, validation and rollback before opening a broad rule.
Read articleAgentOps: validate the Foundry automation handoff before a production action
A production runbook for qualifying the handoff from a Microsoft Foundry agent to Azure Automation, AWX, Azure DevOps or an MCP tool with contract, identity, traces, dry run, human validation and rollback.
Read articleAzure Firewall: validate Threat Intelligence before blocking production
A production runbook for qualifying Azure Firewall Threat Intelligence with logs, false positives, critical flows, targeted exceptions, deny decision and rollback before enabling Deny.
Read articleAzure APIM: diagnose a policy regression before backend rollback
A production runbook for qualifying an Azure API Management regression with APIM traces, policy diff, headers, cache, backend, identity, logs, validation and rollback before changing the API or target service.
Read articleAzure WAF: validate a managed rule upgrade before switching to Prevention
A production runbook for qualifying an Azure WAF OWASP/CRS managed rule upgrade with logs, false positives, Detection mode, policy diff, application validation, Prevention decision and rollback.
Read articleAzure APIM self-hosted gateway: diagnose sync drift before rerouting traffic
A production runbook for qualifying an Azure API Management self-hosted gateway with configuration sync, token or Workload Identity, Kubernetes, logs, backend reachability, validation and rollback before moving traffic.
Read articleAzure Durable Functions: diagnose an orchestration before replay or purge
A production runbook for qualifying a stuck Azure Durable Functions orchestration with instance history, activity state, storage, idempotency, KQL, validation and rollback before replay, terminate or purge.
Read articleAzure Container Apps: diagnose Managed Identity before widening a role
A production runbook for qualifying Azure Container Apps 401/403 failures with Managed Identity, token evidence, RBAC scope, Key Vault, ACR, logs, validation and rollback before widening permissions.
Read articleAzure Storage: validate lifecycle policy impact before deletion
A production runbook for qualifying an Azure Storage lifecycle policy with Blob inventory, versions, snapshots, delete or tiering rules, evidence, validation and rollback before removing data.
Read articleAzure Container Apps: validate revision traffic before rollback
A production runbook for qualifying an Azure Container Apps incident after deployment by separating active revisions, traffic weights, labels, logs, probes, dependencies, validation and rollback.
Read articleMicrosoft Entra Workload ID: diagnose Conditional Access before excluding a CI pipeline
A production runbook for qualifying a Conditional Access block on a workload identity with service principal logs, policy scope, risk, CI identity, KQL evidence, bounded exception and rollback.
Read articleAgentOps: diagnose MCP server drift before a production action
A production runbook for qualifying MCP server drift with tool manifests, schemas, identity, secrets, network path, traces, evaluations, validation and rollback before reauthorizing an AI agent.
Read articleAzure Functions: diagnose a Timer Trigger before replaying a job
A production runbook for qualifying a failed Azure Functions Timer Trigger with execution history, schedule locks, storage, Application Insights, idempotency, validation and rollback before manual replay.
Read articleAzure DevOps: diagnose a Variable Group before rerunning deployment
A production runbook for qualifying Azure DevOps Variable Group drift with variables, secrets, Key Vault, scopes, logs, validation and rollback before rerunning a pipeline.
Read articleAzure Workbooks: diagnose observability drift before changing alerts
A production runbook for qualifying Azure Workbooks drift with KQL, Log Analytics, Application Insights, dimensions, ingestion latency, validation and rollback before changing alerts or dashboards.
Read articleAzure Front Door: qualify origin health before failing over
A production runbook for qualifying Azure Front Door origin health with probes, routing, DNS, TLS, WAF evidence, backend logs, validation and rollback before forcing failover.
Read articleAzure WAF: diagnose rate limiting before increasing the threshold
A production runbook for qualifying Azure WAF rate limiting with logs, counting key, real clients, false positives, load checks, threshold decision and rollback.
Read articleAgentOps: scope an internal MCP server before exposing it to an agent
A production runbook for qualifying an internal MCP server with tool inventory, scopes, identities, secrets, audit, dry runs, evaluations, human validation and rollback before agent access.
Read articleAzure APIM: diagnose mTLS client certificate failures before changing policies
A production runbook for qualifying Azure API Management mTLS failures with client certificate chain, hostname, policy, gateway logs, backend identity, validation and rollback before changing API policies.
Read articleAzure Application Insights: diagnose sampling before changing alerts
A production runbook for qualifying apparent Application Insights telemetry loss with sampling, ingestion, SDK configuration, KQL, alerts, validation and rollback before changing thresholds.
Read articleAzure Service Health: qualify a regional signal before failover
A production runbook for qualifying an Azure Service Health or Resource Health signal with user impact, dependencies, routing, DNS, observability, failover decision and rollback.
Read articleAgentOps: diagnose a failed tool call before retrying a production action
A production runbook for qualifying a failed AI agent tool call with trace evidence, idempotence, identity, backend state, approvals, validation and rollback before retrying.
Read articleAgentOps: validate agent memory before production actions
A production runbook for qualifying AI agent memory with sources, traces, aging, permissions, evaluations, guardrails, human validation and rollback before it influences real actions.
Read articleAzure Backup: validate a restore point before declaring rollback ready
A production runbook for qualifying an Azure Backup restore point with application consistency, dependencies, identity, networking, test restore, evidence, decision and rollback before cutover.
Read articleAzure Monitor: diagnose an Action Group before silencing alerts
A production runbook for qualifying an Azure Monitor Action Group by separating rules, receivers, webhooks, escalation paths, KQL evidence, validation and rollback before reducing notifications.
Read articleAzure App Service: diagnose a Managed Identity token failure before bringing back a client secret
A production runbook for qualifying an Azure App Service Managed Identity failure with IMDS endpoint, Entra ID, RBAC, Key Vault or target API evidence, logs, validation and rollback before reintroducing a client secret.
Read articleAzure DevOps: diagnose a self-hosted agent before rerunning the pipeline
A production runbook for qualifying an Azure DevOps failure on a self-hosted agent with disk, workspace, cache, local services, identity, logs, controlled cleanup, validation and rollback.
Read articleAzure Firewall: qualify a broad rule before tightening production access
A production runbook for tightening an Azure Firewall rule without breaking useful traffic, with inventory, KQL evidence, dependencies, controlled validation and rollback.
Read articleAzure APIM: diagnose a subscription key before rotating it in production
A production runbook for qualifying Azure API Management 401/403 errors with subscription key, product, API, consumer, logs, validation and rollback before rotation.
Read articleAzure Monitor: diagnose ingestion latency before changing a KQL alert
A production runbook for qualifying a delayed Azure Monitor alert by separating application timestamps, Log Analytics ingestion, KQL query windows, evaluation frequency, action groups, validation and rollback.
Read articleAzure App Service: diagnose egress IP drift before changing an allowlist
A production runbook for qualifying App Service egress IP drift with VNet Integration, NAT Gateway, DNS, UDRs, destination logs, validation and rollback before changing a partner allowlist.
Read articleAzure Logic Apps: diagnose a workflow before resubmitting the run
A production runbook for qualifying a failed Azure Logic Apps execution with trigger, connectors, managed identity, payload, logs, validation and rollback before resubmit.
Read articleAzure Cosmos DB: diagnose 429 throttling before increasing RU/s
A production runbook for qualifying Cosmos DB 429 throttling with RU consumption, hot partitions, query shape, SDK retries, indexing, KQL evidence, scaling decision and rollback.
Read articleMicrosoft Foundry: validate guardrails before exposing an agent to production
A production runbook for qualifying Microsoft Foundry agent guardrails with sources, refusals, tools, identity, traces, canary, human validation and rollback before user exposure.
Read articleAgentOps: diagnose prompt injection in retrieval before production actions
A production runbook for qualifying suspected prompt injection in an AI agent retrieval corpus with sources, traces, evaluation, tools, guardrails, validation and rollback.
Read articleAzure Key Vault: diagnose latency and throttling before rotating secrets
A production runbook for qualifying Key Vault degradation by separating latency, throttling, identity, network path, application cache, logs and rollback before starting a secret rotation.
Read articleAzure DevOps: validate approvals and checks before bypassing production
A production runbook for qualifying an Azure DevOps deployment blocked by approvals or checks with environment scope, identity, logs, audit evidence, validation and rollback before bypassing the guardrail.
Read articleAzure Policy: diagnose a deny before creating a production exemption
A production runbook for qualifying an Azure Policy deployment deny with assignment, initiative, deny effect, compliance, scoped exemption, validation and rollback.
Read articleAgentOps: validate an approval policy before production actions
A production runbook for qualifying an agent approval policy change with action scope, identity, traces, evaluation cases, guardrails, human validation and rollback.
Read articleAzure Load Balancer: diagnose a health probe before changing the backend pool
A production runbook for qualifying an unhealthy Azure Load Balancer backend with probe behavior, NSGs, routing, logs, pool configuration, validation and rollback before changing the rule or redeploying.
Read articleAzure Application Gateway: validate a Key Vault certificate before rotation
A production runbook for qualifying an Application Gateway TLS certificate rotation from Key Vault with managed identity, listener checks, WAF evidence, probes, logs, validation and rollback.
Read articleAzure IaC: validate an infrastructure plan before production apply
A production runbook for qualifying a Terraform, Bicep or ARM plan before apply with drift, CI identity, Azure Policy, destructive changes, validation and rollback.
Read articleAgentOps: diagnose Azure OpenAI rate limits before changing models
A production runbook for qualifying Azure OpenAI or Microsoft Foundry throttling with quota, deployment capacity, agent traces, retries, fallback, validation and rollback before changing models.
Read articleAzure Managed Grafana: diagnose a dashboard or alert before changing KQL
A production runbook for qualifying an Azure Managed Grafana dashboard or alert with Azure Monitor datasource, managed identity, Log Analytics permissions, variables, traces, validation and rollback before changing KQL queries.
Read articleAgentOps: rotate an agent runtime identity before tool access fails
A production runbook for rotating or revoking an AI agent runtime identity with scoped permissions, dry-run tool calls, traces, approvals, validation and rollback before breaking production actions.
Read articleAzure Front Door: diagnose an origin before changing WAF or routing
A production runbook for qualifying an Azure Front Door failure with origin groups, health probes, TLS, Private Link, DNS, WAF, logs, validation and rollback before changing routing.
Read articleAzure Event Grid: diagnose dead-lettered events before replay
A production runbook for qualifying Azure Event Grid delivery failures with subscription filters, endpoint health, dead-letter storage, diagnostics, replay scope, validation and rollback before reprocessing events.
Read articleAzure Key Vault: diagnose secret references before rotation
A production runbook for qualifying a Key Vault reference failure with managed identity, RBAC, secret version, diagnostics, KQL, validation and rollback before rotating or broadening access.
Read articleAzure Monitor Action Groups: diagnose missing notifications before changing alerts
A production runbook for qualifying an Azure Monitor alert that fired but did not notify anyone, with action groups, receivers, processing rules, webhooks, evidence, validation and rollback.
Read articleAzure route asymmetry: diagnose before changing a UDR
A production runbook for qualifying Azure asymmetric routing with effective routes, UDRs, Azure Firewall, NAT Gateway, flow logs, validation and rollback before changing route tables.
Read articleAzure Automation: diagnose a runbook before rerunning the job
A production runbook for qualifying an Azure Automation failure with managed identity, parameters, modules, Hybrid Worker, webhooks, logs, validation and rollback before rerun.
Read articleAzure Automation: validate a scheduled runbook before it changes production
A production runbook for qualifying a scheduled Azure Automation job with trigger, managed identity, parameters, dry-run, logs, validation and rollback before allowing real action.
Read articleAgentOps: validate an agent tool before it can change production
A production runbook for qualifying a new AI agent tool with contract review, scoped identity, dry run, traces, approvals, evaluation cases and rollback before enabling real actions.
Read articleAzure APIM: validate a Named Values rotation before breaking a private API
A production runbook for qualifying an APIM secret rotation with Named Values, Key Vault, managed identity, private backend, logs, application validation and rollback.
Read articleAgentOps: validate a retrieval index update before it changes production answers
A production runbook for qualifying a retrieval index update with source diff, metadata, chunking, evaluations, traces, human validation and rollback before changing an AI agent's answers.
Read articleAzure App Configuration: diagnose a feature flag before rollback
A production runbook for qualifying an Azure App Configuration or feature flag regression with labels, identities, refresh, Key Vault references, logs, validation and rollback.
Read articleAzure App Service slots: validate a swap before promoting to production
A production runbook for qualifying an Azure App Service slot swap with configuration drift, warmup, identity, dependencies, logs, validation and rollback before promotion.
Read articleAzure Firewall DNS proxy: diagnose egress before opening rules
A production runbook for qualifying an Azure egress failure with Firewall DNS proxy, FQDN resolution, UDRs, application rules, logs, validation and rollback before broadening traffic.
Read articleAgentOps: diagnose missing agent traces before restoring a production action
A production runbook for qualifying incomplete AI agent traces with conversation events, sources, tool calls, identities, approvals, evaluations and rollback before restoring an action.
Read articleAzure Monitor: diagnose missing logs before changing alerts
A production runbook for qualifying missing Azure Monitor logs with Diagnostic Settings, DCRs, ingestion, KQL, cost controls, validation and rollback before changing alerts.
Read articleAzure RBAC: diagnose authorization drift before widening a role
A production runbook for qualifying Azure RBAC drift with the real identity, scope, PIM, role assignments, deny assignments, logs, validation and rollback before adding permissions.
Read articleAzure DevOps MCP: scope an agent before letting it act on the project
A production runbook for exposing Azure DevOps to an AI agent through MCP with project scope, permissions, allowed actions, evidence, logs, human validation and rollback.
Read articleAzure Private DNS: validate a zone change before breaking private paths
A production runbook for qualifying an Azure Private DNS change with workload-side resolution, VNet links, Private Resolver, TTL, logs, application validation and rollback.
Read articleAzure Workload Identity Federation: diagnose CI authentication before bringing back a secret
A production runbook for qualifying an Azure OIDC authentication failure in a CI pipeline with federated credentials, claims, roles, logs, validation and rollback.
Read articleAgentOps: diagnose agent contract drift before redeployment
A production runbook for qualifying AI agent contract drift with prompts, tool manifests, sources, evaluations, traces, human validation and rollback.
Read articleAzure Private DNS Resolver: diagnose split-brain DNS before changing zones
A production runbook for qualifying Azure hybrid DNS incidents with Private DNS Resolver, forwarding rulesets, private zones, caches, evidence, validation and rollback.
Read articleAzure UDR and NAT Gateway: diagnose egress before opening the firewall
A production runbook for qualifying Azure egress failures with UDR, NAT Gateway, NSG, firewall, DNS, KQL, validation and rollback before adding an overbroad rule.
Read articleAzure WAF: prepare an evidence pack before a policy PR
A production runbook for documenting an Azure WAF change with KQL evidence, policy diff, application probes, security validation, merge decision and rollback.
Read articleAgentOps: diagnose an AI agent action before rollback
A production runbook for qualifying an AI agent action with traces, sources, tools, identity, KQL, human validation and rollback without disabling the whole assistant.
Read articleAzure Container Apps: diagnose outbound egress before changing code
A production runbook for qualifying Azure Container Apps outbound failures by separating DNS, UDR, NSG, NAT Gateway, firewall, identity, logs and rollback before changing the application.
Read articleAzure Monitor and KQL: decide a deployment rollback without silencing alerts
A production runbook for deciding an Azure rollback after deployment with Azure Monitor, KQL, impact correlation, regression evidence, validation and controlled recovery.
Read articleAzure Policy remediation: fix drift without breaking production
A production runbook for running Azure Policy remediation with impact preview, managed identity checks, controlled batches, exemptions, logs and rollback.
Read articleAzure WAF: move a policy from Detection to Prevention without breaking traffic
A production runbook for qualifying an Azure WAF policy before switching to Prevention with KQL evidence, change scope, application validation, rollback window and an operable decision.
Read articleAzure NSG, UDR and Firewall: diagnose blocked traffic before adding a rule
A production runbook for qualifying blocked Azure traffic by separating NSG, UDR, Azure Firewall, DNS, effective routes, logs and rollback before opening access too broadly.
Read articleAgentOps: expose MCP tools without losing control of production actions
A production runbook for exposing MCP tools to an AI agent while keeping action scope, approvals, identities, logs, evaluation and rollback under control.
Read articleAzure Monitor: diagnose an alert storm after deployment
A production runbook for qualifying an Azure Monitor alert storm after deployment by separating real signal, noise, regression, threshold drift, action group behavior and rollback.
Read articleAzure NAT Gateway: diagnose SNAT exhaustion and outbound IP drift
An operational runbook for qualifying Azure outbound incidents involving NAT Gateway, SNAT ports, UDR, NSG, firewalls, allowlists and rollback.
Read articleAgentOps: diagnose an AI agent that calls the wrong tool
A production runbook for qualifying an AI agent that selects the wrong tool, acts without evidence or hides an action behind a plausible answer.
Read articleMonitoring: turn an alert into an actionable operations runbook
Build useful alerts by connecting signal, diagnosis, scope, decision and rollback path instead of accumulating noisy notifications.
Read article