Azure Chaos Studio: bound a resilience test before disrupting production
A production runbook for controlling blast radius, execution identity, evidence, abort criteria and recovery for an Azure Chaos Studio scenario.
Read article
Tag
automation
69 articles connected to this technical signal.
Azure Site Recovery: test a recovery plan without touching production
A production runbook for running an Azure Site Recovery test failover in an isolated network, validating dependencies, measuring application recovery and cleaning up unambiguously.
Read articleAgentOps: shadow a prompt or model change before production cutover
A production runbook for comparing a candidate agent with the active runtime on the same requests without duplicating actions, then deciding promotion, canary or rollback.
Read articleAzure DevOps: diagnose a Key Vault secret after rotation before rerunning production
A production runbook for qualifying an Azure Key Vault secret consumed by Azure Pipelines through a variable group, isolating version, mapping, identity, network and precedence, then validating or rolling back without exposing the value.
Read articleAzure Firewall: diagnose a shadowed rule before opening traffic
A production runbook for proving which Azure Firewall rule handles a flow, separating routing from policy order, and applying a targeted fix with validation and rollback.
Read articleAzure Functions: diagnose a poison queue before replaying messages
A production runbook for qualifying poisoned Queue trigger messages, separating deterministic and transient failures, proving idempotency, then replaying or rolling back without duplicating business side effects.
Read articleOpenTelemetry Collector: diagnose telemetry drops before adding memory
A production runbook for locating telemetry loss across receivers, processors, queues and exporters, then validating capacity changes or rolling back safely.
Read articleAgentOps: contain an AI agent consumption runaway before shutting down production
A production runbook for attributing token and tool-call growth, isolating amplification, enforcing an execution budget, and validating or rolling back an AI agent release.
Read articleGitHub Actions: prevent overlapping production deployments
A production runbook for diagnosing concurrent GitHub Actions deployments, serializing writes, validating the active version, and resuming or rolling back without adding another race.
Read articleAzure Monitor: roll out a DCR transformation without losing telemetry
A production runbook for canarying a KQL transformation in an Azure Monitor Data Collection Rule, comparing volume and schema, detecting rejected records, then validating or rolling back without an observability gap.
Read articleTerraform: upgrade a provider without uncontrolled drift
A production runbook for qualifying a Terraform provider upgrade with a lockfile, baseline plan, canary, operational evidence and an explicit rollback.
Read articleAgentOps: bound MCP tool timeouts, retries and circuit breakers before production
A production runbook for defining latency budgets, idempotency, retries, circuit breakers, traces and rollback for an MCP tool called by an AI agent.
Read articleAgentOps: validate tool output before a production write
A production runbook for qualifying structured AI agent tool output with schema, sources, diff, idempotence, policy, traces, human validation and rollback before writing to production.
Read articleAzure Deployment Stacks: validate unmanage and deny settings before an update
A production runbook for reviewing managed resources, unmanage behavior, deny settings, excluded identities, validation and rollback before updating an Azure Deployment Stack.
Read articleAzure Container Apps Jobs: diagnose a KEDA scale rule before rerunning workers
A production runbook for qualifying an Azure Container Apps Job that no longer consumes correctly with KEDA scale rule, backlog, managed identity, secrets, logs, idempotency, validation and rollback before rerunning workers.
Read articleAzure DevOps: diagnose an offline self-hosted agent before recreating it
A production runbook for qualifying an Azure DevOps self-hosted agent that no longer picks up jobs by separating runner service, identity, network, disk, cache, pool, logs and rollback.
Read articleAgentOps: validate the Foundry automation handoff before a production action
A production runbook for qualifying the handoff from a Microsoft Foundry agent to Azure Automation, AWX, Azure DevOps or an MCP tool with contract, identity, traces, dry run, human validation and rollback.
Read articleAzure Durable Functions: diagnose an orchestration before replay or purge
A production runbook for qualifying a stuck Azure Durable Functions orchestration with instance history, activity state, storage, idempotency, KQL, validation and rollback before replay, terminate or purge.
Read articleAzure Functions: diagnose a Timer Trigger before replaying a job
A production runbook for qualifying a failed Azure Functions Timer Trigger with execution history, schedule locks, storage, Application Insights, idempotency, validation and rollback before manual replay.
Read articleAzure DevOps: diagnose a Variable Group before rerunning deployment
A production runbook for qualifying Azure DevOps Variable Group drift with variables, secrets, Key Vault, scopes, logs, validation and rollback before rerunning a pipeline.
Read articleAgentOps: scope an internal MCP server before exposing it to an agent
A production runbook for qualifying an internal MCP server with tool inventory, scopes, identities, secrets, audit, dry runs, evaluations, human validation and rollback before agent access.
Read articleAgentOps: diagnose a failed tool call before retrying a production action
A production runbook for qualifying a failed AI agent tool call with trace evidence, idempotence, identity, backend state, approvals, validation and rollback before retrying.
Read articleAgentOps: validate agent memory before production actions
A production runbook for qualifying AI agent memory with sources, traces, aging, permissions, evaluations, guardrails, human validation and rollback before it influences real actions.
Read articleAzure App Service: diagnose a Managed Identity token failure before bringing back a client secret
A production runbook for qualifying an Azure App Service Managed Identity failure with IMDS endpoint, Entra ID, RBAC, Key Vault or target API evidence, logs, validation and rollback before reintroducing a client secret.
Read articleAzure DevOps: diagnose a self-hosted agent before rerunning the pipeline
A production runbook for qualifying an Azure DevOps failure on a self-hosted agent with disk, workspace, cache, local services, identity, logs, controlled cleanup, validation and rollback.
Read articleAzure Monitor: diagnose ingestion latency before changing a KQL alert
A production runbook for qualifying a delayed Azure Monitor alert by separating application timestamps, Log Analytics ingestion, KQL query windows, evaluation frequency, action groups, validation and rollback.
Read articleAzure Logic Apps: diagnose a workflow before resubmitting the run
A production runbook for qualifying a failed Azure Logic Apps execution with trigger, connectors, managed identity, payload, logs, validation and rollback before resubmit.
Read articleMicrosoft Foundry: validate guardrails before exposing an agent to production
A production runbook for qualifying Microsoft Foundry agent guardrails with sources, refusals, tools, identity, traces, canary, human validation and rollback before user exposure.
Read articleAgentOps: diagnose prompt injection in retrieval before production actions
A production runbook for qualifying suspected prompt injection in an AI agent retrieval corpus with sources, traces, evaluation, tools, guardrails, validation and rollback.
Read articleAzure DevOps: validate approvals and checks before bypassing production
A production runbook for qualifying an Azure DevOps deployment blocked by approvals or checks with environment scope, identity, logs, audit evidence, validation and rollback before bypassing the guardrail.
Read articleAzure Policy: diagnose a deny before creating a production exemption
A production runbook for qualifying an Azure Policy deployment deny with assignment, initiative, deny effect, compliance, scoped exemption, validation and rollback.
Read articleAgentOps: validate an approval policy before production actions
A production runbook for qualifying an agent approval policy change with action scope, identity, traces, evaluation cases, guardrails, human validation and rollback.
Read articleAzure Event Grid: diagnose dead-lettered events before replay
A production runbook for qualifying Azure Event Grid delivery failures with subscription filters, endpoint health, dead-letter storage, diagnostics, replay scope, validation and rollback before reprocessing events.
Read articleAzure Automation: diagnose a runbook before rerunning the job
A production runbook for qualifying an Azure Automation failure with managed identity, parameters, modules, Hybrid Worker, webhooks, logs, validation and rollback before rerun.
Read articleAzure Automation: validate a scheduled runbook before it changes production
A production runbook for qualifying a scheduled Azure Automation job with trigger, managed identity, parameters, dry-run, logs, validation and rollback before allowing real action.
Read articleAzure APIM: validate a Named Values rotation before breaking a private API
A production runbook for qualifying an APIM secret rotation with Named Values, Key Vault, managed identity, private backend, logs, application validation and rollback.
Read articleAzure App Configuration: diagnose a feature flag before rollback
A production runbook for qualifying an Azure App Configuration or feature flag regression with labels, identities, refresh, Key Vault references, logs, validation and rollback.
Read articleAzure App Service slots: validate a swap before promoting to production
A production runbook for qualifying an Azure App Service slot swap with configuration drift, warmup, identity, dependencies, logs, validation and rollback before promotion.
Read articleAgentOps: diagnose missing agent traces before restoring a production action
A production runbook for qualifying incomplete AI agent traces with conversation events, sources, tool calls, identities, approvals, evaluations and rollback before restoring an action.
Read articleAWX: validate dynamic inventory before a production job
A production runbook for qualifying an AWX dynamic inventory change with source, host diff, variables, identities, limits, validation and rollback before execution.
Read articleAzure Monitor: diagnose missing logs before changing alerts
A production runbook for qualifying missing Azure Monitor logs with Diagnostic Settings, DCRs, ingestion, KQL, cost controls, validation and rollback before changing alerts.
Read articleAzure RBAC: diagnose authorization drift before widening a role
A production runbook for qualifying Azure RBAC drift with the real identity, scope, PIM, role assignments, deny assignments, logs, validation and rollback before adding permissions.
Read articleAzure DevOps MCP: scope an agent before letting it act on the project
A production runbook for exposing Azure DevOps to an AI agent through MCP with project scope, permissions, allowed actions, evidence, logs, human validation and rollback.
Read articleAzure Private DNS: validate a zone change before breaking private paths
A production runbook for qualifying an Azure Private DNS change with workload-side resolution, VNet links, Private Resolver, TTL, logs, application validation and rollback.
Read articleAWX: rerun a failed job without replaying a partial action
A production runbook for qualifying a failed AWX job with Ansible events, changed tasks, affected hosts, variables, limited rerun, validation and rollback.
Read articleAWX snippet: capture failed job evidence before rerun
A short sequence to collect status, variables, affected host and error events before rerunning an AWX job.
Read articleAzure Private DNS Resolver: diagnose split-brain DNS before changing zones
A production runbook for qualifying Azure hybrid DNS incidents with Private DNS Resolver, forwarding rulesets, private zones, caches, evidence, validation and rollback.
Read articleKQL snippet: find risky tool actions from an AI agent
A short query to isolate write-capable tool calls, their execution identity and correlation before disabling an AI agent.
Read articleAzure UDR and NAT Gateway: diagnose egress before opening the firewall
A production runbook for qualifying Azure egress failures with UDR, NAT Gateway, NSG, firewall, DNS, KQL, validation and rollback before adding an overbroad rule.
Read articleAzure WAF: prepare an evidence pack before a policy PR
A production runbook for documenting an Azure WAF change with KQL evidence, policy diff, application probes, security validation, merge decision and rollback.
Read articleAgentOps: diagnose an AI agent action before rollback
A production runbook for qualifying an AI agent action with traces, sources, tools, identity, KQL, human validation and rollback without disabling the whole assistant.
Read articleAzure deployment from a private runner: diagnose the pipeline before rerun
A production runbook for qualifying an Azure deployment failure from a private runner by separating identity, DNS, routing, NSG, NAT, dependency access, logs and rollback.
Read articleAzure Monitor and KQL: decide a deployment rollback without silencing alerts
A production runbook for deciding an Azure rollback after deployment with Azure Monitor, KQL, impact correlation, regression evidence, validation and controlled recovery.
Read articleAzure Policy remediation: fix drift without breaking production
A production runbook for running Azure Policy remediation with impact preview, managed identity checks, controlled batches, exemptions, logs and rollback.
Read articleMicrosoft Foundry: evaluate an agent before giving it a production action
A runbook for validating an AI agent before real action by separating sources, tools, identity, evaluation cases, human approvals, logs and rollback.
Read articleAgentOps: expose MCP tools without losing control of production actions
A production runbook for exposing MCP tools to an AI agent while keeping action scope, approvals, identities, logs, evaluation and rollback under control.
Read articleAzure Storage: diagnose a private endpoint without opening the account
An operational runbook for Azure Storage private access failures by separating DNS, Private Endpoint, firewall, identity, logs and rollback evidence.
Read articleAzure App Service: diagnose a private endpoint before redeploying
Build an operational runbook for App Service private access failures by separating DNS, Private Endpoint, access restrictions, Application Gateway, application logs and rollback evidence.
Read articleAzure Functions: diagnose a private HTTP endpoint before changing code
Build an operational runbook for private Azure Functions failures by separating DNS, Private Endpoint, access restrictions, private storage, Application Insights logs and rollback evidence.
Read articleAzure AKS: diagnose private ingress before changing deployments
Build an operational runbook for AKS private ingress failures by separating DNS, Application Gateway, ingress controller, Kubernetes service endpoints, pod readiness and rollback evidence.
Read articleAzure Container Apps: diagnose private ingress before changing revisions
Build an operational runbook for Azure Container Apps private ingress failures by separating DNS, ingress mode, revision routing, application logs and rollback evidence.
Read articleAzure managed identity: diagnose private access before changing permissions
Build a runbook for Key Vault, Storage or private API access failures with managed identity, RBAC, private DNS, logs and real execution evidence.
Read articleAzure: make private paths verifiable with synthetic probes
Build useful synthetic probes for DNS, TLS, Application Gateway, WAF and Private Endpoint so private Azure paths fail with evidence before production incidents.
Read articleAzure snippet: audit WAF custom rule priorities
A short command to list custom rules in an Azure WAF policy with priority, action and type before an urgent change.
Read articleService identity and secret rotation: a production runbook, not an isolated task
Build operable rotation for secrets, certificates, and application identities with dependency inventory, evidence, change windows, monitoring, and rollback.
Read articleAzure Private Endpoint: detect Terraform, DNS, and network drift before incident
Build an operational drift reading across Terraform, Private Endpoint, private DNS, CI runners, and validation evidence before a private Azure path breaks in production.
Read articleAWX: design job templates that do not become a dangerous remote console
Turn AWX into a controlled operations tool with bounded job templates, limited variables, separated credentials, explicit inventories and post-action validation.
Read articleLightweight AWX production without overbuilding
A concrete use case for running AWX on a small Linux estate with Git inventories, separated credentials, controlled job templates, backups, and post-run checks.
Read articleInstalling AWX is easy. Operating it cleanly is not.
A production-oriented note on AWX with the operator, including namespace design, persistence, exposure, execution environments, validation, backups, and the failure modes that appear after the first successful login.
Read article