Infrastructure
Azure AKS: unblock a node pool upgrade stopped by a PDB
A production runbook for diagnosing an AKS drain blocked by a PodDisruptionBudget, recovering disruption headroom, fixing capacity or readiness, then validating or postponing the upgrade without removing safeguards.
An AKS node pool upgrade starts normally, prepares or adds a node, then stalls while draining an existing one. Kubernetes events show rejected evictions and a PodDisruptionBudget with disruptionsAllowed: 0. The tempting response is to delete the PDB or sharply increase maxUnavailable. That is exactly when a useful safeguard can be turned into an application outage.
This runbook covers a production AKS cluster with several workloads, PDBs managed in Git and a limited maintenance window. Its purpose is to distinguish four causes: an actually over-constrained budget, too few replicas, unready pods or unavailable surge capacity. The desired outcome is not merely a moving upgrade. It is a recorded decision to repair, reschedule or restore the previous controls.
Freeze the operation and blocked node
Capture the exact scope first. A “drain failed” message does not identify which workload consumes the budget, whether a replacement node is ready, or whether the problem predates the upgrade.
cluster:
resource_group: rg-platform-prod
name: aks-prod-weu
node_pool: userpool01
target_version: approved-version
operation:
maintenance_window_utc: 2026-08-01T20:00:00Z/2026-08-01T22:00:00Z
blocked_node: aks-userpool01-000123-vmss00000a
first_eviction_failure_utc: 2026-08-01T20:17:00Z
last_known_good_release: orders-api-2026.08.01.2
evidence:
- node_pool_configuration
- node_conditions_and_allocatable_capacity
- warning_events_for_blocked_pods
- pdb_status_and_selectors
- replicas_readiness_and_topology
- decision_and_rollback_ownerCapture Azure and Kubernetes state before changing anything.
RG="rg-platform-prod"
CLUSTER="aks-prod-weu"
POOL="userpool01"
NODE="aks-userpool01-000123-vmss00000a"
az aks nodepool show --resource-group "$RG" --cluster-name "$CLUSTER" --name "$POOL" --output json
kubectl get node "$NODE" -o wide
kubectl describe node "$NODE"
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp
kubectl get pods -A --field-selector spec.nodeName="$NODE" -o wideIf the replacement node is not Ready, the pool cannot allocate another VM, or the subnet has run out of addresses, the PDB may only be the last visible symptom. Address Azure capacity, quotas, IP space and surge-node health first.
Identify the budget rejecting eviction
A PDB protects selected pods from voluntary disruptions, including evictions used during a drain. The key operational field is disruptionsAllowed. At zero, Kubernetes cannot evict another pod while satisfying the current budget.
kubectl get pdb -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,MIN_AVAILABLE:.spec.minAvailable,MAX_UNAVAILABLE:.spec.maxUnavailable,EXPECTED:.status.expectedPods,CURRENT_HEALTHY:.status.currentHealthy,DESIRED_HEALTHY:.status.desiredHealthy,ALLOWED:.status.disruptionsAllowed'
NS="orders-prod"
PDB="orders-api"
kubectl -n "$NS" describe pdb "$PDB"
kubectl -n "$NS" get pdb "$PDB" -o yaml
kubectl -n "$NS" get pods -l app=orders-api -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,READY:.status.containerStatuses[*].ready,PHASE:.status.phase'Do not stop at the PDB name. Inspect its selector and the actual pods it covers. A broad selector, a label shared by two releases or overlapping PDBs can constrain more pods than intended.
Explain why headroom is zero
Connect the PDB to the workload controller, readiness state and pod topology. A reasonable budget can fall to zero because one replica is already unavailable, a readiness probe fails, or all replicas sit on the node being drained.
NS="orders-prod"
APP="orders-api"
kubectl -n "$NS" get deploy,statefulset -l app="$APP" -o wide
kubectl -n "$NS" get pods -l app="$APP" -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount'
kubectl -n "$NS" describe pods -l app="$APP"
kubectl -n "$NS" get events --field-selector type=Warning --sort-by=.lastTimestamp | tail -n 80Classify the cause before choosing a repair.
One replica was unready before the drain
Repair readiness, dependency, secret, image or saturation
Do not relax the PDB to hide an existing failure
All replicas are ready but no placement is available
Check surge node, quotas, subnet, taints, affinity and resource requests
Recover capacity before resuming the upgrade
The replica count allows no disruption
Add one temporary replica if the application and capacity support it
Revalidate readiness and distribution before draining
The PDB no longer matches the service SLO
Change the budget in Git with approval, expiry and explicit rollback
Never delete the PDB as a troubleshooting shortcut
Several PDBs or a broad selector cover the same pod
Repair selectors and ownership
Prove the effective budget before resumingRecover headroom without disabling protection
The preferred repair is to make an additional replica healthy and schedulable. This preserves the PDB’s intent instead of bypassing it. Check downstream capacity before scaling: database connections, partitions, licences, queues and external limits may not accept another replica.
NS="orders-prod"
DEPLOYMENT="orders-api"
PDB="orders-api"
kubectl -n "$NS" get deployment "$DEPLOYMENT" -o yaml
kubectl -n "$NS" get pdb "$PDB" -o yaml
kubectl -n "$NS" scale deployment "$DEPLOYMENT" --replicas=4
kubectl -n "$NS" rollout status deployment "$DEPLOYMENT" --timeout=10m
kubectl -n "$NS" get pdb "$PDB" -o jsonpath='{.status.currentHealthy}{" current / "}{.status.desiredHealthy}{" desired; disruptionsAllowed="}{.status.disruptionsAllowed}{"
"}'
kubectl -n "$NS" get pods -l app=orders-api -o wideResume only when the new pod is Ready, scheduled away from the blocked node and disruptionsAllowed is strictly positive. A Pending replica creates no disruption headroom.
If the pool lacks replacement capacity, adjust surge through the team’s established IaC or administration path. Check VM quotas, subnet addresses, pod limits and zonal capacity first. Raising maxSurge without those controls merely moves the failure to provisioning.
Relax the PDB only as a production change
Sometimes the budget no longer matches the SLO or actual replica count. Treat its modification as a production change: reviewed diff, owner, window, health signals, stop condition and scheduled or automatic restoration.
change:
object: orders-prod/PodDisruptionBudget/orders-api
reason: "Allow one voluntary disruption during approved AKS upgrade"
previous_policy: "minAvailable: 100%"
temporary_policy: "maxUnavailable: 1"
approval: CHG-20260801-042
expires_at_utc: 2026-08-01T22:00:00Z
preconditions:
- all application replicas are ready
- replicas are distributed across at least two nodes
- error rate and latency are within the service baseline
- replacement node is ready and schedulable
stop_conditions:
- available replicas fall below the service SLO
- a replacement pod remains pending or not ready
- dependency saturation increases
- another node becomes unhealthy
restore:
- reapply the previous PDB manifest from Git
- return the temporary replica count to baseline
- verify disruptionsAllowed and application healthKeep the change in the source of truth. A direct edit outside GitOps may be reverted at the wrong time or forgotten after the upgrade.
Validate the resumed upgrade and prepare the return path
After the repair, let AKS continue through its managed upgrade mechanism. During each replacement, watch nodes, pods, PDBs, events and application signals together. A clean Azure portal operation is insufficient if service availability regresses.
Validation
The replacement node is Ready and schedulable
disruptionsAllowed remains compatible with the next eviction
Every evicted pod becomes Ready within the expected time
Replica placement respects topology spread or anti-affinity
Error rate, latency, saturation and backlog stay within baseline
No critical workload remains Pending
The pool target version is confirmed after the operation
Return and postponement
Restore the previous PDB manifest
Return the temporary replica count to baseline after stabilization
Restore the temporary surge setting through IaC
Postpone remaining work if disruption headroom returns to zero
Preserve node, pod, PDB, event and timestamp evidenceA Kubernetes upgrade already applied to a node does not roll back like an application deployment. The return path is to restore temporary controls, stop expanding impact, stabilize workloads and reschedule the remaining work. Do not promise a version downgrade as the safety net.
Conclusion
An AKS drain blocked by a PDB is not an invitation to remove the safeguard. It means the cluster cannot currently prove that one more replica can disappear without violating the declared availability.
Resume only when a replacement pod is Ready, capacity exists and disruptionsAllowed is positive. Otherwise repair readiness or capacity, revise the budget through an approved change, or postpone the upgrade. The correct outcome is an updated node pool with intact protections, not a successful upgrade bought with a silent outage.