The Kubernetes Upgrade That Never Finished

# The Kubernetes Upgrade That Never Finished

Quarterly infrastructure maintenance.

Everything had been planned.

The node pool upgrade was approved, stakeholders had been notified, and the production Kubernetes cluster was finally getting its long-awaited security patches. It was supposed to be one of those "monitor for 20 minutes and grab a coffee" maintenance windows.

The first node entered Draining.

Pods started moving.

Deployments rolled smoothly.

Then the progress bar... stopped.

One Kafka pod refused to leave.

"That's odd."

The engineer retried the drain.

Still blocked.

Another attempt.

Same result.

The node sat there, forever stuck in Draining.

Now the surprise began.

Every remaining node upgrade was waiting behind it. New workloads couldn't be scheduled there. Cluster autoscaling paused. Maintenance windows stretched longer, and the operations channel slowly filled with a familiar question:

"Why is a single pod holding the entire cluster hostage?"

Nobody had deleted anything.

Nothing had crashed.

Kafka itself looked perfectly healthy.

Yet the upgrade had completely frozen.

Hours later, someone finally spotted a tiny configuration that had quietly taken control of the entire maintenance operation.

A seemingly harmless PodDisruptionBudget.

The Kafka cluster had three replicas with:

minAvailable: 2

During node draining, Kubernetes politely asked if one Kafka pod could be evicted.

The PodDisruptionBudget answered:

"No."

Without that eviction, the node couldn't finish draining.

Without draining, the node couldn't upgrade.

Without upgrading, every other workload sharing that node remained trapped.

Ironically, the very mechanism designed to protect application availability had now blocked infrastructure maintenance altogether.

This wasn't a Kubernetes bug.

It was a coordination problem.

Rolling node upgrades, PodDisruptionBudgets, drain logic, graceful preStop hooks, and StatefulSets all behaved exactly as designed—but nobody had taught them how to cooperate during maintenance.

Production failures rarely come from one broken component. They emerge when individually correct configurations collide under real operational pressure.

At InfraThrone, we recreate production incidents like these instead of teaching Kubernetes features in isolation. Because knowing what a PodDisruptionBudget does is one thing. Knowing why an entire node upgrade can remain frozen for hours because of it is the kind of experience that prepares you for the incidents documentation never fully captures.