The HPA That Never Breathed

The HPA That Never Breathed

2:13 AM.

The alert wasn't dramatic.

CPU was climbing. Requests were piling up. The application was getting slower by the minute.

Someone checked the HPA.

TARGETS: <unknown>/80%

Then came the obvious thought:

"Why isn't Kubernetes scaling?"

The first fix was simple. The deployment had no CPU requests, so the HPA had nothing meaningful to calculate utilization against.

CPU requests were added.

The HPA started breathing again.

But it still didn't scale.

MIN: 1 | MAX: 1

The application had nowhere to go.

The ceiling was the problem.

maxReplicas was raised to six.

Everyone waited.

Still one pod.

Now the room got interesting.

The HPA wanted more replicas. Kubernetes couldn't place them.

The deployment had required pod anti-affinity—and there was only one node.

The application had effectively been told:

"Scale out, but don't you dare place the new pod here."

The anti-affinity rule was relaxed to a preferred rule.

Pods finally started appearing.

Except traffic still wasn't spreading.

The service was using ClientIP session affinity.

Requests kept returning to the same pod.

So even with multiple replicas, one pod was doing most of the work.

Then came the final clue.

Inside the application, a global lock was protecting far more than the shared counter. Slow store reads and CPU computation were happening while holding that lock.

The application had multiple threads.

But effectively, it was behaving like one.

The lock was narrowed to only the shared-state mutation.

Suddenly CPU rose.

Requests distributed.

The HPA saw the pressure.

2 → 3 → 4 → 5 pods.

The dashboard finally turned green.

The outage wasn't caused by a broken HPA.

It was an entire chain of small decisions preventing the HPA from doing its job.

That's what makes production debugging difficult.

The symptom is often one layer.

The cause can be five layers deeper.

At InfraThrone, we build these kinds of incidents into hands-on environments—where Kubernetes doesn't simply tell you what's wrong, and the fix isn't a copy-paste command.

Because learning kubectl is easy.

Learning how to think at 2 AM when Kubernetes is lying to you is the real skill.

The question is: would you have found all five layers?