The Hotfix That Fixed Nothing... Because Nobody Knew What Was Running

# The Hotfix That Fixed Nothing... Because Nobody Knew What Was Running

Friday evening.

A critical bug lands in production just minutes before the weekend.

The engineering team moves fast. A developer pulls an older, stable image, applies a tiny hotfix, and pushes it back to the registry as :latest.

The deployment succeeds.

Alerts calm down.

Everyone heads home.

Monday morning arrives with a new mystery.

The same service starts behaving strangely again. Monitoring confidently reports:

Service X is running latest.

Perfect.

Except...

Which latest?

The one built two hours ago?

The emergency hotfix from Friday?

Or the image someone quietly re-tagged two weeks ago?

The war room suddenly isn't discussing application logs anymore.

They're arguing over which container is actually running.

One engineer checks Kubernetes.

Another opens the container registry.

Someone compares image SHAs.

Another searches CI logs from last week.

The application isn't hiding the bug.

The deployment history is.

Eventually, the truth surfaces.

The CI pipeline had been tagging every build as both :latest and :${GIT\_SHA}. During urgent fixes, developers simply re-pointed :latest to an older image without creating a new immutable release. Kubernetes happily pulled the tag, monitoring displayed "latest," and every dashboard became technically correct—but practically useless.

The cluster wasn't inconsistent.

The versioning strategy was.

This is exactly why production teams avoid treating mutable tags as deployment identities.

Instead, releases should carry immutable semantic versions like v2.4.1, while latest remains only a convenience tag. Every deployment should reference immutable image digests or versioned tags, making every rollout reproducible and every rollback predictable. Release branches, protected registry policies, and OCI manifest digests ensure that what was deployed yesterday is still identifiable weeks later.

Production outages rarely begin with broken code.

Sometimes they begin with uncertainty.

And uncertainty is the hardest bug to debug.

At InfraThrone, we recreate production incidents like this—not to teach Docker tags or Semantic Versioning in isolation, but to expose the hidden operational mistakes that surface only under real deployment pressure. Because in production, solving the issue is only half the battle. Knowing exactly what is running is the other half.