The Disk Wasn't Full. The Logs Were Eating the Server

The Disk Wasn't Full. The Logs Were Eating the Server.

2:14 AM.

The alert looked deceptively simple.

Disk usage critical.

Someone checked the node.

The application was still running. The container was healthy. CPU looked normal.

“So it’s probably just a cleanup issue.”

Except it wasn’t.

A quick look at the Docker logs revealed something ugly: the edge-api container was generating an endless stream of logs, and Docker’s json-file logging driver had no size limit.

The disk wasn't being consumed by the application.

The application was consuming its own server.

The team restarted the container with log rotation enabled.

Problem solved?

No.

A few minutes later, the logs started growing again.

This time, the application itself was stuck in a tight reconciliation loop because a configuration key — reconcile.interval_ms — was missing.

Fix the configuration.

Restart.

The disk looked better.

Then another alarm appeared.

app.log was growing rapidly on the tiny filesystem.

Buried inside the application was another problem: log rotation was disabled.

Enable rotation.

Restart.

Finally, someone inspected the actual log lines.

They were enormous.

Every request body was being Base64-encoded and dumped into the logs.

The application wasn't just logging.

It was logging the entire crime scene.

That still wasn't the end.

A metrics thread was performing an fsync() for every sample, creating an I/O storm and competing with everything else for disk access.

Five layers.

Five different causes.

One outage.

And the most dangerous part?

Every fix exposed the next failure.

This is what real production debugging feels like. The first symptom is rarely the root cause. Docker, application configuration, logging behaviour, filesystem limits and I/O patterns can interact until a seemingly harmless issue becomes an outage.

At InfraThrone, we build labs around exactly these situations — not “run this command and get the expected output,” but incidents where you have to peel back the layers and discover what is actually happening.

Because production doesn't give you a checklist.

It gives you 2 AM, a red dashboard, and a system that refuses to tell you why.