The Database Was Running. The Books Were Still Broken.
The Database Was Running. The Books Were Still Broken.
08:41 AM.
The finance team is waiting.
"We can't reconcile this morning's books."
You open the accounting dashboard. Every request to /balance is failing—sometimes 500, sometimes nothing but connection refused.
You check the containers.
They're there.
Then they restart.
Again.
And again.
The API looks suspicious, but the logs point somewhere else. You follow the request down the stack and find the database container.
Postgres is running.
But something feels wrong.
There are no ledger tables. No seed data. Nothing the accounting API expects to find.
How can a database be running and still be completely unusable?
You check the configuration. Then the Compose file. Then the initialization directory.
That's when you notice something easy to miss.
The SQL files are present. The bootstrap logic is correct. But Postgres doesn't simply run those scripts every time the container starts. Initialization happens only when its data directory is empty.
The stack had been rebuilt.
The data volume hadn't.
So the database came back healthy—with none of the schema the application needed.
You fix that layer.
The API finally responds.
HTTP 200.
Balance: 1500.00.
But there's still one more requirement hiding in the incident: the ledger needs a unique constraint that isn't created by the initial bootstrap at all.
A second failure. A second layer.
And that's the point.
Production troubleshooting isn't about finding the first broken container. It's about understanding why the system reached that state.
At InfraThrone, you don't get a clean architecture diagram and a list of steps. You get the incident, the symptoms, and the responsibility to bring the system back.
Think you could find what the on-call engineer missed?
Discussion
to read and post comments.