Hands-On DevOps Labs: Fix 300+ Real Outages

Why most engineers freeze

Tutorials teach YAML.
Production breaks at 3 AM.

ERR03:02:11 never_been_in_a_war_room

Your first outage happens live

Most engineers meet a real incident on the job, with users watching and a manager on the call.

ERR03:02:14 happy_path_only

Courses stop at "it works"

Real tasks mix four or five tools at once. Most courses teach them one at a time, on systems that never break.

ERR03:02:19 no_proof_of_fix

Nothing proves you can fix it

A certificate shows you watched the videos. It doesn't show you brought a broken system back.

3:025G

Wednesday, 14 October

3:02

!

ON-CALLnow

CRITICAL · api-gateway

5xx error rate 42% for 5 min. Checkout is failing.

!

ON-CALLnow

HIGH · payments-db

Replica lag 38s and climbing.

#

WAR ROOMnow

#inc-2041 · you're up

14 people joined and are waiting on you.

slide to acknowledgeAcknowledged · 3:03 AM

How the Park works

Five levels. Every topic climbs the same ladder.

Labs, challenges, incidents and War Zones are four different things, each training a different skill. You move up as you get better.

Tap a level to see what it looks like inside the Park

LVL 01 · DOCS

Every topic opens with a real on-the-job scenario.

1 of 5 levels

INSIDE EVERY TRACK DocumentationLabsChallengesIncidentsProjectsCertification

The main event · War Zones

300+

Real production outages, rebuilt as live broken labs.

The outages teams hit every week, plus the famous ones that took down AWS, Cloudflare, Meta, GitHub and Reddit. You don't read the postmortem. You fix the system.

Kubernetes 82Networking 32Logging 4Observability 4Debugging 3Tcp/Ip 2Service Discovery 2Routing 1

AAtlassian 1AAWS 1CCloudflare 2GGitHub 1GGoogle 1KKnight Capital 1MMeta 1RReddit 1

These are 6 of 300+ War Zones Go to War Zones

Interactive preview of the War Zone catalog. Try the tier tabs and the search.

Inside a War Zone

Pick it. Fix it. Explain it.

Follow one real War Zone from the catalog to a reviewed RCA. Click any step, or let it play.

Step 1 of 6

War ZonesShowing 6 of 300+

AllCoreOpsBattleOpsEliteOps

Kubernetes 82Networking 32Logging 4Observability 4Debugging 3Tcp/Ip 2Service Discovery 2

API Service Selector TLS Retry Cascade

KubernetesSelector-labels

CoreOps30 mins50 SS

Silent Recovery Collapse

KubernetesReliability+6

CoreOps60 mins100 SS

Stale Lease Blocks Leader Election

KubernetesController+1

CoreOps30 mins50 SS

Validating Webhook Admission Stall

KubernetesAdmission controller webhook

CoreOps30 mins50 SS

Mission Brief

API Service Selector TLS Retry Cascade

KubernetesSelector-labelsCoreOps

On-call gets paged because traffic-worker is throwing intermittent TLS handshake failures against api-service, and request latency has climbed as retries pile up. Nothing looks outright down, pods are Running, the Service has…

Incident Brief

Scenario

Calls from traffic-worker to api-service:443 intermittently fail TLS verification (wrong/unexpected certificate CN) instead of cleanly connecting, so the client's retry logic keeps firing and load creeps up with no matching error in the backend logs.

The setup involves default namespace, a Service named api-service, a traffic-worker Deployment acting as the client, and at least two backend pod sets distinguished by a role label (only one of which is the real, correctly-certed backend).

Your Objective

The validator reports success:true, and running openssl s_client against api-service:443 from traffic-worker returns the expected certificate subject every time, with no handshake retries showing up in the client logs.

Scoring

DURATION~30 min

50SS on completion

4 of 5 free incidents used

Start as individual

←API Service Selector TLS Retry Cascadens/lab-cmo4cem4-mv89u6sb

00:00SubmitEnd

API Service Selector TLS Retry Cascade

CopyPaste− 14 +

root@ubuntu-fc-uvm:~# kubectl get endpoints api-service

NAME ENDPOINTS AGE

api-service 10.42.0.15:8443,10.42.0.21:8443 6m

root@ubuntu-fc-uvm:~# kubectl get pods --show-labels

NAME STATUS LABELS

api-7c9f4 Running app=api,role=backend

api-legacy-5d2b Running app=api,role=legacy

root@ubuntu-fc-uvm:~# kubectl describe svc api-service | grep Selector

Selector: app=api ← matches both backends

root@ubuntu-fc-uvm:~# kubectl set selector svc/api-service app=api,role=backend

service/api-service selector updated

root@ubuntu-fc-uvm:~# kubectl get endpoints api-service

api-service 10.42.0.15:8443 7m

root@ubuntu-fc-uvm:~#

MissionHintsKubie

SCENARIO

Scenario

Calls from traffic-worker to api-service:443 intermittently fail TLS verification (wrong/unexpected certificate CN) instead of cleanly connecting…

Diagnosis

kubectl -n default get svc api-service -o yaml
kubectl -n default get pods --show-labels
kubectl -n default get endpoints api-service
kubectl -n default logs deploy/traffic-worker --tail=100

←API Service Selector TLS Retry Cascadens/lab-cmo4cem4-mv89u6sb

18:42SubmitEnd

API Service Selector TLS Retry Cascade

service/api-service selector updated

root@ubuntu-fc-uvm:~# kubectl get endpoints api-service

api-service 10.42.0.15:8443 7m

root@ubuntu-fc-uvm:~#

MissionHintsKubie

Running the validator…

Validator · success: true

api-service selects only the role=backend pods
openssl s_client returns the expected certificate subject
No handshake retries in the traffic-worker logs

+50SS earned

Global rank #128 → #121

ROOT CAUSE ANALYSIS

API Service Selector TLS Retry Cascade

Draft

SUMMARY

traffic-worker hit intermittent TLS failures calling api-service:443. Retries piled up and latency climbed.

ROOT CAUSE

The api-service selector matched only app=api, so it also sent traffic to role=legacy pods serving a certificate with the wrong CN.

FIX

Set the selector to app=api,role=backend. Endpoints now list only the correctly-certed backend.

PREVENTION

Require a role label in Service selectors and alert when a Service's endpoint count changes.

RCA checklist

Impact summary
Root cause with evidence
Fix applied
Prevention

Submit RCA

ROOT CAUSE ANALYSIS

API Service Selector TLS Retry Cascade

Reviewed

SUMMARY

traffic-worker hit intermittent TLS failures calling api-service:443. Retries piled up and latency climbed.

ROOT CAUSE

The api-service selector matched only app=api, so it also sent traffic to role=legacy pods serving a certificate with the wrong CN.

FIX

Set the selector to app=api,role=backend. Endpoints now list only the correctly-certed backend.

PREVENTION

Require a role label in Service selectors and alert when a Service's endpoint count changes.

Reviewer feedback

Strong. You proved it with the endpoints list instead of guessing.

Reviewer feedback

Good call. Also add a readiness check that verifies the certificate CN.

Approved · counts toward your readiness report

Browse 300+ War Zones by tier, topic or company.

Walkthrough built from the live platform screens. Lab output, timings and rank are illustrative.

Roadmaps

Your path to production, mapped.

Follow a role-based roadmap from zero to job-ready, or go deep on one skill at a time. Every roadmap is included, and every lesson ends in a hands-on lab.

ROLE ROADMAPS 1 / 4 Keep scrolling

DevOps Engineering

9 technologies, each with docs, labs, challenges, incidents and a portfolio project.

Start this roadmap

Site Reliability Engineering

6 technologies to keep systems up, measure reliability and lead the room when an incident hits.

Start this roadmap

Cloud Engineering

6 technologies to design, build, secure and run infrastructure on the cloud.

Start this roadmap

MLOps

6 technologies to ship, serve and monitor machine learning systems in production.

Start this roadmap

Inside a skill

‹ DevOps - Engineering Roadmap - Linux

Linux

Master Linux fundamentals, system administration, processes, filesystems, permissions, networking, and troubleshooting for real-world DevOps environments.

Continue Learning ›

112 Modules52 Labs3120 Mins

ModulesLabsChallengesProjectsCertification

1 / 112 Modules

1%Completed

About Linux

DifficultyBeginner

Hands-on LabsIncluded

Labs52

What You'll Learn

Navigate the command line with confidence
Understand core concepts and architecture
Manage files, users and permissions
Work with common day-to-day tools

Linux Mastery1 / 112 modules

⌄Module - 1: Linux Introduction1 / 4

Episode 1: Meet Linux - What Actually Runs on a Server?Documentation walkthrough

▶ Start

Episode 2: How You Talk to Linux?Documentation walkthrough

▶ Start

Episode 3: Your First Remote Linux ServerDocumentation walkthrough

▶ Start

Get on the box - Your first linux serverCoreOps · 60 min hands-on lab

Practice Again

›Module - 2: Navigation and Filesystem0 / 5

›Module - 3: File and Directories0 / 5

›Module - 4: Reading Files and Getting Help0 / 5

›Module - 5: Editing Files0 / 5

1 / 112 Modules

1% Completed

About Linux

DifficultyBeginner

Hands-on LabsIncluded

›DevOps - Engineering›Linux›Episode 1

LinuxDocumentation

Episode 1: Meet Linux - What Actually Runs on a Server?

Module - 1: Linux Introduction

7 min readBy Infrathrone

Your First Server

It is your first day working with an infrastructure team. Your senior sends you a message:

"payment-service is running on server-01. Can you quickly check the machine?"

You are given access to the server. But before running commands, there is a more important question: what exactly is a server?

And when someone says, "This server runs Linux," what does that actually mean? Is Linux the complete operating system? What is Ubuntu, then? Where does the kernel fit?

That is what we will make clear first.

1. So… What Exactly Is a Server?

A server is not a magical type of computer. It is a regular computer whose job is to serve other computers over a network.

Daily challenge

One outage drops every single day.

Each day a new War Zone becomes the daily challenge. Solve it to keep your streak alive and stack SS points. A month in, an outage feels routine instead of terrifying.

A fresh outage every day, across all three tiers
Streaks reward showing up, not cramming
Every solve adds SS points to your global rank

47-day streak

BEST 63

TODAY'S DROP · BATTLEOPS

WAF regex pins every CPU

Inspired by Cloudflare · July 2019

SAMPLE ACCOUNT

Gamified

Start at the bottom. Climb to #1.

Every War Zone you fix earns Sudo Shells and moves you up the global leaderboard, one rank at a time.

Global leaderboardSee where you stand against every engineer in the Park.

StreaksKeep the daily chain going. Miss a day and it resets.

Sudo ShellsEarned for every fix. Harder tiers pay more.

TiersClimb from CoreOps to BattleOps to EliteOps.

GLOBAL LEADERBOARD · SAMPLE DATA

Proof of work

Leave with proof, not just a certificate.

Portfolio projects

One in every skill module. Real infrastructure you can walk a recruiter through.

Skill certifications

Certify each skill you complete, from Linux to Kubernetes.

Readiness report

Generated from your work on the platform. Attach it to your resume.

24/7

Community

You never debug alone.

Learning community

Engineers learning together, sharing RCAs and helping each other around the clock.

24/7 support

Stuck in a lab at 2 AM? Get unstuck at any hour.

Built by InfraThrone

From the team behind InfraThrone Elite.

5,000+engineers transformed

100K+community across socials

#1DevOps mentor on Topmate, globally

300+production outage scenarios

Pricing

Choose how you want to enter the War Room.

Get hands-on with real production outages, build your skills, and be ready for what's next.

Yearly works out to ~₹667/month instead of ₹1,999.

Best value · Most popular

YEARLY PLAN

Best value. Go further.

₹7,999/ year

~~₹23,988~~You save ₹15,989 (~67%)

That's about ₹667 a month, billed once a year.

Get Yearly Access

On monthly you pay ₹23,988 over a year.

EVERYTHING INCLUDED

Access to 300+ outages
30 new outages every month
50+ real-world organization outages
Daily challenge
Global leaderboard
Every role and skill roadmap
Hands-on labs, challenges and incidents
Portfolio projects and skill certifications
24/7 community and support
Cancel anytime

Questions

Before you enter the Park

What's the difference between labs, challenges, incidents and War Zones?

Labs are where you practise the exact topic you just studied. Challenges combine four or five topics into a task a DevOps engineer does day to day. Incidents are those challenges broken on purpose, so you find and fix the fault. War Zones are multi-stack outages that really happened in tech history, at organisations like AWS and Cloudflare.

Which roadmaps are included?

All of them. Your membership covers every role-based roadmap in the Park (DevOps Engineering, Site Reliability Engineering, Cloud Engineering and MLOps) and every skill roadmap, such as Linux, Docker, Kubernetes and Terraform.

How does the daily challenge work?

Every day one outage is released as the daily challenge. Solving it keeps your streak going and earns SS points that count toward your place on the global leaderboard.

What do I get to show recruiters?

A portfolio project in every skill module, a skill certification for each skill you complete, and a readiness report generated from your work on the platform that you can attach to your resume.

What if I get stuck?

Ask the 24/7 learning community or reach support at any hour. You can also read reviewed RCAs from other engineers once you've attempted an outage.

Break production. On purpose.

Tutorials teach YAML.
Production breaks at 3 AM.

Your first outage happens live

Courses stop at "it works"

Nothing proves you can fix it

Five levels. Every topic climbs the same ladder.

Real production outages, rebuilt as live broken labs.

Pick it. Fix it. Explain it.

Your path to production, mapped.

DevOps Engineering

Site Reliability Engineering

Cloud Engineering

MLOps

Inside a skill

One outage drops every single day.

Start at the bottom. Climb to #1.

Leave with proof, not just a certificate.

Portfolio projects

Skill certifications

Readiness report

You never debug alone.

Learning community

24/7 support

From the team behind InfraThrone Elite.

Choose how you want to enter the War Room.

Before you enter the Park

Your first outage is waiting.

Tutorials teach YAML.Production breaks at 3 AM.

Your first outage happens live

Courses stop at "it works"

Nothing proves you can fix it

Five levels. Every topic climbs the same ladder.

Real production outages, rebuilt as live broken labs.

Pick it. Fix it. Explain it.

Your path to production, mapped.

DevOps Engineering

Site Reliability Engineering

Cloud Engineering

MLOps

Inside a skill

One outage drops every single day.

Start at the bottom. Climb to #1.

Leave with proof, not just a certificate.

Portfolio projects

Skill certifications

Readiness report

You never debug alone.

Learning community

24/7 support

From the team behind InfraThrone Elite.

Choose how you want to enter the War Room.

Before you enter the Park

Your first outage is waiting.

Tutorials teach YAML.
Production breaks at 3 AM.