Outage Lab teaching notes

For high school IT, CTE, networking and computer science teachers, and anyone teaching toward CompTIA A+, Linux+ or a cloud certification. Two periods through six nights, each one a different way a real website breaks, and an incident report students write the way operations teams do.

Using this with a class? Make a free class and add your students yourself: each gets a username and password, with no email or Google account needed. Or give them one join code. Using Google Classroom? Post the link there; with a free class, the link also puts each student in your class, so you see their work. mytekDev is also available from the Google Workspace Marketplace, so your school's Google admin can install it for everyone.A first lesson: Open Night 1 on the projector and press Acknowledge. Before anyone types, ask the class to read the four graphs out loud: when did it break, and what stopped? Then type uptime. The server says it has been up three minutes. Ask what that means, and let the class choose the next command.

Set up a free class Share to Google Classroom
Put this on your class site

Students can use Outage Lab right on your page, with no account. What they make stays on that page; to keep their work, they make a free account.

Google Sites: Insert, then Embed, then By URL, and paste this address.

https://mytekdev.com/try/outagelab?embed=1

Canvas, a blog, or any page that takes HTML: paste this code where you edit the page as HTML.

<iframe src="https://mytekdev.com/try/outagelab?embed=1" width="100%" height="640" style="border:0" allow="fullscreen" title="Outage Lab demo from mytekDev"></iframe>

About these notes

Outage Lab is one small web server, web-01, running a made-up school district's lunch ordering site. The site is nginx in front of an app server, with a database and a cache behind it. Each night something breaks; the student is the person on call.

The work follows the troubleshooting method CompTIA teaches and every operations team uses: find out what is wrong and what changed, form a theory, test it, fix it, check it stays fixed, and write it up. The app is built so that skipping a step costs something: a fix without the cause comes back when the night is fast-forwarded.

Nothing in the app touches a real server. The terminal is the platform's terminal, but the machine it talks to is pretend: about 40 commands answer the way Ubuntu 22.04 prints them, and nothing typed runs anywhere.

Two periods

The first period is about the method. The second is about range: four more ways a site breaks, faster.

  1. Night 1, the reboot (15 minutes) The site stopped at 2:02. Before anyone types, read the graphs together: when did it break? Then uptime (up 3 minutes), systemctl status nginx (inactive, disabled). Most students start nginx and fast-forward; at 4:00 a planned restart takes it down again. Ask why, then enable it.
  2. Night 2, the full disk (15 minutes) Orders fail, the menu loads. df -h says 100%; du -sh /var/log/* finds a 9 GB log. Most students delete it and df still says 100%. That is the lesson: sudo lsof +L1 shows the app still holds the deleted file. Restart it, then set the log level back to info so it does not fill again.
  3. The report (15 minutes, or homework) Write the report for one night. Read two aloud and compare their root causes and triggers. If someone tried to name a person, talk about why real teams do not.
  4. Nights 3 to 6 (second period) A bad release, an expired certificate, a breakfast rush and a job that runs out of memory. Pairs can split them and teach each other what the fix was and what kept it fixed.

The six nights

Each night opens on its own page and keeps its own state, so they can be played in any order.

NightWhat brokeWhat it is built to show
1. The rebootAn upgrade restarted the server at 2:02 and nginx, which someone disabled last month, did not come backStart fixes tonight; enable fixes every night after
2. The full diskA retry loop logging at debug level filled the disk at 2:31; orders fail, the menu loadsA deleted file still open keeps its space; a config change needs a restart
3. The bad deployRelease 42 at 1:52 renamed a field the order page still readsRoll back first, fix the code after; the auto-deploy at 5:00 ships it again unless paused
4. The expired certificateA firewall rule blocked port 80, so 30 days of renewals failed and the certificate ran out at midnightBrowsers refuse before asking, so traffic falls while errors stay low; nginx keeps the old certificate until reloaded
5. The spikeFree pizza Friday at 7:00 sends five times the traffic, and the cache was turned off for a testCapacity: neither the cache nor a second server alone carries the rush; started is not enabled
6. The 2 a.m. jobAn hourly export loads every order into memory; at 2:03 the kernel's OOM killer took the app serverMemory as saturation; a job that grows each hour; turn off the cause, not just restart

What students get wrong, and where the app answers it

Novice troubleshooters rerun and restart without changing anything, fix one problem by making another, and stop at the first thing that looks wrong. The nights are built to make each of those cost something.

What students thinkWhere the app answers it
Restarting fixes thingsNight 3: restarting the app changes nothing, because the bug is in the release
If it is back up, it is fixedFast-forward on every night: a fix that leaves the cause in place comes back before morning
Deleting a file frees its spaceNight 2: df stays at 100% until the program holding the file lets go
Errors are the only sign of troubleNight 4: traffic falls to almost nothing while the error rate stays under 1%
curl -k means it worksNight 4: curl -k skips the check that every visitor's browser makes, and the timeline says so
Someone made a mistake, so the cause is that personThe report will not print while it blames a person; it asks what made the mistake easy to make

What the app simplifies, on purpose

The incident report as an assessment

The report follows the order of Google's example postmortem: summary, impact, root causes, trigger, resolution, detection, action items, what went well, what went wrong, where we got lucky, and a timeline. Impact, detection and the timeline are filled in from the student's own night.

The last section, For the next person on call, is CSTA 3A-CS-03 almost word for word: a troubleshooting guideline someone else can use. It is the part most worth grading.

Print waits until every section has an answer, the root causes say more than the trigger, and no section names a person as the cause. A student who writes Kai forgot is asked what made the mistake easy to make.

Setting it up for a class

No setup is needed. With free accounts, each student's nights and reports save as one project; the shared project shows the graphs around the outage, the stars and the report.

Grade level in the header switches the graph names between the golden signals (High school) and plain words (Middle school). Big screen makes everything larger for a projector.

Stars: Restored (the site came back), Stays fixed (still up in the morning, and on Night 5 the cache is enabled), Method (evidence pinned and a theory written before the fix).

Standards

AP Computer Science Principles touches fault tolerance (Topic 4.2) for networks, not the work of keeping a service up; Outage Lab is an extension there, not the assessed content.

More help

Is something on this page wrong or not clear? Write to us.

Page last reviewed November 2026.
Google, Google Classroom and Chromebook are trademarks of Google LLC.