Outage Lab teaching notes
For high school IT, CTE, networking and computer science teachers, and anyone teaching toward CompTIA A+, Linux+ or a cloud certification. Two periods through six nights, each one a different way a real website breaks, and an incident report students write the way operations teams do.
Using this with a class? Make a free class and add your students yourself: each gets a username and password, with no email or Google account needed. Or give them one join code. Using Google Classroom? Post the link there; with a free class, the link also puts each student in your class, so you see their work. mytekDev is also available from the Google Workspace Marketplace, so your school's Google admin can install it for everyone.A first lesson: Open Night 1 on the projector and press Acknowledge. Before anyone types, ask the class to read the four graphs out loud: when did it break, and what stopped? Then type uptime. The server says it has been up three minutes. Ask what that means, and let the class choose the next command.
Set up a free class Share to Google Classroom- About these notes
- Two periods
- The six nights
- What students get wrong, and where the app answers it
- What the app simplifies, on purpose
- The incident report as an assessment
- Setting it up for a class
- Standards
About these notes
Outage Lab is one small web server, web-01, running a made-up school district's lunch ordering site. The site is nginx in front of an app server, with a database and a cache behind it. Each night something breaks; the student is the person on call.
The work follows the troubleshooting method CompTIA teaches and every operations team uses: find out what is wrong and what changed, form a theory, test it, fix it, check it stays fixed, and write it up. The app is built so that skipping a step costs something: a fix without the cause comes back when the night is fast-forwarded.
Two periods
The first period is about the method. The second is about range: four more ways a site breaks, faster.
- Night 1, the reboot (15 minutes) The site stopped at 2:02. Before anyone types, read the graphs together: when did it break? Then uptime (up 3 minutes), systemctl status nginx (inactive, disabled). Most students start nginx and fast-forward; at 4:00 a planned restart takes it down again. Ask why, then enable it.
- Night 2, the full disk (15 minutes) Orders fail, the menu loads. df -h says 100%; du -sh /var/log/* finds a 9 GB log. Most students delete it and df still says 100%. That is the lesson: sudo lsof +L1 shows the app still holds the deleted file. Restart it, then set the log level back to info so it does not fill again.
- The report (15 minutes, or homework) Write the report for one night. Read two aloud and compare their root causes and triggers. If someone tried to name a person, talk about why real teams do not.
- Nights 3 to 6 (second period) A bad release, an expired certificate, a breakfast rush and a job that runs out of memory. Pairs can split them and teach each other what the fix was and what kept it fixed.
The six nights
Each night opens on its own page and keeps its own state, so they can be played in any order.
| Night | What broke | What it is built to show |
|---|---|---|
| 1. The reboot | An upgrade restarted the server at 2:02 and nginx, which someone disabled last month, did not come back | Start fixes tonight; enable fixes every night after |
| 2. The full disk | A retry loop logging at debug level filled the disk at 2:31; orders fail, the menu loads | A deleted file still open keeps its space; a config change needs a restart |
| 3. The bad deploy | Release 42 at 1:52 renamed a field the order page still reads | Roll back first, fix the code after; the auto-deploy at 5:00 ships it again unless paused |
| 4. The expired certificate | A firewall rule blocked port 80, so 30 days of renewals failed and the certificate ran out at midnight | Browsers refuse before asking, so traffic falls while errors stay low; nginx keeps the old certificate until reloaded |
| 5. The spike | Free pizza Friday at 7:00 sends five times the traffic, and the cache was turned off for a test | Capacity: neither the cache nor a second server alone carries the rush; started is not enabled |
| 6. The 2 a.m. job | An hourly export loads every order into memory; at 2:03 the kernel's OOM killer took the app server | Memory as saturation; a job that grows each hour; turn off the cause, not just restart |
What students get wrong, and where the app answers it
Novice troubleshooters rerun and restart without changing anything, fix one problem by making another, and stop at the first thing that looks wrong. The nights are built to make each of those cost something.
| What students think | Where the app answers it |
|---|---|
| Restarting fixes things | Night 3: restarting the app changes nothing, because the bug is in the release |
| If it is back up, it is fixed | Fast-forward on every night: a fix that leaves the cause in place comes back before morning |
| Deleting a file frees its space | Night 2: df stays at 100% until the program holding the file lets go |
| Errors are the only sign of trouble | Night 4: traffic falls to almost nothing while the error rate stays under 1% |
| curl -k means it works | Night 4: curl -k skips the check that every visitor's browser makes, and the timeline says so |
| Someone made a mistake, so the cause is that person | The report will not print while it blames a person; it asks what made the mistake easy to make |
What the app simplifies, on purpose
- One small server with four services. A real site has many machines, and its logs are much longer.
- Time runs a minute a minute until Fast-forward. Real incidents often take hours, and nobody fast-forwards.
- The graphs are drawn from the night's story, not measured. The numbers are realistic in shape, not exact.
- About 40 commands work. Others say command not found, and help lists the ones that do.
- There is no team: one person handles the page, the fix and the report. Real incidents have roles, like an incident commander and someone who talks to users.
- Each night's real outage card is a short summary of the company's own write-up (Microsoft did not publish one, so the Teams card links a news report).
The incident report as an assessment
The report follows the order of Google's example postmortem: summary, impact, root causes, trigger, resolution, detection, action items, what went well, what went wrong, where we got lucky, and a timeline. Impact, detection and the timeline are filled in from the student's own night.
The last section, For the next person on call, is CSTA 3A-CS-03 almost word for word: a troubleshooting guideline someone else can use. It is the part most worth grading.
Setting it up for a class
No setup is needed. With free accounts, each student's nights and reports save as one project; the shared project shows the graphs around the outage, the stars and the report.
Grade level in the header switches the graph names between the golden signals (High school) and plain words (Middle school). Big screen makes everything larger for a projector.
Stars: Restored (the site came back), Stays fixed (still up in the morning, and on Night 5 the cache is enabled), Method (evidence pinned and a theory written before the fix).
Standards
- CSTA 3A-CS-03: Develop guidelines that convey systematic troubleshooting strategies that others can use to identify and fix errors.
- CSTA 2-CS-03: Systematically identify and fix problems with computing devices and their components.
- CSTA 3B-NI-03: Describe the issues that impact network functionality (e.g., bandwidth, load, delay, topology). Night 5.
- CSTA 3A-NI-04: Evaluate the scalability and reliability of networks. Night 5.
- CSTA 3B-AP-22, on how a change to a program can break other parts of it (paraphrased). Night 3.
- CompTIA A+ troubleshooting methodology, and Linux+ troubleshooting: services, logs, storage and processes.
AP Computer Science Principles touches fault tolerance (Topic 4.2) for networks, not the work of keeping a service up; Outage Lab is an extension there, not the assessed content.
More help
Is something on this page wrong or not clear? Write to us.
Page last reviewed November 2026.
Google, Google Classroom and Chromebook are trademarks of Google LLC.