Wolf Incident Postmortem

January 8th, 2023
kids, satire, tech

Incident #210

Status

Complete, one action item outstanding.

Summary

Sentinel consumed by wolf after repeated false alarms.

Impact

Loss of sentinel. No flock impact.

Root causes

Sentinel generated noisy alerts due to premature deployment, incomplete training, and overly monotonous task. Oncalls failed to respond to true positive due to alert fatigue.

Trigger

Wolf.

Resolution

Gathered flock. Deployed replacement sentinel.

Detection

Sentinel did not report at end of shift.

Action Items

Priority Action Item Type Status
P0 Gather flock mitigate complete
P0 Deploy replacement sentinel mitigate complete
P1 Update playbook for wolf alerts prevent complete
P2 Update remaining sentinels prevent complete
P2 Revise sentinel training program prevent complete
P2 Investigate equipping sentinels with flutes or slings prevent in progress

Lessons Learned

What went well

  • Flock gathering proceeded without issues.
  • No flock injuries or losses.
  • Replacement sentinel did not exhibit false positive alerts.

What went wrong

  • Noisy alerts not addressed.
  • Alerts silenced contrary to playbook.
  • Loss of sentinel.

Where we got lucky

  • Only one wolf.
  • Wolf sated after sentinel consumption.
  • Replacement sentinel available.

Timeline

All times local

March 3rd:

  • 16:32 Oncalls paged "wolf".
  • 16:34 First oncall arrives at sentinel location.
  • 16:34 Alert diagnosed as false positive. No corrective action performed.

March 4th:

  • 14:15 Oncalls paged "wolf".
  • 14:19 First oncall arrives at sentinel location.
  • 14:19 Alert diagnosed as false positive. No corrective action performed.

March 5th:

  • 17:03 (Reconstructed) Outage begins, sentinel notices wolf.
  • 17:03 Oncalls paged "wolf".
  • 17:04 Oncalls paged "wolf".
  • 17:04 Oncalls paged "real wolf".
  • 17:05 (Reconstructed) Wolf consumes sentinel.
  • 18:45 Sentinel does not report at end of shift.
  • 19:05 Primary oncall dispatched to field.
  • 19:10 Oncall diagnoses issue.
  • 19:10 Incident begins, secondary and tertiary oncalls paged.
  • 19:15 First sheep located.
  • 19:52 Last sheep located.
  • 20:05 Flock safe in pens.
  • 20:05 Outage ends, flock protection fully restored.
  • 20:45 Replacement sentinel identified.
March 6th:
  • 07:38 Replacement sentinel deployed
  • 18:45 Replacement sentinel reports at end of shift
  • 18:45 Incident ends, 24hr without wolf alerts or activity (exit criterion).

Comment via: facebook, lesswrong, hacker news, mastodon

Recent posts on blogs I like:

What Percentage of People Are Bi?

More than half?

via Thing of Things February 26, 2024

How I build and run behavioral interviews

This is an adaptation of an internal doc I wrote for Wave. I used to think that behavioral interviews were basically useless, because it was too easy for candidates to bullshit them and too hard for me to tell what was a good answer. I’d end up grading eve…

via benkuhn.net February 25, 2024

Diseconomies of scale in fraud, spam, support, and moderation

If I ask myself a question like "I'd like to buy an SD card; who do I trust to sell me a real SD card and not some fake, Amazon or my local Best Buy?", of course the answer is that I trust my local Best Buy1 more than Amazon, which is notoriou…

via Posts on February 18, 2024

more     (via openring)