Five Real Postmortem Examples and What Makes Each One Worth Reading

The fastest way to improve the postmortems your team writes is to study ones that people still share years later. Templates help. Reading how strong engineering orgs narrate failure helps more.

Below are five public postmortems. Each section focuses on structure and culture, not on re-litigating the outage. Links go to the originals so you can read them yourself.

1. Cloudflare control plane and analytics outage (November 2023)

Post mortem on the Cloudflare Control Plane and Analytics Outage

On 2 November 2023, a power failure at a Portland data center took Cloudflare's control plane and analytics offline for an extended period while the edge data plane largely kept serving traffic. The writeup walks through utility power, generator behavior, UPS drain, and the discovery that some services believed to be highly available still depended on systems that only lived in the failed facility.

What it does well. It separates customer impact by product surface. Dashboard and API pain is not blurred into "Cloudflare was down." That distinction alone makes the document useful to readers deciding how nervous they should be.

Timeline. Event times are concrete and sequenced from facility failure through disaster recovery decisions. You can follow operator judgment under incomplete information, including the call to fail over to European DR sites when restoration ETAs were unclear.

Root cause framing. The piece does not stop at "power went out." It names dependency drift. Kafka and ClickHouse pieces that powered analytics were tighter coupled to the failed site than the HA story implied. That is real root cause work. Physical failure was the trigger. Architecture assumptions were the amplifier.

Action items. Follow-ups target dependency audits, failover testing that actually takes a facility offline, and recovery improvements. They read like engineering work, not slogans.

Takeaway. Spell out which invariant failed in practice. "We thought this was multi-AZ" is a better learning artifact than "a data center had a power issue."

2. Cloudflare Workers KV deployment incident (October 2023)

Cloudflare incident on October 30, 2023

A misconfiguration in a deployment tool used by Workers KV took multiple Cloudflare services down for 37 minutes. The post is blunt about how Cloudflare's own products depending on KV made the blast radius larger and the recovery harder.

What it does well. It owns the recursive dependency problem without theatrics. When your control and product surfaces rely on the thing you just broke, incident response gets weird. Naming that clearly helps other teams audit their own "we run on ourselves" edges.

Timeline. The UTC table from detection through rollback and break-glass routing is tight. Short incidents still benefit from minute-level honesty.

Root cause framing. The proximate cause is a bad deploy configuration. The useful framing is blast radius and tooling safety. Why could one misconfiguration reach so many services, and why was recovery slower because responders also depended on degraded internals?

Action items. Priorities around deployment tooling, reducing blast radius of critical dependencies, and product-level controls read as specific enough to track.

Takeaway. For change-related incidents, spend page space on why the change was allowed to be so wide, not only on which commit was bad.

3. AWS S3 disruption in US-EAST-1 (February 2017)

Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region

An authorized operator ran a playbook command with an incorrect input while debugging S3 billing slowness. More capacity was removed than intended. Index and placement subsystems restarted. S3 APIs in us-east-1 failed for hours, taking dependent AWS services with them. Even the Service Health Dashboard update path was impaired because it depended on S3.

What it does well. It is still one of the clearest public examples of "human error" rewritten as systems error. The person followed an established playbook. The tool accepted a dangerous input. Subsystems that had not been fully restarted at that scale in years took longer to validate than anyone expected.

Timeline. Recovery is staged by API class. GET/LIST/DELETE returning before PUT is the kind of detail other teams should copy. It teaches readers what "partially recovered" actually meant.

Root cause framing. Incorrect input is stated plainly. The lasting lesson is insufficient safety in operational tooling and recovery time that had not been re-validated as the system grew.

Action items. Tooling changes to remove capacity more slowly, hard minimum-capacity checks, audits of similar tools, and plans to partition the index into smaller cells. Concrete and falsifiable.

Takeaway. If your postmortem ends at "someone typed the wrong thing," you stopped one layer too early.

4. GitHub Availability Report (April 2024)

GitHub Availability Report: April 2024

GitHub's monthly availability reports summarize multiple incidents in one post. The April 2024 report covers a database load balancer change that drove elevated error rates and Actions failures, and a compute-heavy query that starved a critical database cluster.

What it does well. The format is operationally honest for a large multi-product platform. Each incident gets impact numbers (error rate peaks, which products hurt), a cause in plain language, and remediations. You can skim or go deep.

Timeline. These reports are denser and less narrative than a Cloudflare longform post. They trade novelistic detail for coverage and regularity. That is a feature if your audience is customers scanning for "did this affect me" and engineers scanning for patterns.

Root cause framing. Causes are specific enough to learn from (load balancer change, expensive query) without dumping internal architecture diagrams. Contributing process gaps, like incomplete CI detection for heavy queries, are acknowledged.

Action items. Rollback, better detection in the deploy pipeline, and service resilience to a hot dependency. Some items are broader "we are improving X," which is weaker than owner-dated tasks, but the monthly cadence itself creates accountability pressure.

Takeaway. A lighter, recurring public format can still teach if every incident includes impact, cause, and a change you can imagine implementing.

5. Stripe API degradation RCA (July 2019)

Stripe's CTO, David Singleton, published a detailed public RCA after API degradations on 10 July 2019. The narrative covers two separate failure windows the same day. A database election-protocol issue and a later interaction between a reverted database version and a production configuration change produced cascading unavailability across critical services. Contemporary writeups and analyses still cite the original post; search for "Stripe details of the July 10 outage" for archived copies if the live URL has moved.

What it does well. It treats remediation as a first-class risk. The team reverted to a "known stable" database version after the first degradation. That revert interacted poorly with a config change and contributed to the second event. Many postmortems pretend the fix path was clean. This one does not.

Timeline. Two incidents with overlapping symptoms and different mechanisms are kept distinct. That alone prevents the false lesson "we fixed it, then it broke the same way."

Root cause framing. The summary line is famous for a reason. Two database bugs and a configuration change interacted in an unforeseen way. That is contributing-factor thinking done properly. No single villain commit.

Action items. The public narrative emphasizes understanding and safer operational sequencing more than a bullet list of tickets. Internally you would still want owners and dates. Externally, the gift is the causal honesty.

Takeaway. Write the story so a future engineer can see how a reasonable fix created the next failure. That is rarer than it should be.

What the best postmortems have in common

They make impact measurable. Minutes, error rates, which products, which APIs. Vague "degraded performance" does not teach.

They separate trigger from amplifier. Power loss, bad input, bad deploy, expensive query. Then the dependency, tooling, or process that let it spread.

They keep timelines evidence-based. Clocks beat vibes. Staged recovery beats a single "resolved at" stamp.

They write actions a skeptic could verify. "Add a capacity floor to the drain tool" beats "improve operational excellence."

They sound like engineers talking to engineers. Reputation management is secondary. Learning is primary.

What makes a postmortem useless

A root cause that stops at "human error" or "bug." No system context, no why the failure was possible.

Action items like "be more careful," "add monitoring," or "improve documentation" with no owner, no scope, and no due date.

A timeline reconstructed from memory a week later, missing the awkward twenty minutes where people chased the wrong theory.

A blameless label with a blame tone. Soft words around a narrative that still centers who messed up.

A document written only to close a compliance checkbox. Those get filed. They do not get read. They do not change the next incident.

If your team stalls on the blank page after a long incident, tools that draft a timeline from alerts, deploys, and acknowledgements can remove the empty-document problem. Vigiles AI postmortem generation does that as a starting draft. The quality bar in the examples above still depends on humans doing the analysis the draft cannot.