CloudWatch incident review and postmortem workbench

PromptCreate

I want an incident review workbench I open the morning after something breaks, so my team can turn a messy pile of CloudWatch alarm noise into a real postmortem without anyone scrolling the AWS console at 2am. This is an app a person sits in and works through, not a scheduled report. Nothing in it should run on a timer or a cron schedule.

The app opens on an incident setup view where I pick a start time and an end time for the incident window. When I confirm the window, the app pulls my alarms with Describe Alarms and then calls Describe Alarm History for each alarm, filtered to that date range and to state transition items, and merges everything into a single chronological timeline. Every entry shows the alarm name, the timestamp, the direction of the change such as OK to ALARM or ALARM to OK, and the reason CloudWatch recorded for it. Alarms whose first transition into ALARM happened earliest are pinned to the top of the timeline and labelled as first movers, because the first thing to fire is usually the closest thing we have to a root cause.

Clicking any alarm on the timeline opens a detail panel. It shows that alarm's configuration from Describe Alarms, including the metric it watches, the threshold, the evaluation periods, and the namespace and dimensions. For an alarm currently in ALARM state it also calls Describe Alarm Contributors to show which individual time series inside that alarm were actually in breach, so I can tell whether one instance broke or the whole fleet did. The panel has a button to chart the underlying metric across the incident window using Get Metric Statistics, so I can see the shape of the failure, whether it was a cliff, a ramp, or a sawtooth, without leaving the app.

The curation step is the heart of this app and must not be skipped or automated away. Next to every alarm on the timeline I can tag it as part of the incident, collateral damage, or unrelated noise. Only alarms I tagged as part of the incident or as collateral feed the postmortem. This human separation of real signal from noise is the entire point of the workbench, so make it fast and obvious, with quick keyboard friendly ticking and a running count of how many alarms are still untriaged.

Once I have curated the timeline, the app drafts a structured postmortem. It should include the detection time, the alarm that caught the incident first, the time to first response, the total duration, and the affected resources gathered from the dimensions on the alarms I kept. Be honest in the wording about where these numbers come from. Detection and response times are inferred from when alarms changed state, not from any paging or acknowledgement tool, so label them that way in the draft, for example 'first response inferred from the gap between the first ALARM transition and the first recovery activity'. Never imply we have real acknowledgement tracking. Every field in the draft is editable inline before anything leaves the app.

When the draft looks right, a File to wiki button creates the postmortem as a new page in our engineering wiki using the Notion Create a Page handler, under a parent page or database I choose in settings. The app stores the resulting Notion page URL against the review.

In the same workbench I add follow up actions, each with a title, an owner, and a short description. Filing them creates one ticket per action through the Linear Create Issue handler, into a team I pick, and every ticket description links back to the filed Notion postmortem page so the context travels with the work. Once filed, the actions stay listed in the app with their Linear ticket links, and I can edit a ticket's title or description from here using Linear Update Issue rather than switching tools.

The app keeps a list of every past review it has produced, stored in the app itself, showing the incident window, the first mover alarm, the duration, the postmortem link, and the follow up tickets. Across that history the app counts how many separate incident reviews each alarm has appeared in, and flags any alarm appearing in three or more as a chronic offender, because that is a problem nobody has actually fixed. Surface these chronic offenders prominently on the past reviews view, not buried in a submenu.

Two scope limits to build around. CloudWatch Logs is a separate service and is not part of this integration, so the postmortem cannot contain log excerpts or error traces. Keep the evidence strictly to alarm state transitions and metric shapes, and do not write UI copy that promises log analysis. Also, CloudWatch only retains alarm history for a limited period, so handle an empty or truncated history window gracefully. If I pick a window with no history or only partial history, say plainly that the history is unavailable or incomplete for that range rather than rendering an empty timeline or implying the incident had no alarms.

Read only CloudWatch access is enough for everything on the AWS side. The app never creates, edits, deletes, or silences an alarm.

What does this prompt do?

Pick the start and end of an incident and get every alarm state change in that window merged into one timeline, with whatever fired first pinned to the top as the closest thing you have to a root cause.
Tick off which alarms genuinely belonged to the incident and which were collateral or unrelated noise, so the writeup reflects what actually happened instead of every alarm that flickered.
Chart the underlying metric for any alarm across the incident window to see the shape of the failure, without opening the AWS console.
Get a drafted postmortem with detection time, first alarm, duration and affected resources, edit it inline, file it to Notion, and turn follow up actions into Linear tickets that link back to it.

What do I need to use this?

An AWS account with CloudWatch alarms already set up. Read only access is enough, since the app never creates, edits, or silences an alarm.
A Notion workspace, and a page or database where your team keeps engineering postmortems.
A Linear workspace and a team to file follow up tickets into.
Alarms that were actually firing during the incident you want to review. CloudWatch keeps alarm history for a limited window, so an incident from a year ago may no longer be reconstructable.

How can I customize it?

Change the default incident window so the timeline opens pre-filled with the last few hours instead of asking you to pick times every session.
Point the filing step at a different Notion parent page or database, and add your own fields to the postmortem template such as severity, customer impact, or an incident commander.
Set how many separate reviews an alarm has to appear in before it gets flagged as a chronic offender. Three is the default.

FAQs

Does this pull in our application logs or error traces?

No. The evidence is alarm state changes and metric shapes only. Logs live in a separate part of AWS that this app does not connect to, so the postmortem will tell you which alarms fired, when, and what the metrics were doing, but it will not quote stack traces. Most teams paste the relevant log detail in themselves while editing the draft.

How does it know how quickly we responded?

It infers response timing from the gaps between alarm state changes, for example how long it took an alarm to move from firing back to healthy. It is not connected to a paging tool, so it cannot tell you when a human actually acknowledged the page. The draft says so in plain terms rather than presenting it as real acknowledgement tracking.

Can I review an incident that happened months ago?

Only if the alarm history is still there. CloudWatch keeps alarm history for a limited period, so older incidents may return nothing or only part of the picture. If you pick a window with missing history, the app tells you the history is unavailable or incomplete for that range instead of showing you an empty timeline.

Will this change anything in our AWS account?

No. Everything it does on the AWS side is read only. It reads your alarms, their history, and their metrics. It never creates, edits, deletes, or silences an alarm, so it is safe to hand to anyone on the team.

Do I have to use Notion and Linear?

Those are the defaults, but the filing step and the ticket step are both swappable for whatever your team already uses. The part that matters is the curation step in the middle, where a person decides which alarms were real signal before anything gets written down.

Related templates

Share of voice dashboard for your brand and competitors

See how your brand's news coverage and sentiment stack up against four competitors, then let an assistant write the weekly report for you.

App

Influencer campaign roster board with AI creator briefs

Drag creators through Sourced to Wrapped on a board grouped by campaign, with audience stats on every card and a one-click brief for each creator.

App

Smart home entity health cockpit for Home Assistant

Open one screen to find every unavailable, stale, or low-battery device in your smart home, see which hub is really at fault, and clear out the dead entries in bulk.