Weekly alert noise review board for incident.io on-call

Build me an alert noise board for incident.io that my team works out of every week to decide which alerts deserve to keep waking people up. It is an internal review surface, not something that runs on a schedule. I open it before our on-call review, we sort the mess by how loud it is, and we leave with decisions made.

The main view is a ranked table of every alert route and every alert source over a window I choose, defaulting to the last 7 days, with a picker for 7, 14 and 30 days. Each row shows the route or source name, how many alerts it fired in the window, and then the quality columns: the share of those alerts that ever became a real incident, the share that resolved on their own with nobody doing anything, how many fired outside working hours, and how many triggered an escalation that actually paged a human. Sort by volume by default and let me re-sort by any column.

Handlers assemble this from incident.io: List Alert Routes for the routes and their configuration, List Alerts for every alert in the window with its source, title, status and timestamps, List Incident Alerts for the connections between alerts and incidents, and List Escalations for which alerts escalated to a person. The connections between alerts and incidents are the thing that separates genuine signal from noise, so treat them as the backbone of the whole board rather than a nice-to-have column. Paginate through all four fully for the window rather than reading only the first page.

Define the scoring precisely. The core noise signal is the ratio of a route's alerts that never linked to any incident in List Incident Alerts. An alert counts as self-resolved when it reached a resolved status with no linked incident and no escalation, meaning it fired and cleared itself while nobody acted. An alert is out of hours when its creation time falls outside the working hours and timezone configured in settings. Weight alerts that triggered an escalation and paged a human more heavily than alerts that merely fired, because those are the ones causing real fatigue, and surface a single noise score per row that combines volume, the never-became-an-incident ratio and that paging weight. Show the inputs next to the score so nobody has to trust a black box.

Clicking a route opens a detail view of its recent firings grouped by repeated alert title, so the same flapping check stands out immediately. Each group shows the alert title, how many times it fired, when it first and last fired in the window, whether any of those firings linked to an incident, and whether any escalated. Below the groups, list the individual firings with their timestamps and status. Pull the route's current configuration with Show Alert Route so I can see what conditions and escalation path it is actually using while I look at what it produced.

From the detail view I need to act. A button files a tuning ticket with Linear Create Issue, prefilled with the route name, its volume and noise numbers, the top repeated alert titles, and a suggested change, into a team I pick in settings. A second button lets me adjust the route itself with Update Alert Route, showing the current configuration first and asking me to confirm before it writes. A third button posts the week's noise summary to my team channel with the Slack Bot Send a Message action, covering the top noisy routes, their never-became-an-incident ratios, out-of-hours counts and whatever tickets we filed during the review.

Add an Investigate this alert button on every row that runs a background agent. The agent pulls that alert's recent history with List Alerts and Show Alert, finds any incidents it was linked to with List Incident Alerts, checks whether it escalated with List Escalations, and reads the route configuration with Show Alert Route. It then writes a verdict on whether this is real signal or noise, with a recommended threshold or routing change stated concretely enough to act on, files a Linear ticket with Create Issue carrying that recommendation, and leaves its reasoning on the row so the next person sees the call was already made. Show the row as investigating while the agent runs and then swap in the verdict, the recommendation and a link to the filed ticket.

Persist the review state so the board carries our decisions forward. Store every agent verdict and its reasoning, every ticket we filed, and a per-row status of untriaged, tuning filed, accepted as signal or muted, keyed to the route or alert source so it survives across windows. Rows we already settled should render as settled with the date and who decided, and should be filterable out of the ranking so each week we only argue about what is new. Settings hold the review window default, working hours and timezone, the Linear team and default labels, and the Slack channel for the summary.