Datadog alert triage to Linear and Slack, every weekday at 9am

Build me an agent that runs every weekday at 9am in my local timezone and triages my currently-triggered Datadog monitors into a Slack digest plus Linear issues for the persistent ones. Trigger: cron.

Step 1 — Pull active alerts. Use Datadog Search Monitors to fetch every monitor whose current state is Alert or Warn. Capture for each: monitor id, name, link to the monitor in Datadog, severity (Alert vs Warn), tags, the time it most recently transitioned into its current state, and the message/owner field if present.

Step 2 — Group and clean. Group monitors by service or by the most meaningful tag (service:, team:, env: in that order of preference). Drop obvious duplicates that point at the same underlying check. Skip known-flapping monitors based on a configurable allow/deny list of monitor names or tags (default deny list is empty, but make it easy for me to add). Tag each monitor as either NEW today or CARRIED OVER (was already firing in yesterday's run — keep a small piece of state across runs to know this, e.g. write the previous run's monitor ids to a workflow memory store and diff).

Step 3 — Post the Slack digest. Use Slack Bot Send a Message to post one message to the channel I configure (default #alerts-triage). Format: a one-line headline with total counts ("7 firing: 3 Alert, 4 Warn — 2 new, 5 carried over from yesterday"), then top offenders first (highest severity, then longest-firing), grouped by service. Each line should show the monitor name as a link to Datadog, severity, how long it has been firing, and the NEW/CARRIED OVER tag. Use Slack mrkdwn formatting (*bold*, <url|text> for links). Keep it scannable — no walls of text.

Step 4 — File Linear issues for persistent alerts. For any monitor that has been firing for longer than a configurable window (default 4 hours), before doing anything, use Linear Search Issues to check whether an open issue already exists for that monitor. Match on the monitor name and/or monitor id (include the monitor id as a stable token in issue titles or descriptions so future runs can find them). If a matching open issue exists, skip — do not create a duplicate. If none exists, use Linear Create Issue in the team I configure, with a clear title like "[Datadog] <monitor name>", a description containing the monitor link, current severity, how long it has been firing, recent state transitions, and the monitor's tags. Suggest an assignee based on a team:/owner: tag when present, otherwise leave unassigned. Set priority based on severity (Alert → High, Warn → Normal).

Step 5 — Close the loop in Slack. At the bottom of the digest, list the Linear issues that were created this run with their identifiers and links, and the ones that were skipped because a matching open issue already existed. If nothing was filed, say so explicitly so I know dedupe is working.

Configurable inputs I want exposed: Slack channel to post to, Linear team to file in, the persistence window (default 4 hours), the monitor name/tag deny list for known flappers, and the timezone for the schedule.

Edge cases to handle gracefully: zero firing monitors (post a short "all clear" digest, file nothing), Datadog API rate limits (retry with the Retry-After header), and Linear partial errors (continue filing the remaining issues, summarize failures in the Slack digest).

Additional information

What does this prompt do?

Pulls every Datadog monitor that is currently in Alert or Warn and groups them by service or tag so you see the real picture, not 40 separate pings.
Posts a single morning digest in Slack with the top offenders at the top, counts by severity, and a one-line note on what is new versus carried over from yesterday.
Opens a Linear issue for any monitor that has been firing longer than your chosen window (default 4 hours), with the monitor name, link, recent state, and a suggested owner from the tags.
Checks Linear before filing so you do not end up with a new duplicate ticket every single morning.

What do I need to use this?

A Datadog account with an API key and application key so the agent can read your monitors.
A Slack workspace where the agent can post the morning digest in a channel of your choice.
A Linear workspace and a team where new triage issues should land.
A rough idea of how long a monitor should be allowed to burn before it earns a Linear ticket (default is 4 hours).

How can I customize it?

Change the schedule. Move it earlier for on-call handoff, later for standup, or run it twice a day if your team prefers two passes.
Tune the ticket threshold. Drop the four-hour window to one hour for tighter discipline, or raise it so only truly persistent monitors become tickets.
Pick which channel gets the digest and which Linear team owns the new issues, and add your own rules for tags or services to skip (known flappers, noisy staging monitors, etc.).

Frequently asked questions

Will this spam my channel with one message per alert?

No. The whole point is one digest per morning. The agent groups by service or tag, drops obvious duplicates, and posts a single prioritized message so the channel stays readable.

How does it avoid filing the same Linear ticket every day?

Before opening anything, the agent searches Linear for an open issue that matches the monitor. If it finds one, it skips. New tickets only get filed for monitors that are persistently firing and do not already have a home in Linear.

What counts as a persistent alert worth a ticket?

Anything that has been firing longer than your chosen window. The default is four hours, which filters out short blips while catching the ones your team has clearly stopped paying attention to.

Can I run this more or less often than every weekday at 9am?

Yes. The schedule is a starting point. You can move it to any time, run it daily including weekends, or trigger a second pass in the afternoon for follow-up.

Do I need to connect anything besides Datadog?

You also connect Slack so the digest can be posted, and Linear so the agent can search for existing tickets and open new ones for persistent alerts.