Investigate New Relic alerts before waking your on-call

When a New Relic alert fires, have it call this workflow as a webhook and decide whether the problem actually deserves to wake someone up. Use a webhook trigger and point a New Relic alert workflow destination at it, so the incoming payload carries the condition name that fired, the affected application or entity, the threshold that was breached, and a link back to the New Relic incident.

Start by pulling the affected application out of the alert payload. If the payload identifies the target by entity name or entity GUID rather than by APM application name, resolve it first with New Relic List Applications and keep the numeric application id, because the host check later needs that id.

Then measure the real blast radius with New Relic Run NerdGraph GraphQL Query, running NRQL through the actor account nrql field. Query one, current error rate and throughput over the last 30 minutes: SELECT count(*) AS throughput, percentage(count(*), WHERE error IS true) AS errorRate FROM Transaction WHERE appName = 'APP' SINCE 30 minutes ago. Query two, the same measurement for the same window yesterday as a baseline: SELECT count(*) AS throughput, percentage(count(*), WHERE error IS true) AS errorRate FROM Transaction WHERE appName = 'APP' SINCE 1 day 30 minutes ago UNTIL 1 day ago. Query three, the transactions actually failing: SELECT count(*) FROM TransactionError WHERE appName = 'APP' FACET transactionName SINCE 30 minutes ago LIMIT 10. Query four, the shape of the incident so you can tell climbing from recovering: SELECT percentage(count(*), WHERE error IS true) FROM Transaction WHERE appName = 'APP' SINCE 30 minutes ago TIMESERIES 5 minutes.

NerdGraph reports query and validation failures as a top level errors array inside an HTTP 200 body, so check for that array before trusting any result. If a query fails, say so in the brief rather than reading the failure as zero errors.

Next, check how widely the problem has spread using New Relic List Application Hosts for that application id. Count how many hosts are reporting for the application, and compare that against where the failures are landing. A long host list where the errors trace back to a single host is a single host blip, not an outage.

Judge severity from that evidence rather than trusting the raw alert threshold. Treat impact as broad when the error rate is materially above the same window yesterday, the application runs on more than one host and the failures are not confined to one of them, the failing transactions include something on a main user path, and the timeseries is flat or climbing. Treat impact as narrow when the error rate is close to yesterday's baseline, the errors are confined to a single host, the affected throughput is tiny, or the timeseries already shows the error rate falling back toward normal.

If impact is broad, open a PagerDuty incident with PagerDuty Create Incident against the on-call service. Title it with the application name and the headline symptom, and put the full brief in the incident body. Then post that same brief to Slack with Slack Send a Message, so the channel sees exactly what the responder was paged with.

If impact looks narrow or is already recovering, post to Slack only and state plainly that no page was raised and why, for example that the errors are confined to one host and the rate has fallen from 12 percent to 2 percent over the last 15 minutes. Do not open a PagerDuty incident for a single host blip that resolves on its own.

Always include the actual numbers pulled from NRQL so the responder can sanity check the call: current error rate and throughput, the same two figures from yesterday's window, the top failing transactions with their error counts, and how many hosts are involved. Name the condition that fired and link back to the New Relic incident from the payload. Keep it to a short brief someone can read on a phone: lead with the decision, paged or not paged, then the one line reason, then the numbers behind it.