Weekly New Relic application performance review in Slack

Every Monday at 8am, write me a plain-English review of how our applications actually performed last week. Run this on a cron schedule.

Start with New Relic List Applications to get every APM application reporting to the account. Keep each application's name and id so later steps can line the numbers up with the right service.

Then use New Relic Run NerdGraph GraphQL Query to run NRQL for two windows: last week (SINCE 1 week ago UNTIL now) and the week before it (SINCE 2 weeks ago UNTIL 1 week ago). For each application and each window pull four figures: the Apdex score, the error rate, the throughput, and the 95th percentile response time. A query shaped like SELECT apdex(duration, t: 0.5), percentage(count(*), WHERE error IS true), count(*), percentile(duration, 95) FROM Transaction FACET appName covers all four at once, and faceting by appName lets you cover many applications per query instead of one query per application, which keeps the run inside NerdGraph's per-query complexity limits. Note that NerdGraph reports query and validation failures as a top-level errors array inside an HTTP 200 response, so check for that rather than trusting the status code.

Ignore applications with negligible traffic so low volume services do not dominate the regression list. As a default, skip any application with fewer than 1,000 transactions in the week being reported, and skip any application that did not report data in both weeks, since there is nothing to compare.

Name only the applications that moved materially week over week, and say plainly whether each one got better or worse. As defaults for material: Apdex moved by at least 0.05, error rate moved by at least half a percentage point or at least 25 percent in relative terms, or the 95th percentile response time moved by at least 20 percent. Everything inside those bands is normal variation and should not be named.

Order the narrative by how many real users were affected, not by raw percentage change. Estimate impact from the volume sitting behind the move: the change in error rate multiplied by the week's transaction count gives roughly the number of additional failed requests, and response time regressions should be weighted by throughput the same way. A 40 percent error rate jump on a service handling 200 requests a week ranks well below a 3 percent jump on one handling two million.

Post the narrative to Slack with Send a Message to the channel I name. Keep it readable in a channel: a one line headline with how many applications were checked and how many moved, then a short paragraph per application that moved, worst real impact first. Use Slack mrkdwn formatting, so *bold* rather than **bold**. If nothing moved materially, still post a short all steady note naming the number of applications checked, so I know the review ran.

Finally, append to the trend tracker with Google Sheets Append Values. Append one row per application for the week that just ended, with the week start date, application name, Apdex score, error rate, throughput, and 95th percentile response time. Do not append rows for the prior week, it is only there for comparison and re-appending it would duplicate a week of history on every run. Log every application that reported data, including the low traffic ones left out of the narrative, so the accumulated history stays complete. Write the values as user entered so dates and numbers land as real dates and numbers rather than text.