Weekly Buildkite CI health email for engineering leads

Every Monday at 8:00am, run automatically on a cron schedule and email our engineering leads a CI health digest covering the past seven days of activity in Buildkite. The email should be a short, skimmable executive summary of how our continuous integration is doing, with the key numbers first and the ranked detail below. All queries are scoped to our Buildkite organization, so discover the organization slug first if you need it.

First, gather build reliability data. Use Buildkite's List Builds for an Organization operation, filtered to builds from the last seven days (pass the date filter, and branch and state filters if useful), paginating so the counts cover the full window. From those builds compute: total build volume, the pass rate and fail rate, the average build duration, and the slowest builds. Then identify the pipelines and the branches that failed most often over the week. Also pull the previous seven-day window with the same operation so you can report the week-over-week direction of the key metrics.

Next, find likely flaky tests using Buildkite Test Engine. Call List Test Suites to enumerate our suites, then for each suite call List Test Runs to get the recent runs, and for those runs call Get a Test Run's Failed Executions. Aggregate the failed executions across recent runs to spot tests that fail intermittently, meaning they fail in some runs but pass in others, and rank these as our likely flaky offenders. The Buildkite REST API has no single flakiness score, so infer flakiness by counting how often each test appears in failed executions across runs while not failing consistently.

Write a short executive summary at the top of the email. Lead with the headline numbers (build volume, pass and fail rate, average and slowest durations), state the week-over-week direction for the key metrics (are we improving or regressing versus the prior week), and call out the top three things to fix. Keep the tone concise and decision-oriented.

Lay the email out to be skimmable: the key metrics first, then a ranked list of the worst pipelines and branches by failure count, then a ranked list of the flakiest tests. Include links back to Buildkite for the failing pipelines, builds, and tests so leads can click straight through. Finally, send the digest using Gmail's Send a Message operation to our engineering leads, with a clear subject line such as 'Weekly CI health digest' plus the date range.