Tell real CircleCI failures from flaky tests in Slack

Watch my CircleCI project and run whenever a workflow finishes, using CircleCI's workflow-finished poll trigger. Act only on workflows whose status is failed, and only when they ran on my default branch (main or master). Ignore successful runs, canceled runs, and any run on another branch, so the on-call channel never gets noise.

When a failed workflow comes in on the default branch, call Get a workflow's jobs to read the sequence of jobs and pull out the ones whose status is failed. Keep their names so the alert can list exactly which jobs broke.

Then decide whether this is a known flaky test or a genuine regression. Call Get test metrics for a workflow to see which tests failed in this recent window, and call Get flaky tests for a project to get the tests CircleCI has already flagged as flaky (tests that passed and failed on the same commit). Cross-reference the two: if every failing test is already on the known-flaky list, treat the failure as safe to rerun; if any failing test is not a known flaky one, treat it as a likely real break that needs a human. CircleCI has no raw build log or per-run test-detail action, so base the judgment on these jobs, test-metrics, and flaky-test signals only, and do not claim certainty you cannot get from them.

Post a triage message to my on-call Slack channel using the Slack Bot Send a Message action. Name the failed jobs, state plainly whether this looks like a known flaky test that is safe to rerun or a likely real break that needs a human, and include a link back to the CircleCI build so anyone can dig in. Tag the on-call channel so the right people see it. When every failing test is already a known flaky one, say that clearly and up front so the team does not chase noise; when it looks like a real break, make the call to action for a human obvious.