Alert Slack the moment a Fly.io Machine goes unhealthy

Every 15 minutes, watch my Fly.io infrastructure and alert my team in Slack the moment a Machine goes unhealthy. Run this on a cron schedule that fires every 15 minutes.

First, scan every Machine across my whole Fly.io organization. Use Fly.io List Apps to enumerate my apps, then use List All Machines (the organization-scoped operation) to pull every Machine in one pass, or fall back to List Machines per app if needed. For each Machine, read its current state and its health checks. Note: seeing every app requires an organization-scoped token; an app-scoped deploy token only sees one app.

Decide what is genuinely worth paging on. Flag a Machine when it is stopped unexpectedly, in a failed or error state, or failing one or more of its health checks. Ignore Machines that are intentionally stopped or scaled to zero, so routine idle capacity never triggers a false alarm. For every Machine you judge to be a real problem, call Fly.io Get Machine (or List Events) to pull its most recent events, so you can include the latest event in the alert.

If, and only if, at least one Machine is genuinely unhealthy, post one concise message with the Slack Bot Send a Message operation to my alerts channel (I will specify which channel). Put each affected Machine on its own line with the app name, Machine ID, region, current state, the specific check that is failing, and the latest event. Keep it scannable so an on-call engineer can triage at a glance.

If every Machine is healthy, do not post anything at all. Stay silent so the channel only lights up on real problems.