Alert on-call the moment a Fivetran pipeline breaks

Every 30 minutes, check my Fivetran data connections for broken syncs and open an incident when one has failed. Run this on a schedule (cron) rather than waiting for a webhook, since a periodic status check is the reliable way to catch pipeline failures here.

Start by calling Fivetran's List Connections to get every connection in the account. For each connection, call Get Connection State to read its current sync state. Treat a connection as broken when its last sync has failed or errored. Skip connections that are syncing healthily, and skip ones that are paused on purpose, since a paused connection is an intentional state and not a failure. I do not want to be alerted about pipelines I deliberately turned off.

For each broken connection, judge how severe the outage is (for example, a core reporting pipeline that many people depend on is more urgent than a minor one), then draft a clear, human-readable incident summary. Name the connection, the source it pulls data from and the destination it writes to, and the most likely cause of the failure based on its state (such as expired source credentials, a permissions problem, or a destination that is unreachable). Keep the summary plain enough that whoever is on call can understand it at a glance.

Open a PagerDuty incident with Create Incident on my incident-response service, using the drafted summary for the incident title and details and setting an urgency that matches the severity you judged. Put the connection's name and id in the incident title so the same connection is easy to recognize later. Then post a short heads-up in Slack with Send a Message to my on-call channel, naming the connection and noting that an incident has been opened, so the team knows a pipeline is down.

Only raise one incident per failing connection per outage. Before opening a new PagerDuty incident, check the currently open incidents and make sure there is not already an open one for that same connection, keying the match on the connection's id and its current failure. If an incident for a connection is already open, leave it alone instead of opening a duplicate every 30 minutes, so a lingering outage does not spam on-call. Once a connection is syncing healthily again, it becomes eligible to alert again on a future failure.