Triage Hightouch sync failures into Slack and Linear tickets
Every 15 minutes, scan Hightouch for newly failed syncs, post a triage summary in Slack, and open a Linear ticket when a sync keeps failing.
Build an agent workflow that triages Hightouch sync failures and gets them in front of our data team fast.
Trigger: cron, every 15 minutes.
On each tick, the agent should:
1. Call Hightouch List Syncs to enumerate every sync in the workspace.
2. For each sync, call Hightouch List Sync Runs and identify any runs that newly entered an error or aborted state since the previous tick. The agent must remember the last seen run id per sync between runs so we only alert on genuinely new failures, never re-alert on the same failed run twice.
3. For each newly failed run, gather a short context block: destination name, model name, rejected row count vs added/changed/removed row counts, the failure reason or error message from Hightouch, and whether the previous 5 runs of that sync were also failing.
4. Post a Slack message via Slack Bot Send a Message to a configurable channel (default #data-alerts) with that triage summary plus a direct link back to the sync in the Hightouch app. One Slack message per failed sync, formatted for quick scanning.
5. For any sync that has failed 3 or more consecutive runs (a true incident, not a flake), also escalate to Linear. Before filing, use Linear Search Issues to look for an existing open issue mentioning that sync by name in the data team. If one already exists, skip Linear and only post the Slack message. If none exists, call Linear Create Issue in the data team with a clear title like "Hightouch sync <sync name> failing repeatedly", the full diagnostic context (destination, model, recent run history, last error) in the description, and priority High.
Important behavior:
- Persist per-sync state (last seen run id, consecutive-failure count) across ticks. On the very first run, treat all currently-known runs as the baseline and do not alert retroactively.
- Never duplicate a Linear ticket. Search by sync name first, every time.
- A single-run failure should only ping Slack, never Linear. Linear only opens once we cross the 3-consecutive-failure threshold.
- Disabled or paused syncs in Hightouch should be ignored.
- If Hightouch rate-limits the run, back off and finish on the next tick rather than erroring the whole workflow.
Integrations: Hightouch (List Syncs, List Sync Runs), Slack Bot (Send a Message), Linear (Search Issues, Create Issue).
Additional information
What does this prompt do?
- Runs every 15 minutes and only flags syncs that just entered an error or aborted state, so you do not get pinged about the same failure twice.
- Posts a clear triage summary to your data alerts Slack channel for each new failure: destination, model, rows rejected versus changed, the failure reason, and whether the previous runs also failed.
- Opens a high-priority Linear ticket only when a sync has failed three runs in a row, so flaky one-off errors do not clutter your backlog.
- Checks Linear for an existing open ticket on the same sync before filing a new one, so you never end up with duplicate work in the queue.
What do I need to use this?
- A Hightouch workspace with API access. A workspace Admin can generate the key from Settings.
- A Slack workspace, with a channel you want failure alerts to land in (for example, #data-alerts).
- A Linear workspace, with a team that owns data quality where escalation tickets should be filed.
How can I customize it?
- Change the cadence. The default is every 15 minutes, but you can run it every 5 minutes during business hours or hourly overnight.
- Route alerts to the right place. Send all failures to one channel, split by destination (Salesforce vs HubSpot vs Iterable), or DM an on-call data engineer for the worst ones.
- Tune the escalation threshold. Three consecutive failures is the default for opening a Linear ticket, but some teams escalate sooner or only file tickets for production destinations.
Frequently asked questions
Will it alert on the same failed run over and over?
How does it avoid filing duplicate Linear tickets for the same broken sync?
What actually counts as a failure?
We already use Hightouch's built-in alerts. What does this add?
Can we route different syncs to different Linear teams?
Stop scrubbing Hightouch for failed syncs.
Connect Hightouch, Slack, and Linear once, and Geni triages every new sync failure for your data team every 15 minutes.