Page on-call when a production deploy fails in Buildkite

The moment a build fails on one of your production deploy pipelines, automatically open a PagerDuty incident so on-call gets paged right away.

Deterministic Code
BuildkitePagerDutyEngineeringNotifications & Alerts
PromptCreate

When a build finishes on one of my Buildkite pipelines, run this workflow automatically. Trigger on Buildkite's build.finished webhook, and only continue when the build's state is failed and its pipeline is on my allowlist of production deploy and release pipelines. Any other pipeline, or any non-failed state, should stop the workflow immediately so routine feature-branch failures never page anyone. No AI reasoning is needed: this is a fixed, deterministic mapping from the failed-build payload to a PagerDuty incident.

First, read the full build using Buildkite's Get a Build operation for the pipeline slug and build number from the webhook payload. This gives me the build number, branch, commit hash, commit author, build message, and the web URL of the build.

Then create a PagerDuty incident using PagerDuty's Create Incident operation on my deployment service. Set the incident title to name the pipeline, the branch, and the build number, for example 'Deploy failed: <pipeline> on <branch> (build #<number>)'. Set the incident body to include the commit hash, the commit author, and a direct link to the failed build in Buildkite so on-call can jump straight to it.

Derive the incident urgency from the pipeline: release and production deploy pipelines should page at high urgency, while lower-priority deploy pipelines can page at low urgency. Keep the mapping from pipeline name to urgency explicit so it is easy to adjust. Because only allowlisted pipelines reach this step, on-call is only paged for the deployments that actually matter.

Additional information

What does this prompt do?
  • Watches your production deploy and release pipelines and reacts the instant a build fails.
  • Opens a PagerDuty incident on your deployment service so the on-call engineer is paged.
  • Fills the incident with the pipeline, branch, build number, commit, author, and a direct link to the failed build.
  • Ignores routine feature-branch failures so on-call is only paged for the pipelines that matter.
What do I need to use this?
  • A Buildkite account with the deploy and release pipelines you want to monitor.
  • A PagerDuty account with a deployment service and an on-call schedule set up.
  • The list of pipeline names that are allowed to page on-call, so feature branches never do.
How can I customize it?
  • Choose exactly which pipelines are allowed to page on-call.
  • Set the urgency (high or low) per pipeline so release builds can page louder than others.
  • Adjust what goes into the incident title and body, such as the branch, commit, and build link.

Frequently asked questions

Will this page on-call for every failed build?
No. It only pages for builds that fail on the specific pipelines you allowlist, so routine feature-branch failures never wake anyone up.
Does this use AI to decide when to page?
No. It is a fixed, deterministic mapping from the failed build to an incident, so it behaves the same way every single time.
Can I control how urgent the incident is?
Yes. Urgency is set per pipeline, so you can make release and production deploys high urgency while other pipelines page at low urgency.
What information ends up in the PagerDuty incident?
The pipeline, branch, and build number go in the title, and the commit, author, and a direct link to the failed build go in the body so on-call can jump straight to it.
Do I need to write any code to set this up?
No. You connect Buildkite and PagerDuty and provide your list of deploy pipelines, and the workflow handles the rest.

Stop missing failed production deploys.

Page the right on-call engineer automatically the moment a deploy pipeline goes red in Buildkite.