Weekly Railway and Sentry reliability report to Notion

Every Friday at 16:00 in our team timezone, build a weekly reliability report and write it as a single page in Notion so the on-call handoff is actually useful. This should be an agent workflow on a cron trigger.

Step 1. Gather Railway deploys. Use Railway's List Projects to find our production project(s), then for each production service use List Deployments to pull every deployment from the last 7 days. For each deployment capture: service name, status (success, failed, crashed, removed), commit SHA and message if available, the deploy URL, the staticUrl/domain, and the createdAt and finishedAt timestamps. Mark a deploy as a rollback if it was triggered as a rollback or a redeploy that immediately followed a failed deploy on the same service.

Step 2. Gather Sentry signal. Use Sentry's List Organization Issues filtered to the same 7-day window, sorted by frequency, to get the active issues. For the top ~10 issues, also call List Project Error Events to get accurate event counts and the firstSeen/lastSeen timestamps. Capture the Sentry issue URL, project slug, issue title, level (error/fatal/warning), event count for the week, firstSeen, and the service or environment the issue belongs to.

Step 3. Correlate. We do NOT use Sentry's Create a Deploy to mark deploys, so do the correlation post-hoc by timestamp. For each Sentry issue, check whether its firstSeen (or a clear spike in event count) falls within 1 hour AFTER a Railway deployment on a service whose name maps to the Sentry project (use fuzzy/substring matching on service name to project slug). If yes, flag it as a likely regression and link it to that specific deployment. Also flag: services with one or more failed deploys followed by a rollback, and services that had zero deploys AND zero new Sentry issues for the week (the "quiet" services).

Step 4. Write a single Notion page. Use Notion's Create a Page with the parent set to a configured "Engineering / On-call" page (the user will paste the parent page ID during setup). Title it something like "Weekly reliability report — week ending {Friday date}". The page body should be one concise page (not a database, no child pages) with these sections in order: Week summary (1-2 sentences plus headline numbers: total deploys, failure rate, new regressions, quiet services); Deploys by service (a simple list or table: service, deploy count, success/failure counts, failure rate); New regressions linked to deploys (each item: Sentry issue title, event count, firstSeen, the Sentry issue URL, and the Railway deployment URL it correlates to); Quiet services (just a list); Recommendations for next week (3-5 short bullets the agent infers from the data, e.g. "investigate flaky deploys on api-worker" or "add Sentry alerting for billing-service since it's been silent for 3 weeks").

Keep the page concise and skimmable. No Slack post. The user will share the Notion page link manually. If the week was quiet (no deploys or no new errors), still write the page and say so plainly in the summary.

Additional information

What does this prompt do?

Pulls every production Railway deploy from the last seven days with status, service, commit, and timestamp.
Pulls the same week of Sentry issues and flags the ones that spiked within an hour of a deploy.
Calls out failed deploys followed by rollbacks, and the quiet services that had no deploys or new errors.
Writes one clean Notion page under your Engineering or On-call parent, with linked deploys and issues plus recommendations for the next week.

What do I need to use this?

A Railway account with a token that can see your production projects and deployments.
A Sentry account with read access to your organization's issues and projects.
A Notion workspace, plus a parent page (for example Engineering or On-call) shared with the General Input connection.

How can I customize it?

Change the day and time. Default is Friday at 16:00 in your team's timezone, but Monday morning works just as well.
Adjust the lookback window. Seven days is the default, but two weeks is a popular pick for slower-moving teams.
Tighten the deploy-to-error correlation window. The default flags issues that started or spiked within an hour of a deploy.
Swap the parent Notion page or rename the report headings to match your on-call playbook.

Frequently asked questions

Do I need to mark my Railway deploys inside Sentry for this to work?

No. Most teams skip that step, so the report correlates deploys to errors by timestamp instead. If you do use Sentry releases, the report still works and the correlation only gets sharper.

Does it post to Slack as well?

No. The output is a single Notion page so on-call has one durable link to share. You can paste that link into Slack or Teams manually if you want to broadcast it.

Will the Notion page be a database or a normal page?

A normal single page. It keeps the handoff short and skimmable instead of spawning a new database row every week.

What happens if there were no deploys or no new errors that week?

The report still gets written, with a short summary that calls out the quiet week and lists which services stayed silent.

Can I run it on demand instead of waiting for Friday?

Yes. The schedule is just the default trigger. You can run the same workflow ad hoc any time you want a fresh report, for example before an incident review.