Did a secret change break production? Correlate Sentry with Doppler
Every 15 minutes, check new Sentry errors against recent Doppler secret activity and tell on-call in Slack whether a secret change is the likely cause.
Every 15 minutes, sweep Sentry for new high-severity errors in our production projects and tell on-call in Slack whether a recent Doppler secret change is a likely cause. Trigger: cron, every 15 minutes.
Project mapping (edit this list to match your setup). Each line maps one Sentry project slug to one Doppler project and its production config:
- sentry:web-api -> doppler:web-api / prd
- sentry:worker -> doppler:worker / prd
- sentry:web-app -> doppler:web-app / prd
Also configure: Sentry organization slug, the Slack channel id or name to post into (e.g. #oncall), and the time window for de-duplication (default: the last sweep window, so 15 minutes). If you do not yet have a project mapping, ask me for it before publishing.
Step 1: pull new Sentry issues. Use Sentry's List Organization Issues for the configured organization, filtering to unresolved issues with level error or higher, sorted by first-seen descending. Keep only issues whose first-seen timestamp falls inside the current sweep window (the last 15 minutes by default). For each kept issue, also call List an Issue's Events with a small limit (e.g. 1-3 latest events) to get the affected project slug, the first-seen timestamp, the error message, and the top stack frames. Carry the issue's permalink forward.
Step 2: de-duplicate. Track which Sentry issue ids have already been posted in this sweep and skip any repeats so each issue gets at most one Slack message per run.
Step 3: pull recent Doppler activity for the matching project. Look up the Sentry project slug in the mapping above. If there is no mapping entry, post the Slack message with a 'no Doppler mapping configured for this project' note instead of correlating. Otherwise, for the mapped Doppler project and production config, call List Activity Logs and List Config Logs and keep entries from the last 24 hours. If you need to confirm the config exists, you may call List Configs for the project first.
Step 4: reason about correlation. For each Sentry error, decide which secret-name family the error domain belongs to and check whether any recent Doppler change touched a secret whose name matches that family. Use these patterns as the starting rules (this section is meant to be edited by the user to match their own naming conventions):
- Auth / 401 / 403 / 'unauthorized' / 'forbidden' / 'invalid token' -> AUTH_*, OAUTH_*, JWT_*, API_KEY_*, *_SECRET, *_TOKEN
- Database / connection / 'ECONNREFUSED' / 'timeout' / 'pool' -> DATABASE_URL, DB_*, POSTGRES_*, MYSQL_*, REDIS_*, MONGO_*
- Payment / billing / charge / subscription -> STRIPE_*, PAYPAL_*, BRAINTREE_*
- Storage / upload / bucket -> S3_*, GCS_*, AWS_*, AZURE_STORAGE_*
- Email / SMTP / sendgrid -> SMTP_*, SENDGRID_*, MAILGUN_*, POSTMARK_*
A Doppler change counts as suspect when (a) the change landed before the Sentry error's first-seen timestamp and within the last 24 hours, and (b) the changed secret name matches the error's family. If multiple changes match, pick the most recent one before the error. Compute minutes_before_error = first_seen_at minus change_timestamp, in minutes.
Step 5: post to Slack. For each new Sentry issue, send one message via Slack Bot's Send a Message to the configured on-call channel. Format:
Header line: ':rotating_light: New Sentry error in <service>' with a link to the Sentry issue.
Body lines: error message (one line), first-seen time in UTC, affected project.
Verdict line, one of:
(a) ':warning: Possible secret-change cause: <SECRET_NAME> was <action> by <actor> <N> minutes before the error.' followed by a short link or reference to the Doppler change.
(b) ':white_check_mark: No correlating Doppler changes in the last 24h, secrets are unlikely to be the cause.'
Keep each Slack message compact (one message per Sentry issue, no threads, no @here unless the user adds it). Do not include actual secret values, ever. Only secret names.
Guardrails: never modify Sentry issues, never modify Doppler secrets, never resolve or reopen anything. This workflow is read-only on Sentry and Doppler. Only Slack is written to. If Sentry returns zero new issues in the window, exit quietly without posting.
Additional information
What does this prompt do?
- Sweeps Sentry every 15 minutes for new high-severity errors in your production projects.
- For each new error, checks the matching Doppler production config's recent activity for any secret rotation, value update, or deletion in the last 24 hours.
- Reasons about whether the changed secret's name plausibly matches the error (auth errors with auth keys, database errors with database URLs, payment errors with payment keys, and so on).
- Posts one Slack message per new error to your on-call channel with either the suspect secret change, the actor, and how long before the error it landed, or a clear 'no recent secret changes' so the team can stop looking and move on.
- De-duplicates so the same Sentry error never gets posted twice in the same sweep.
What do I need to use this?
- A Sentry account with read access to your production projects.
- A Doppler account with read access to the production configs you want correlated.
- A Slack workspace with a channel for on-call alerts.
- A short note mapping each Sentry project to its matching Doppler project and production config, for example 'sentry web-api maps to doppler web-api production'.
How can I customize it?
- Edit the project mapping to cover every Sentry project you want watched and its matching Doppler production config.
- Adjust the secret-name patterns so they match your team's naming conventions (for example add MONGO_, KAFKA_, or VENDOR_X_ groups).
- Change the cadence (every 5, 15, or 30 minutes), the severity floor, or the Slack channel the alert posts to.
Frequently asked questions
Does this auto-rollback secrets in Doppler?
What if there are no recent secret changes when an error fires?
Can it watch more than one Sentry project?
How does it decide a secret change is suspicious?
Will the same error get posted on every run?
Stop guessing whether the last secret change broke prod.
Connect Sentry, Doppler, and Slack once. Geni sweeps every 15 minutes and tells on-call exactly what changed before each new error.