RunPod serverless health watchdog with Slack alerts

Build me a RunPod serverless endpoint health watchdog that pages my on-call channel in Slack the moment an endpoint degrades. I want this running as a scheduled agent.

Trigger: cron, every 10 minutes.

Integrations: RunPod and Slack Bot.

Each run should do the following:

1. Call RunPod's List Serverless Endpoints to get every serverless endpoint on the account.

2. For each endpoint, call RunPod's Get Serverless Endpoint Health to read worker counts (ready vs initializing vs throttled) and queue stats (in-queue, in-progress, completed, failed, retried).

3. Decide a status for each endpoint: healthy, degraded, or down. Use rules along these lines, and treat them as defaults I can tweak: down = zero ready workers AND zero initializing workers AND queue depth > 0 for two consecutive checks; degraded = zero ready workers while queue depth > 5, OR failed jobs in the last interval > 10% of completed jobs, OR throttled workers > 50% of total workers for two consecutive checks; healthy = everything else. Make the thresholds easy to edit at the top of the agent instructions.

4. For every endpoint that is degraded or down, write a concise incident summary that names the endpoint, the symptom (e.g. "queue of 47 jobs with 0 ready workers"), and a probable cause (cold start storm, worker crash loop, queue backlog, throttled by max workers, sustained traffic spike). Suggest a recommended action: increase max workers, raise execution timeout, purge the queue, check the worker image for crash loops, etc.

5. Send one Slack message per incident to my on-call channel using Slack Bot's Send a Message. Make me set the channel as a parameter. Format the message clearly: a status emoji and endpoint name on the first line, then the symptom, probable cause, and recommended action as short bullets. Keep it scannable.

6. Stay silent when every endpoint is healthy. Do not send a "nothing to report" message. The goal is zero alert fatigue.

Nice-to-have refinements:

- Track previous-run state per endpoint so a degraded endpoint that stays degraded does not re-alert every 10 minutes. Send a follow-up only when the status changes (e.g. degraded becomes down, or recovers to healthy).

- Skip disabled or paused endpoints entirely.

- Allow an optional name pattern or tag filter so I can monitor only my production endpoints and ignore experimental ones.

- When the watchdog itself fails to reach RunPod (auth error, 5xx, timeout), post a single "watchdog could not reach RunPod" message to the same Slack channel instead of failing silently.

Audience for the Slack message is the on-call engineer. Keep tone factual and short. No emojis beyond a single status indicator. No marketing copy.