Review every email your AI agents sent before customers complain

Build me an app for reviewing the email my AI agents send out. The person using it is the ops lead who owns our agent fleet, not an engineer. They sit down each morning, work through what the agents sent, and grade it, because right now nobody checks any of it until a customer complains.

The main screen is a sent mail review queue. At the top there are controls for the review window (presets for Today, Last 7 days, Last 30 days, plus a custom range), an inbox filter, a status filter (Unreviewed, Flagged, Good, All), and a sample toggle. To load the queue, call AgentMail List Inboxes to get every inbox, then for each inbox call List Messages filtered to the sent label and the selected date range. AgentMail list endpoints paginate with next_page_token and return newest first, so page through until the range is covered or a page cap is reached, and give the user a Load more control rather than assuming one fetch returns everything. Remember that inbox_id is the full email address, so URL encode it wherever it goes into a path.

Each row shows the inbox the message came from, the recipient, the subject, how long ago it went out (for example "3h ago"), and a review badge. Sort unreviewed messages first, newest first within that group, then everything already reviewed. When someone opens a row, mark it as in review by that person for the next fifteen minutes so other reviewers can see it is already being handled and two people never grade the same message twice.

Clicking a row opens a detail pane. Fetch the full message with AgentMail Get Message and render the body, preferring HTML and falling back to plain text, with the recipient, sent time, labels and any attachments. Next to it show the conversation the message belongs to: take the thread_id off the fetched message and show the other messages from that inbox sharing the same thread_id, oldest to newest, each one expandable through Get Message. There is no separate thread endpoint, so build the thread view by matching thread_id against a slightly wider List Messages window for that inbox.

In the detail pane the reviewer marks the message Good or Flagged. Flagging requires a reason picked from a short list (wrong facts, off brand tone, overpromised, should have escalated to a human) that an admin can edit in settings, plus an optional free text note. Save the verdict in the app keyed by inbox id and message id, recording the verdict, reason, note, the reviewer who made it and the timestamp, and make those verdicts visible to everyone on the team. Add keyboard shortcuts so a reviewer can move through the queue quickly.

Nobody wants to read a thousand emails, so default the queue to a sample of the ten newest unreviewed messages per inbox, with the sample size configurable in settings and a switch to show everything when someone wants a full audit.

Flagging a message reveals three follow up actions. The first is Linear Create Issue, filed against the team that owns our agent prompts (populate the team picker with Linear List Teams and remember the default in settings). Prefill the title with the flag reason and subject, and the description with the recipient, the sending inbox, the sent time, the flag reason, the reviewer note and the quoted message body. Show it as an editable draft before creating anything, then store the returned issue identifier and URL on the verdict and surface it as a link on the queue row so it is obvious a fix is already tracked.

The second is Slack Bot Send a Message to our team channel, for anything that needs a same day human response. Pick the channel from Slack Bot List Channels and remember it as the default. The message should carry the inbox, recipient, subject, flag reason, reviewer note and the Linear link if one was created. Show an editable preview before posting, and record on the verdict that it was escalated.

The third is a Reply To Message button so the reviewer can send the correction from the original inbox without leaving the review screen. Open a composer prefilled with a quote of the original message, let the reviewer write the correction, and send it with AgentMail Reply To Message so it stays on the same thread. Record the correction text alongside the flag, because the fix is as useful to keep as the complaint.

Nothing sends by itself. Every outbound action, whether it is the ticket, the Slack post or the email reply, opens as an editable draft and requires an explicit confirm.

Across the top of the queue, show a header strip with one tile per inbox for the selected window: how many messages that inbox sent, what share of them has been reviewed, and its flag rate, plus a totals tile for the whole fleet. Clicking a tile filters the queue to that inbox, and the tile with the worst flag rate should stand out so I can tell at a glance which agent is drifting.

Add a second tab called Trends, built only from the saved verdicts so it needs no extra fetching: flag rate per inbox week by week for the last eight weeks, the most common flag reasons, and how many messages each reviewer graded.

A few practical things: cache each inbox and range listing for the session with a manual Refresh, back off and show partial results if AgentMail rate limits instead of failing the whole screen, and give the queue a clean empty state for when everything in the window has already been reviewed.