# Audit what Intercom's Fin AI actually resolved before you pay

> Review every conversation Fin closed as resolved, judge which ones actually stuck, and see what the gap is worth against your bill.

- Workflow type: app
- Services: Intercom, Google Sheets, Slack Bot
- Categories: Customer Support, Operations
- Published: 2026-08-24

## What it does

- Lists every conversation Fin closed as resolved across a date range you pick, ranked so the ones least likely to have stuck come first: the customer reopened it, a human took over straight after, or the rating was poor.
- Expands any row into the full conversation so you can read exactly what Fin said and what happened next, then record a verdict of genuine, not genuine, or content gap.
- Counts resolutions you were billed for against resolutions you judged genuine, and turns the difference into money at your own per-resolution price.
- Runs a Diagnose this one assistant that finds the help centre article which should have answered the question and drafts the fix, with nothing published until you approve it.

## What you'll need

- An Intercom workspace with Fin switched on, and access to read conversations and help centre articles
- Permission in Intercom to reply to, reopen and tag conversations, and to save article drafts
- A Google account and a spreadsheet you want the audit log written to
- A Slack workspace and the channel your support team works in
- Your per-resolution price from your Intercom bill

## Prompt

I want an app my support lead opens every week to audit what Intercom's Fin AI agent actually resolved. We are billed for every resolution Fin claims, and I suspect a good share of them did not stick. This is deliberately not a rebuild of Intercom's own Fin performance dashboards, which already report a resolution rate. The whole point of this app is the human verdict pass over individual resolutions, and the reconciliation of what we were billed against what we judged genuine.

The main view is a queue of resolutions. At the top I pick a date range. The app uses Intercom Search Conversations to pull every conversation that Fin took part in and that was closed or marked resolved inside that range, using the AI agent participation and conversation state filters, and paginating until it has the whole range. Each row shows the customer, the opening question, when Fin closed it, the conversation rating if there is one, and a did-not-stick score.

Rank the rows by did-not-stick signals, worst first. The signals are: the customer reopened the conversation after Fin closed it, a human admin replied or took the conversation over within a short window after the close (make that window configurable, default 24 hours), the conversation carries a poor rating, or the same contact opened another conversation soon afterwards. Show which signals fired on each row as small labels so the lead can see why a row is near the top. Rows with no signals still belong in the list, lower down, so the sample stays complete and the counter is honest.

Any row expands in place to the full transcript, read with Retrieve a Conversation, which returns up to 500 conversation parts. Show the parts in order with a clear visual distinction between the customer, Fin, and any human admin, and mark the point where Fin closed the conversation so the lead can see at a glance what happened after that moment.

On each row I record a verdict of genuine, not genuine, or content gap, plus an optional note. Store verdicts in the app, keyed by conversation id, along with who recorded it and when. Two leads should be able to split a week's queue without overwriting each other, and a row that has already been judged must show its existing verdict whenever that date range is loaded again.

At the top of the view, a counter compares resolutions billed against resolutions we judged genuine. Let me enter our per-resolution price in the app settings. Show: resolutions billed in this range, resolutions judged genuine, resolutions judged not genuine, how many are still unreviewed, and the money difference between billed and genuine at that price. This is the number I take to the Fin invoice, so make it the most prominent thing on the page and never hide the unreviewed count, since the gap is only meaningful next to how much of the range has actually been judged.

Each row has a Diagnose this one button that kicks off a background agent. The agent reads the full transcript with Retrieve a Conversation, works out what the customer actually needed, then lists our help centre with List All Articles and finds the article that should have answered the question. Because Fin answers from help centre content at query time, the fix for a bad resolution is always a specific article, so the agent must end in a concrete article proposal: either a revision of the article it matched, or a fresh article when nothing covers the question. It writes back into the app the question it identified, the article it matched or a note that nothing matched, its reasoning, and the proposed article body. At this stage it sends nothing to Intercom.

I review the agent's proposal inside the row. If I approve a revision, the app calls Update an Article on the matched article. If I approve a new one, it calls Create an Article. In both cases write the article in draft state and never published, so a human still makes the final call in Intercom. Show the agent's status on the row while it runs, as queued, running, or ready for review, and keep the finished proposal attached to the row so I can come back to it later.

From a row I can also act on the conversation itself: reopen a wrongly closed one with Open a Conversation, answer it with Reply to a Conversation, and mark it audited with Tag a Conversation using a tag I configure, defaulting to fin-audited. After any of these actions, refresh that row so its state and signals are accurate rather than stale.

Every verdict I record also appends a row to a Google Sheets audit log using Append Values, with the date, the conversation id and a link to it, the customer, the did-not-stick signals that fired, my verdict, my note, who reviewed it, and the billed price. Finally, a Post weekly summary button sends a message to our support channel with Slack Bot, covering the date range, resolutions billed against resolutions judged genuine, the money difference, the breakdown by verdict, and the top content gaps the diagnose agent found, so the team sees the same number I am taking to the invoice.

## How to customize

- Change how soon a human takeover counts as a sign the resolution did not stick, from the default of 24 hours.
- Set your own per-resolution price so the billed versus genuine gap is shown in the terms your invoice uses.
- Rename the tag applied to audited conversations, and pick which Slack channel the weekly summary goes to.

## FAQ

### Does this replace Intercom's own Fin reporting?

No. Intercom already shows you a resolution rate on its own dashboards. This app exists for the question those dashboards cannot answer, which is whether a human agrees each paid resolution actually stuck, and what the difference is worth in money.

### Will it change anything in Intercom without me?

No. Reopening, replying and tagging only happen when you press the button on a row. Any help centre article the assistant writes is saved as an unpublished draft for you to review, so nothing reaches customers on its own.

### What counts as a resolution that did not stick?

The app looks for the customer reopening the conversation, a human agent stepping in shortly after Fin closed it, a poor rating, or the same customer coming back with a new conversation soon after. Those signals only sort the queue. You still make the final call on every row.

### Do I have to review every conversation?

No. The queue is ranked so the most suspicious resolutions sit at the top. Most teams work the top slice each week, and the counter always shows how much of the range is still unreviewed.

### Can more than one person work through the queue?

Yes. Every verdict is saved with who recorded it and when, so two leads can split a week between them without overwriting each other, and each verdict is appended to your spreadsheet audit log.

Use this prompt in General Input: https://www.generalinput.com/prompts/audit-what-intercoms-fin-ai-actually-resolved-before-you-pay