Screen Typeform survey answers for AI-generated responses

Every morning at 6am, screen my Typeform survey's open-ended answers for AI-generated text so bot and LLM-assisted submissions do not quietly poison my research data. Every step is fixed, so keep this fully deterministic with no judgement calls at runtime.

Start with Typeform "Retrieve Responses" to fetch the responses submitted since the previous run. Keep a watermark of the latest submission timestamp I actually retrieved and use that as the lower bound on the next run, rather than the clock time the run started. Typeform can take around 30 minutes to expose very recent submissions through the responses API, so anchoring on retrieved data instead of run time keeps late arrivals from being skipped forever. If no new responses came in, finish the run without writing anything or alerting.

For each response, collect the free-text answers and join them into a single block of text per response, keyed by the response ID. Skip any response whose combined text is shorter than my minimum length, defaulting to 25 words, because it is too short to score reliably. Record those as skipped with the reason rather than dropping them silently, so the log still accounts for every response.

Submit the remaining texts together with Pangram "Create Bulk AI Detection Job", one item per response. Bulk requests are capped at 1,000 billable units, where a unit is one started word block per item, so estimate the units before submitting and split a large batch into multiple sequential jobs when it would exceed the cap. The job is asynchronous, so poll it until it reaches a terminal status of succeeded, failed, or partial.

Read the scores back with Pangram "Get Bulk Job Results", paging until every item is retrieved. Handle partial jobs properly: individual items can fail with a per-item error string such as "Text must contain at least one valid token", so treat those as unscored and log them next to the too-short skips instead of failing the whole run. Bulk results are only retained for 48 hours after completion, so write everything to the spreadsheet in the same run rather than deferring it.

Append one row per response to my Google Sheets tracking tab with "Append Values", containing the response ID, the submission date, the AI likelihood score, and a verdict of clean or suspect. Mark a response suspect when its AI likelihood score is at or above my threshold, defaulting to 0.75. Responses that could not be scored carry their reason in the verdict column instead of a score. Write rows in submission date order so the tab reads as a running audit log I can filter before analysis.

Finally, work out the share of suspect responses among the scored responses in this batch. If that share is at or above my alert percentage, defaulting to 10 percent, send a Slack alert with "Send a Message" naming the suspect count, the number scored, and the percentage, along with the form name and the batch date, so I can pause the panel or tighten screening before I waste more incentive budget. Stay silent when the batch comes in under the threshold.