Flaky test triage board for Buildkite, with Linear quarantine

Build me a flaky test triage board on top of my Buildkite Test Engine data. Our QA lead and each team's code owners open it every week to work through the worst offenders. Buildkite is the data source and Linear is where quarantine tickets get filed. Nothing runs on a schedule; a person opens this and works out of it.

Main view. Start with a suite picker: use List Test Suites to show the organization's Test Engine suites, and Get a Test Suite for details on the one that is selected. Remember the suite choice so returning users land back where they were. Note for the build: Test Engine endpoints live under a separate /analytics path prefix (/analytics/organizations/{org.slug}/suites/...) and the Buildkite token needs the read_suites scope.

The ranked board. For the selected suite, pull the most recent test runs with List Test Runs (default the last 25, configurable) and for each of those runs pull Get a Test Run's Failed Executions. Aggregate the failed executions by test to compute three numbers per test: failure rate, meaning runs where the test failed divided by runs the test appeared in; how many of those recent runs the test actually touched; and when it last failed. Use List Tests for the suite roster so the app knows the full set of tests and can show or filter ones that never failed. Sort by failure rate descending so the worst offenders float to the top rather than being buried. Apply a minimum appearances threshold, default 3 runs, so a test that failed once in a single run does not outrank a chronic flake; show tests below the threshold in a separate low confidence section rather than dropping them. Columns: test name, scope or file, failure rate, runs touched, last failed as relative time, triage state, and the linked ticket if one exists.

Drill-down panel. Clicking a test opens a side panel so a human can judge whether it is a real bug or a genuine flake. Use Get a Test for the test's own record, and Find a Test to resolve a test's UUID by scope and name when a failed execution only gives you the name. In the panel show the recent failed executions for that test with their failure messages and expanded output, which run each failure came from, the branch if available, and timestamps. Group identical failure messages together with a count, because five failures with the same message read very differently from five with different ones.

Quarantine tickets, with deduplication first. The panel has a File quarantine ticket action. Before offering to create anything, run Search Issues in Linear using the test name plus a consistent marker such as the test scope. If a matching issue comes back, show it inline with its current status via Get Issue and link straight to it, instead of showing the create button, so nobody opens a duplicate. If nothing matches, use Create Issue in the configured Linear team with a title like "Quarantine flaky test: <test name>" and a description carrying the test name, its scope, the failure rate over the last N runs, how many runs it touched, when it last failed, two or three representative failure messages, and a link back to the Buildkite suite and test. Then optionally use Add Link to Issue to attach the suite link as an attachment on the issue. Once created, show the issue identifier and URL on that test's row so the board reflects it immediately.

Triage state that makes the board a working queue. Let any user mark a test as triaged or ignored, and persist that per organization rather than per user, so everyone works from the same queue. Store who set the state and when, and show that on the row. By default the board hides triaged and ignored tests so the queue empties as the week goes on, with toggles to bring them back into view. Put counters in the header: open flakes, triaged this week, and ignored. If a test that was marked triaged starts failing again in newer runs, surface it back into the open queue with a small "regressed" marker rather than leaving it hidden.

Details that matter: nothing in this app writes to Buildkite, it is read only there, and Linear is the only system it creates records in. Add a refresh control with a visible last-updated time, since aggregating runs takes a moment. Cap how many runs get fetched at once and show progress while aggregating so a large suite does not feel frozen. Handle the empty states properly: a suite with no recent runs, a suite where nothing has failed, and an organization with no Test Engine suites at all.