Compare Fireworks AI models against your golden test set

I want an internal app my AI team opens every time we change a prompt or swap models on Fireworks AI. The main screen is a bench for our golden test set: rows are the test cases, columns are the Fireworks AI models we are comparing, and every cell shows what that model answered, the score a judge gave it, and how long it took.

The test cases live in a Google Sheet. Read them with Get Values from a Cases tab where each row is one golden case with an id, the input prompt or question, an ideal answer or reference notes, and a category tag. Read the scoring rubric and the system prompt we are testing from a second Rubric tab with Get Values as well, so the people who own the prompt can edit both in the spreadsheet without anyone touching the app. Let me enter the spreadsheet id and the tab names on a settings screen and remember them, and give me a Reload button that re-reads the sheet so a case someone added mid-session shows up.

Populate the model columns from Fireworks AI List Models. Show the available models as a checklist I pick from before a run and remember my selection between visits. Keep the fully qualified model path internally, since Fireworks names look like accounts/fireworks/models/llama-v3p1-70b-instruct rather than a bare slug, but show a short readable name in the column header.

Hitting Run evaluates every selected case against every selected model. For each case and model pair, call Create Chat Completion with the system prompt from the Rubric tab and the case input as the user message, and record the answer, the wall-clock latency in milliseconds, and the prompt and completion token counts the response returns. Then run a second Create Chat Completion pass that acts as the judge: send it the rubric, the case input, the ideal answer, and the model's answer, and ask for a numeric score plus a one-line reason. Let me choose which model does the judging, defaulting to the strongest one I have selected, and keep the judge model fixed for the whole run so the scores stay comparable. Fill the grid in progressively as results land instead of showing a blank spinner, and pace the calls so a large grid does not trip Fireworks rate limits.

Each cell shows the score, the latency, and the first line of the answer. Clicking a cell opens the full answer with the judge's reason beside the ideal answer so I can argue with the score. Column footers show that model's pass rate against the rubric threshold, its average score, its median latency, and its total tokens for the run. If a model errors or times out on a case, show that in the cell rather than silently scoring it zero, and let me re-run just that one cell.

Add a cost row under each model column pulled from Fireworks AI Get Billing Usage for the run window, broken down by model, so I can see what the run actually cost per model. Make the tradeoff obvious on the screen: mark which models cleared the rubric threshold and rank those by cost, because the cheapest model that still passes the rubric is usually the right answer.

When a run finishes, append one row per model per run to a History tab in the same spreadsheet using Append Values, capturing the timestamp, the prompt version label, the model path, the number of cases run, the pass rate, the average score, the median latency, and the cost. Read that tab back with Get Values to power a History screen and to compare the current run against the previous baseline run for the same model: show pass rate now versus then with the delta, and list the individual cases whose score dropped. Let me set the score drop that counts as a regression, and have the app state plainly whether this run passes or regresses against the baseline.

Add an Investigate regressions button that kicks off a background agent. The agent takes only the cases whose score dropped versus baseline, re-runs each of them with Create Chat Completion against both the current and the previous model or prompt version, reads the old and new outputs side by side against the rubric, and decides whether each drop is a real quality regression, judge noise, or a case that was always borderline. It writes a plain-language diagnosis back into the app: what broke, which cases show it, whether it is worth blocking the change, and a clear recommendation naming the model and the prompt version to ship. Show the diagnosis in a panel on the bench attached to the run it belongs to, keep past diagnoses readable, and show that the agent is working while it runs.

A few things to bake in. The golden set stays small on purpose because it has to run on every prompt change, so design the grid for roughly 20 to 200 cases rather than thousands. The prompt version is a label I type before starting a run, and it gets stored with the run so history stays comparable. Anyone on the team can open the app and read it, and running an evaluation or investigating regressions is a button any of them can press.