Model bake-off bench for fal.ai image and video models

Build me a model bake-off bench for fal.ai that my creative and engineering team opens every time we need to decide which generation model to use for a job. The point is to turn model choice from a recurring argument into a measurable decision, and to leave behind a house register of what we picked and what it cost.

The main surface is the bench. I type one plain English brief, pick a job type from product shot, hero illustration and social video, and set a cost ceiling for the run. Then I choose candidate models from the fal.ai catalog. A handler uses List Models to browse and filter the catalog down to models that suit the selected job type, and it should request them with expand=openapi-3.0 so the app has each model's own input schema and can build a valid payload per model instead of assuming every model takes the same fields.

Before I commit, show me the projected spend. A handler calls Get Model Pricing for each shortlisted model and Estimate Pricing for the planned grid, and the bench shows the per-model unit price, the projected cost of the whole grid, and whether that lands under my cost ceiling. If the lineup would blow the ceiling, say so plainly and let me drop models until it fits.

Hitting Run fires every candidate in parallel. Use Submit Queue Request (Async) to fire the runs and Stream Queue Request Status to drive a live status on each tile, so I watch the grid fill in rather than staring at one spinner. For the models that have dedicated operations, use them: Generate Image with FLUX1.1 [pro], Generate Image with FLUX.1 [dev], Generate Image with Recraft V3, Generate or Edit Image with Nano Banana, and Generate Video with Kling, Generate Video with Google Veo 3, or Generate Video with MiniMax Hailuo. Record the wall clock time from submit to completion for every single run.

The results view is a side-by-side grid, one tile per candidate model. Every tile shows the generated image or video, what that run cost, how long it took, and the cost per output. Never show me only a grid total, because cost per output is the number I actually compare on and it has to be on every tile. Reconcile the billed figure with Get Model Usage once the runs land, so the tile shows what fal.ai really charged rather than just the estimate. Keep the tuned prompt used for each tile visible or one click away so any run is reproducible.

Bake in a blind mode and have it on by default. While blind mode is on, tiles are labelled Model A, Model B, Model C with model names and vendor branding hidden, so nobody just votes for the brand they recognize. Cost and generation time stay visible, because those are part of the judgement. As soon as I mark a favorite, reveal every name at once and tell me whether the one I picked was the cheap option or the expensive one.

Add a Run the bake-off button that kicks off a background agent. The agent reads my plain English brief and the job type, browses the catalog with List Models to shortlist candidate models that suit the job and fit my cost ceiling, checks Get Model Pricing and Estimate Pricing to keep that shortlist inside budget, writes a tuned prompt for each candidate built from that model's own input schema from List Models with expand=openapi-3.0 rather than reusing one generic prompt everywhere, fires the whole grid, waits for the runs to finish, and then writes a recommendation back into the app: which model it would pick for this job, why, and what the tradeoff against the runners up is in quality versus cost. The recommendation lands next to the grid with its reasoning visible so I can agree with it or overrule it.

When I pick a winner it saves to a house register in Google Sheets using Append Values: the date, the job type, the brief, the winning model, the cost per output, the generation time, the runners up, who picked it, and a link to the output. The register view inside the app reads that sheet back with Get Values and is the thing people come back to, so make it good: which model we chose for which job, at what cost, and when we last checked. Let me group it by job type, show the last checked date on every row, and flag any decision older than 90 days as worth re-running, because prices and models move fast.

Give me a button to push a comparison to a channel with the Slack Bot Send a Message operation: the brief, the job type, the winning model with its cost per output, and the runners up with theirs, so the team sees the decision without needing to open the app.

House rules across the whole app. Model input schemas differ per model on fal.ai, so always build each payload from that model's own schema instead of a shared shape. Every price shown is in US dollars. Keep past bake-offs browsable so I can reopen one and see the grid exactly as it was. And treat the register as the product: if someone opens this app and only ever reads the register, it should still have earned the visit.