From Artificial Analysis to production: the useful eval loop
How to turn Artificial Analysis, Arena, LiveBench, SWE-bench, and open leaderboards into a runnable shortlist, workload eval, and production model decision.
The fastest way to misuse an AI leaderboard is to read the first row, copy the model name, and call the decision finished. A public benchmark is much more useful as shortlist evidence: it tells you which candidates deserve a run on your work.
We built the app.nz eval hub around that boundary. It links the live source and methodology, says what question the benchmark can answer, calls out what it cannot answer, and then opens the closest model we can actually run.
Five sources, five different questions
| Source | Strong question | Important limitation |
|---|---|---|
| Artificial Analysis | How do broad capability, API speed, and price compare? | A composite index is not your task distribution. |
| Artificial Analysis media methodology | Which image outputs do blind voters prefer, and how do serverless price and generation time compare? | Standardized defaults do not reproduce your workflow or GPU. |
| Arena | Which response or media output do people prefer head to head? | Preference is relative and is not correctness, safety, or business value. |
| LiveBench | How do language models perform on fresh, objectively scored tasks? | Fresh questions still may not resemble your users. |
| SWE-bench Verified | Can a model-plus-agent resolve real repository issues? | Scaffold, tools, retries, and harness version materially change results. |
The old Hugging Face Open LLM Leaderboard is also a useful warning about treating one table as permanent. Hugging Face retired it in 2025 as model capabilities and evaluation needs changed; the maintained direction is now a directory of task-specific OpenEvals leaderboards.
Move from the source into a runnable candidate
For language and coding models, use Model Comparison to send exactly the same prompt through the gateway. It keeps output, elapsed time, and billed cost together. A leader that is slower, more expensive, or unreliable on the task may be a worse production choice.
For image and video models, the launch path matters even more:
- Hosted routes such as GPT Image 2, FLUX, and LTX 2.3 are ready behind platform APIs.
- Open workflows such as RealVisXL/SDXL, LTX-Video, and Wan VACE are in the Comfy workflow gallery. Their API-format JSON, model files, minimum VRAM, provenance, and verification state travel together.
- A public Hugging Face or Civitai weight can be linked from Comfy Studio and warmed into the global R2 cache before workers need it.
Those links mean “this is the closest runnable path,” not “we reproduced the leaderboard score.” Reproduction requires the exact prompts, settings, endpoint, and scoring protocol.
Build the acceptance set
After the shortlist, make a small dataset with four kinds of inputs:
- Normal traffic that represents most of the volume.
- Hard inputs that separate plausible candidates.
- Known failures from support tickets or previous models.
- Policy, safety, and formatting edges that must never regress.
Run it through Bulk Evals with a frozen rubric. Keep the individual failures; a mean score can hide the one case that matters. For media, store prompt, seed, workflow JSON, model hash, sampler settings, hardware, queue time, generation time, and the output artifact.
Optimize cost per accepted output
Per-token and per-second prices are inputs. The business metric is usually:
total model + retry + review cost / outputs that pass the acceptance bar
A cheap model that causes two retries and human repair can lose to an expensive model on final cost. A high-ranked model can lose to a smaller one when latency is the hard constraint. The eval loop makes both outcomes visible.
The complete path is now: public eval source → runnable model or Comfy workflow → representative regression set → production route. That is much more useful than another frozen leaderboard screenshot.