How to optimize prompts with evals (LLM-as-a-judge)

Most teams write a prompt, eyeball a few outputs, and ship. That works until it doesn't — and you have no way to tell whether a change made things better or worse. Here is the loop we use instead, and the two tools that run it for you.

Treat prompts like code: test them

The core idea is LLM-as-a-judge. You write a second prompt — the eval — whose only job is to score an output from 0 to 100 against a rubric. Now any prompt change is measurable: run it over a dataset, judge every output, and compare the mean and median.

input row ──▶ [ your prompt + model ] ──▶ output
output + rubric ──▶ [ judge model ] ──▶ score 0–100
aggregate across rows ──▶ mean / median / pass-rate

Step 1 — build a tiny dataset

You do not need thousands of rows. 15–25 representative inputs catch most regressions. Paste them, one per line, into Bulk Evals, write your prompt and a rubric, and run. You get a per-row score plus aggregate metrics in one pass.

A good rubric is specific and bounded:

Score 0–100. 100 = the reply resolves the issue with a concrete next step and a friendly tone. Deduct for vagueness, missing steps, or wrong information. 0 = generic or incorrect.

Step 2 — let the optimizer search for you

Hand-tuning prompts is slow. The Prompt Optimizer takes your base prompt, your eval, and your dataset, then:

1. Scores your base prompt across every row.
2. Generates several improved variants — seeded with your lowest-scoring cases so it fixes real failures.
3. Re-scores every variant across the whole dataset.
4. Returns the winner by mean score, with the full leaderboard.

It is a small, practical cousin of techniques like DSPy and OPRO: optimize against a metric instead of vibes.

Why mean and median

A model that scores 100 on most rows but 0 on a few has a high mean and a lower median — that spread is the signal. We sort candidates by mean, break ties by median, and surface pass-rate (the fraction scoring ≥ 60) so you can see consistency, not just average quality.

Close the loop

Once you have a winner, run it across more models with the Model Comparison tool to find the cheapest one that still passes, estimate cost with the Token & Cost Estimator, and ship it through the gateway or your agent. Same credits, same account, one loop.