Run an eval across 5 frontier models on your own data in 10 minutes

Build it in four steps

1. STEP 01

   Bring your dataset

   A CSV with two columns: `input` (the user prompt) and `reference` (the ideal answer, if you have one — otherwise the judge scores on rubric alone). Fifty rows is usually enough to see a gap.

   # dataset.csv
   # input,reference
   # "Summarize this earnings call in 80 words: ...","<ideal summary>"
   # "Translate to Japanese preserving tone: ...","<ideal translation>"
   # ...

2. STEP 02

   Dispatch to five models in parallel

   One async loop, five tasks per row. The OpenAI SDK points at AIgateway; the only thing that changes between tasks is the model slug.

   import asyncio, csv
   from openai import AsyncOpenAI
   
   client = AsyncOpenAI(
       base_url="https://api.aigateway.sh/v1",
       api_key="sk-aig-...",
   )
   
   MODELS = [
       "anthropic/claude-opus-4.7",
       "openai/gpt-5.4",
       "moonshot/kimi-k2.6",
       "google/gemini-3.1-pro",
       "meta/llama-4.1-405b",
   ]
   
   async def run_one(model: str, prompt: str) -> str:
       r = await client.chat.completions.create(
           model=model,
           messages=[{"role": "user", "content": prompt}],
           extra_headers={"x-aig-tag": f"eval.{model.split('/')[-1]}"},
       )
       return r.choices[0].message.content or ""
   
   async def eval_row(row: dict) -> dict:
       outs = await asyncio.gather(*(run_one(m, row["input"]) for m in MODELS))
       return {"input": row["input"], "reference": row.get("reference", ""), **dict(zip(MODELS, outs))}

3. STEP 03

   Score with an LLM judge

   A small prompt asks Opus to score each candidate 0–100 against the reference (or against a rubric if you have none). One judge call per candidate per row — parallelized the same way.

   JUDGE_PROMPT = """You are a strict evaluator. Score the CANDIDATE 0-100 against the REFERENCE.
   Criteria: factual accuracy, completeness, style match. Reply JSON only: {"score": <int>, "reason": "<15 words>"}.
   INPUT: {input}
   REFERENCE: {reference}
   CANDIDATE: {candidate}"""
   
   async def judge(row: dict, model: str) -> dict:
       r = await client.chat.completions.create(
           model="anthropic/claude-opus-4.7",
           messages=[{"role": "user", "content": JUDGE_PROMPT.format(
               input=row["input"], reference=row["reference"], candidate=row[model]
           )}],
           response_format={"type": "json_object"},
           extra_headers={"x-aig-tag": "eval.judge"},
       )
       import json
       return json.loads(r.choices[0].message.content)

4. STEP 04

   Write the scorecard

   Average the per-row scores per model. Sort. Print the winner. That's the whole eval — under 40 lines and cheap enough to run before every prompt change.

   async def main():
       rows = list(csv.DictReader(open("dataset.csv")))
       evaluated = await asyncio.gather(*(eval_row(r) for r in rows))
   
       scores = {m: [] for m in MODELS}
       for row in evaluated:
           verdicts = await asyncio.gather(*(judge(row, m) for m in MODELS))
           for m, v in zip(MODELS, verdicts):
               scores[m].append(v["score"])
   
       board = sorted(((sum(s)/len(s), m) for m, s in scores.items()), reverse=True)
       for avg, m in board:
           print(f"{avg:6.1f}  {m}")
   
   asyncio.run(main())
   
   # => 84.3  moonshot/kimi-k2.6
   #    82.1  anthropic/claude-opus-4.7
   #    79.8  openai/gpt-5.4
   #    77.4  google/gemini-3.1-pro
   #    72.6  meta/llama-4.1-405b

FAQ

More examples