Ranking models with ELO: evals your team will trust

Model selection is one of the highest-leverage decisions an AI platform team makes. The wrong choice costs money, degrades user experience, or introduces subtle quality regressions that take weeks to surface. Most teams reach for leaderboard scores — MMLU, HumanEval, LMSYS Arena ratings — and discover they don’t predict performance on their actual workload.

ELO-based pairwise evaluation does. Here’s why it works, and how to run it against your own traffic using ManyLayers Gateway and Workspace’s eval pipeline.

Why aggregate benchmarks mislead

Aggregate benchmarks measure average performance across a standardized dataset. Your workload is not that dataset. If your support bot handles a narrow domain — medical device compliance questions, say — a model that scores well on broad academic benchmarks may underperform a smaller, domain-adapted model on your specific prompt distribution.

ELO ranking avoids this by comparing models directly on your traffic. A human reviewer (or an LLM judge) sees two responses to the same prompt and picks a winner. The ELO algorithm converts pairwise outcomes into a stable ordinal ranking. Models that consistently beat others on your actual prompts rise; models that lose fall — regardless of their leaderboard position.

There’s a second problem with aggregate benchmarks: they measure capability at a point in time on a static dataset. As models are updated by providers, benchmark scores may not be refreshed. And benchmarks create an adversarial dynamic: model providers optimize for benchmark performance specifically, which can inflate scores without improving real-world utility.

Setting up the eval pipeline

ManyLayers Gateway supports a shadow-request mode: when you configure two model targets on the same alias at a 50/50 split, the gateway serves one response to the user and logs both responses to the ManyLayers Workspace eval queue. Neither user sees additional latency — the second response is generated asynchronously.

aliases:
  chat-eval:
    routes:
      - target: openai/gpt-4o
        weight: 50
      - target: anthropic/claude-sonnet-4-5
        weight: 50
    eval:
      enabled: true
      mode: shadow        # second response generated async, not served
      queue: main-eval

Once requests flow into the eval queue, Workspace surfaces them as comparison tasks. Your team sees two anonymized responses — Model A and Model B — and selects the better one. The interface hides which model produced which response to eliminate brand bias.

Choosing your judges

Human evaluation is the gold standard but doesn’t scale. For high-volume queues, LLM-as-judge is practical: configure a judge model in Workspace → Evals → Judge Configuration. The judge receives a structured prompt containing the original user query, both responses, and a rubric. It returns a winner and a one-sentence rationale.

Rubrics matter. A generic “which is better?” prompt produces inconsistent results. Task-specific rubrics — “which response is more accurate given the attached policy document?”, “which response has fewer factual errors?” — produce consistent, auditable judgments that your team can validate by sampling.

Blend human and LLM judging: route novel prompt types to human reviewers, use LLM judging for prompt patterns your team has already validated at high volume.

Calibrating your LLM judge

Before running at scale, validate the judge against a set of comparisons where you already know the preferred outcome. If you have 50 human-judged pairs, run the LLM judge on the same pairs and measure agreement rate. Agreement below 75% indicates the rubric needs revision. Agreement above 90% suggests the judge is reliable enough to run autonomously on that task type.

Document the calibration results and attach them to the rubric in Workspace. When you update the rubric, re-run calibration. This creates an audit trail showing that your eval methodology is validated — important if you ever need to justify a routing decision to a compliance reviewer or executive stakeholder.

Reading the ELO rankings

ManyLayers Workspace updates ELO scores after each judged comparison using the standard Elo update formula with K=32. Scores start at 1000 and diverge as evidence accumulates. A 100-point ELO difference corresponds to roughly a 64% win rate for the higher-ranked model.

After 200–300 comparisons, rankings stabilize. Before that, treat them as directional. Display them in Workspace → Evals → Leaderboard alongside the raw win/loss counts so your team can see whether a ranking is based on 20 comparisons or 2,000.

Watch for two patterns that indicate a ranking is not yet trustworthy: models with fewer than 50 comparisons (not enough evidence) and models whose rating has moved more than 50 points in the last 20 comparisons (still converging). Both are flagged visually in the Leaderboard interface.

Closing the loop to routing

Once your ELO ranking is stable enough to trust, close the loop to routing. A model with a 150-point ELO advantage over alternatives is a strong candidate for the production alias. Promote it via a canary rollout — set it to 20% of traffic on the production alias while running evals in parallel, then ramp up as quality holds.

If a new model release arrives, add it to the eval split at 10% weight and run comparisons before any production promotion. This makes model upgrades a measured, reversible decision rather than a judgment call — which is what your compliance team and your engineering team both need.

Practical notes

• Eval fatigue: keep human review sessions under 20 comparisons. Beyond that, attention drops and judgment quality falls.
• Stratify by prompt type: run separate ELO rankings for different task types (summarization, code, classification). A model that wins at summarization may lose at code generation.
• Export: ManyLayers Workspace exports eval results as structured JSON or CSV. Ingest into your data warehouse and track ranking drift over time as models are updated by providers.
• Audit: every judgment — human or LLM — is logged with timestamp, judge identity, rubric version, and outcome. This audit trail is essential if you ever need to explain a routing decision to a compliance reviewer.
• Minimum viable eval cadence: run at least 50 new comparisons per week per model pair you’re actively evaluating. Letting the queue stall means your rankings go stale and stop reflecting current model behavior.