Design an evaluation platform for 50 LLM features

The prompt: “Fifty product teams ship LLM features. Quality regressions are being found by customers. Design an evaluation platform they will all use.”

This question is asked more at platform-heavy companies, and it is a strong one to be good at, because most candidates have never built evaluation as infrastructure. They know “use an LLM as a judge.” They haven’t had to answer how do you know the judge is right, is a 2-point drop real, or who pays for 50 teams’ judging bills.

From production LLM evaluation and observability in production is a large part of my work. This page is a generalised design for the interview, with illustrative numbers, and doesn't describe any employer's internal systems.

1. Clarify, then commit to numbers

Question	Assumption
Who are the users?	50 product teams, each owning one or more LLM features: chat, summarisation, extraction, agents.
How often do they change things?	~20 changes per feature per week (prompts, models, retrieval settings, code), so ~1,000 evaluation runs a week.
Test data?	Each feature has, or will have, a dataset of 300–2,000 cases.
Production volume?	~10M LLM calls a day across all features.
Human labelling?	Limited: a few hundred labels per feature per week, at most.
What does done look like?	No feature ships a change without an evaluation result, and regressions are caught before customers see them.

2. Success metrics

The platform has its own metrics, separate from the features’ metrics:

Coverage: share of features whose releases are gated on evaluation.
Regressions caught before release versus found in production. This is the real outcome.
Judge–human agreement on each criterion that is used to gate.
Time to result in CI: under 15 minutes, or teams will bypass it.
Evaluation cost per feature per month, visible to each team.

3. Back-of-the-envelope

One full evaluation run of a 500-case dataset:

generate 500 outputs from the system under test: ~$6 at ~$0.013 each
judge each on 3 criteria with a frontier model at ~2,000 input / 200 output tokens per judgement: 1,500 × ~$0.009 ≈ $13.50
≈ $20 per run. At 1,000 runs a week, that’s ~$86K a month of evaluation spend.

Online: judging a 1% uniform sample of 10M daily calls is 100K judgements a day, ~$27K a month.

Neither number is outrageous, but both grow linearly with adoption, and the platform’s success depends on adoption. Cost has to be designed in, not managed later.

4. The architecture

The feature adapter is the only team-specific piece. Everything else is shared.

The contract with teams is what makes 50 teams adopt it:

A team provides an adapter: a function or endpoint that takes a test case and returns the feature’s output (and trace).
A team owns its datasets and picks evaluators from the registry, or adds its own.
The platform provides everything else: running, caching, storage, statistics, gates, dashboards and judge calibration.

The components

Datasets. Immutable versions. Each case has an input, optional reference answer or rubric notes, and slice tags (language, customer tier, task type, “adversarial”). Cases come from four sources: written by the team, sampled from production (PII scrubbed), promoted from incidents, and generated adversarially.
Evaluator registry. Three kinds, cheapest first:
1. Deterministic checks: valid JSON, schema, required fields, exact tool-call match, length, banned content. Free and perfectly reliable. Use them for everything they can cover.
2. LLM judges: a pinned (rubric prompt, judge model, parameters) triple with a version number.
3. Human review: for calibration and for criteria no judge passes.
Run orchestrator. Fans out cases with rate limits, uses batch APIs where latency allows, and caches outputs keyed by (case, system version) and judgements keyed by (output hash, evaluator version). Re-running an unchanged case costs nothing.
Results and statistics. Every run is compared to a baseline on the same cases (see below).
CI gate. Per feature: which metrics and slices block a release, which only warn.

5. Deep dives

The planes that matter are evaluation itself (judges and statistics), the platform contract, and cost. Model serving and retrieval aren’t part of this system.

Can you trust the judge?

An uncalibrated LLM judge is a random number generator with good grammar. For every criterion used to gate releases:

Calibrate against humans. Label 200–300 cases by hand, run the judge, and measure agreement. Only gate on criteria where agreement clears the bar you set. Keep the others as warnings.
Prefer pairwise to absolute when comparing two versions. “Is A or B more correct?” is far more stable than “rate this 1–10”. Use absolute scores for monitoring trends.
Control known biases. Position bias: run both orders and count only consistent verdicts. Verbosity bias: tell the rubric that length is not quality, and check it. Self-preference: don’t judge a model’s output with the same model family when that model is one of the candidates.
Pin the judge. A judge is (prompt, model, version). Changing any part creates a new evaluator version, which needs a new baseline. Otherwise a provider’s silent model update looks like your feature regressing.
Re-check weekly against a fixed calibration set to catch drift in the judge itself.

Is the regression real?

With 500 cases, a drop from 86% to 84% can be noise. Showing teams noise as a red X is the fastest way to get the gate ignored.

Paired comparison: baseline and candidate run on the same cases, and the platform reports the difference with a bootstrap confidence interval.
Gate on significance: block only when the interval for a critical metric is clearly below zero, or below a tolerance the team chose.
Report slices. An aggregate that holds steady can hide a 15-point drop on one language. Critical slices gate; the rest inform.
Mind multiple comparisons. Ten metrics × eight slices is eighty chances of a false alarm. Keep the gating set small and deliberate.
Tell teams their dataset size. If a dataset can only detect a 6-point change, say so. That’s a reason to grow it.

Online evaluation

Offline sets go stale. Production is where new failure types appear first.

Sample smartly. A small uniform sample (~0.2%) for unbiased trend lines, plus targeted sampling: thumbs-down, user retries, abandoned sessions, low retrieval scores, new customer segments.
Online judges run on samples asynchronously. They never sit in the request path.
Alert on drift, both in judge scores and in input distribution (new topics, longer prompts).
Close the loop. A human confirms a sampled failure, and it becomes a dataset case. This is how the offline set keeps up with reality.

Privacy

Production samples contain customer data. Scrub PII before anything is stored, limit retention, give each team access only to its own samples, and log who viewed what.

6. Failure, scale and cost

Failure modes

Failure	Behaviour
Judge provider down or rate-limited	Runs queue and retry. Deterministic checks still report. The gate shows “incomplete”, not “pass”.
Judge model silently changed by the provider	Weekly calibration check fails and alerts. Pin model versions where the provider allows.
Flaky gate (noise)	Confidence intervals and slice discipline. Track the gate’s false-alarm rate as a platform metric.
Team bypasses the gate	Visible in coverage metrics. Escalate through the AI governance process, not the platform.
Dataset leaks into training or prompts	Keep held-out sets that teams can’t see in full, and rotate cases.

The cost table

Design	Monthly cost (illustrative)
Naive: full suite, 3 frontier judges, on every change; 1% online sample	~$113K
Small smoke set (~100 cases) per change, full suite nightly; targeted online sampling	~$55K
Plus caching unchanged outputs and judgements, and deterministic checks or a small-model judge for the easy criteria	~$25–30K

The accepted trade-off: a change might pass the smoke set and fail the nightly full run, so a regression can sit on the main branch for up to a day. That’s acceptable when releases come from nightly-validated builds. Make it a rule.

7. Critique

Calibration labels are the bottleneck. Every new criterion and every judge change needs human labels, and labelling is the scarcest resource here. I’d invest early in labelling tooling and in teaching teams to write rubrics that humans agree on: if humans disagree with each other, no judge will be reliable.
Adapters can drift from production. If a team’s adapter calls the feature differently from how production does, you’re evaluating something nobody uses. I’d require that the adapter use the same entry point as production, and compare offline outputs with production traces on the same inputs.

What a strong answer looks like at each level

Level	What they do
Senior	Golden datasets, LLM-as-judge, CI integration, dashboards. A working pipeline.
Staff	Calibrates judges against humans and handles their biases. Treats statistical significance as a product requirement of the gate. Designs online targeted sampling that feeds the offline set. Puts cost into the design with caching and tiered evaluators.
Principal	Designs the contract that makes 50 teams adopt it, and the governance that makes evaluation a launch requirement. Makes judge versions and baselines first-class so model upgrades are measured, not feared. Says what the platform deliberately won’t do.

Follow-up questions to practise

A team’s feature is an agent with ten steps. How do you evaluate a trajectory, not just a final answer?
How would you evaluate a feature for which there is no reference answer at all?
The company wants to move every feature to a new model next quarter. How does this platform make that safe?
A judge and a human disagree on 30% of cases. What do you do?