A framework for LLM system design interviews

Classic system design interviews have a well-worn shape: requirements, API, data model, high-level design, deep dives. That shape still works for LLM systems, but it misses the three things that make them different:

1. The expensive resource is tokens, not servers. The bottleneck is almost always GPU tokens per second or a provider’s rate limit, and the cost is per token. If you never write a token count on the board, you haven’t designed an LLM system.
2. Correctness is statistical. The same input can produce a different output, and “is it right?” needs its own measurement system. Evaluation is part of the architecture, not a test phase.
3. The model can be manipulated by its inputs. Retrieved documents, tool outputs and user text are all attack surface. Safety is a plane of the design.

The framework below adds those three concerns to the classic shape. Every design on this site follows it, step by step.

The seven steps

1. Clarify, then commit to numbers

Ask the questions that change the architecture, then write your assumptions on the board and move on. You are not expected to know the real numbers. You are expected to state them.

• Users and load. Who uses it and how often? Convert to average and peak QPS. Peak is usually 3–10× average.
• Tokens per request. Input and output, separately. They are priced and timed differently.
• Latency target. Interactive or asynchronous? For interactive, give a time-to-first-token target and a total, at p50 and p95.
• Quality bar. What does a wrong answer cost? A wrong movie recommendation and a wrong medication answer need different designs.
• Data. How much, how fresh, how sensitive, and is it multi-tenant?
• Actions. Read-only, or does the system change things in the world? Actions bring approvals, idempotency and audit with them.

A useful sentence: “Let me state my assumptions, and correct me if any are off.”

2. Define success before architecture

Name three to five metrics before drawing a single box:

• the task metric (answer accepted, ticket resolved, code suggestion kept)
• a quality guardrail (groundedness, hallucination rate)
• p95 latency
• cost per successful task, not cost per request
• a safety metric that must never regress

This takes two minutes and is the single clearest signal that you have operated an LLM system rather than built a demo.

3. Back-of-the-envelope

Turn the step 1 assumptions into the numbers that size the system:

• QPS × tokens per request → tokens per second → dollars per day (API) or GPUs (self-hosted)
• corpus size → chunks → vector index size → does it fit on one node?
• context length × concurrency → KV cache memory, if you serve your own model

Show the arithmetic. Round aggressively. The numbers page has the constants, and the KV cache calculator shows the serving math.

4. The happy path

Draw the request path left to right, from client to response. For most LLM systems it looks like this:

client → gateway (auth, rate limit) → input checks → retrieval / context assembly
       → model call(s) or agent loop → output checks → streamed response
                         ↘ traces → evaluation and monitoring

Then, if they exist, draw the offline paths separately: ingestion and indexing, batch jobs, evaluation runs.

5. Deep dives on the planes that matter

An LLM system has five planes. You won’t have time for all of them, so say which two or three matter most for this problem, and why:

Plane	What you design
Data	Ingestion, parsing, chunking, embeddings and their versioning, index type, access-control metadata, freshness, deletion
Model	Which model per step, routing from cheap to expensive, provider failover, caching, structured output, token budgets
Orchestration	Single call vs. chain vs. agent loop, tool schemas, retries, idempotency, step and cost limits, human approval
Evaluation	Golden datasets, LLM-as-judge and its calibration, regression gates in CI, online sampling, A/B tests
Safety	Prompt injection (including through retrieved content), PII, tool permissions, output filtering, abstaining, audit

Choosing the planes, and saying why, is itself what gets scored at Staff level.

6. Failure, scale and cost

• Failure. For each external dependency, say what happens when it is slow or down. The model provider rate-limits you. The vector store times out. A tool call half-succeeds.
• Scale. What breaks first at 10× load? Name the component and the fix.
• Cost. Write a three-line table: the naive design, the optimised design, and the trade-off you accepted to get there.

7. Critique your own design

Close with: “The two weakest parts of this design are X and Y. With another week, I would…”

Interviewers at Staff and Principal level are looking for calibration: knowing where your design is fragile. Ninety seconds of honest critique says more about seniority than ten more boxes.

A time plan for 45 minutes

Minutes	Step
0–6	1. Clarify and commit to numbers
6–8	2. Success metrics
8–12	3. Back-of-the-envelope
12–20	4. Happy path
20–34	5. Deep dives (two or three planes)
34–41	6. Failure, scale and cost
41–45	7. Critique, and questions

Say the plan out loud at the start. At Staff level you are expected to run the clock, not the interviewer.

What changes with seniority

Common mistakes

• Architecture in minute two. Skipping step 1 means every later choice is unjustified.
• Naming products instead of properties. Say “I need approximate nearest-neighbour search with metadata filtering at 20 ms p95”, then name a candidate.
• Multi-agent by reflex. Several agents cost more, are slower and harder to debug. Justify it or use one.
• No evaluation plane. If you can’t say how you would know the system got worse, the design isn’t finished.
• Happy path only. Keep seven minutes for failure and cost.

The designs on this site each work through these steps in full. Start with enterprise RAG under a sub-second latency budget.