LLM System Design
Reference
Numbers every LLM engineer should know
Tokens, latencies, GPU memory, KV cache sizes and prices, rounded so you can quote them from memory in a system design interview and do the arithmetic live.
You don’t need exact figures in a design interview. You need figures that are right to within a factor of two, so the arithmetic you do on the board leads to the right decision. These are the ones I use.
Hardware and model figures are stable. Prices and API latencies change every few months, so treat those rows as orders of magnitude and say “check current pricing” out loud.
Text and tokens
| Quantity | Working figure |
|---|---|
| 1 token (English) | ≈ 4 characters ≈ 0.75 words |
| 1 page of prose | ≈ 500–700 tokens |
| A typical RAG prompt | 2K–8K input tokens |
| A typical chat answer | 200–800 output tokens |
| Code | more tokens per character than prose; budget 1.3–1.5× |
Traffic
| Quantity | Working figure |
|---|---|
| Seconds per day | ≈ 86,400 ≈ 10⁵ (use 10⁵ for mental math) |
| 1M requests per day | ≈ 12 QPS average |
| Peak to average | 3–10× (business-hours products sit at the high end) |
| 100K daily users × 10 requests | ≈ 12 QPS average, 40–100 QPS peak |
Latency of the pieces
| Step | Typical |
|---|---|
| Self-hosted embedding of a short query | 5–20 ms |
| Embedding via a hosted API | 30–150 ms (network dominates) |
| Approximate nearest-neighbour search, 1M–10M vectors in memory | 1–10 ms |
| Same, through a managed vector database over the network | 10–40 ms |
| BM25 keyword search | 5–30 ms |
| Cross-encoder rerank of 50 passages on a GPU | 30–100 ms |
| Time to first token, hosted frontier model, few-K-token prompt | 0.3–1 s, with a long p99 tail |
| Output speed, hosted frontier model | 30–150 tokens/s per stream |
| Small model classifier or guardrail | 10–50 ms |
The useful pattern: retrieval is cheap, reranking is moderate, and the model call dominates almost every interactive latency budget. Streaming changes what the user perceives: time to first token matters more than total time.
Vectors
| Quantity | Working figure |
|---|---|
| Embedding size | 768–3,072 dimensions |
| One 1,536-dim vector in FP32 | 6 KB (1,536 × 4 bytes) |
| 1M such vectors | ≈ 6 GB, plus ~0.1–0.3 KB per vector of HNSW graph links |
| Same in FP16 / INT8 | ÷2 / ÷4 |
| Typical chunk | 300–800 tokens, 10–20% overlap |
So 10M chunks at 1,024 dimensions in FP16 is about 20 GB: one large-memory node, or a few shards if you need headroom and replicas.
GPUs
Approximate dense figures. Always check the vendor spec sheet for the exact SKU.
| GPU | Memory | Memory bandwidth | BF16 compute |
|---|---|---|---|
| NVIDIA L4 | 24 GB | 0.3 TB/s | ~121 TFLOPS |
| NVIDIA A100 80GB | 80 GB | ~2.0 TB/s | 312 TFLOPS |
| NVIDIA H100 SXM | 80 GB | 3.35 TB/s | ~989 TFLOPS |
| NVIDIA H200 | 141 GB | 4.8 TB/s | ~989 TFLOPS |
Realistic efficiency: about 70% of peak memory bandwidth during decode, and 40–60% of peak compute during prefill.
Model memory
| Bytes per parameter | |
|---|---|
| FP32 | 4 |
| BF16 / FP16 | 2 |
| FP8 / INT8 | 1 |
| INT4 | 0.5 |
Weights = parameters × bytes per parameter. A 70B model in BF16 is about 141 GB, so it needs at least two 80 GB GPUs before you’ve stored a single token of context.
KV cache
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value. The 2 is for keys and values.
| Model (BF16 KV cache) | Layers | KV heads | Per token | One 8K-token sequence |
|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 KB | ≈ 1.1 GB |
| Llama 3.1 70B | 80 | 8 | 320 KB | ≈ 2.7 GB |
| Llama 3.1 405B | 126 | 8 | 504 KB | ≈ 4.2 GB |
These models use grouped-query attention (8 KV heads shared by many query heads). Without it, the 70B model’s KV cache would be 8× larger. Try your own numbers in the KV cache calculator.
Decode speed, from first principles
At small batch sizes, generating each token means reading every weight from GPU memory once. So:
Time per output token ≈ bytes read per step ÷ (memory bandwidth × 0.7)
- Llama 3.1 8B in BF16 on one H100: 16 GB ÷ 2.35 TB/s ≈ 7 ms per token, so about 140 tokens/s for a single stream.
- Llama 3.1 70B in BF16 on two H100s: about 30 ms per token, so about 33 tokens/s per stream, before communication overhead.
Batching many sequences shares that weight read across all of them. That is why throughput rises with batch size while each user’s speed falls slightly.
Prices (illustrative)
| Item | Round figure used on this site |
|---|---|
| H100, on demand | ~$2–4 per GPU-hour (we use $2.50) |
| Frontier API model | ~$3 per 1M input tokens, ~$15 per 1M output tokens |
| Small API model | ~$0.10–0.50 per 1M input tokens |
| Cached input tokens | ~10% of the normal input price |
| Embeddings via API | ~$0.02–0.13 per 1M tokens |
Cost per 1M tokens on your own GPUs = (GPU $/hour × GPUs) ÷ (tokens/s × 3,600) × 10⁶.
Output tokens cost about 5× input tokens on most APIs, and they are also what makes responses slow. Shorter answers are the cheapest optimisation there is.
Most of the hardware and serving figures here come from my book, LLM Inference Math for Interviews, which works through 52 problems like these step by step.