LLM System Design

Tool

KV cache and serving calculator

Will it fit, how fast will it decode, and what will each token cost? Estimates from first principles, using the formulas on the numbers page.

Model
Workload and hardware
KV cache per token
KV cache per sequence
Total KV cache
Weights
Usable GPU memory (90%)
Max concurrent sequences
Decode step (lower bound)
Speed per sequence
Total output throughput
Cost per 1M output tokens

How it's calculated

  • KV per token = 2 × layers × KV heads × head dimension × bytes. The 2 is keys plus values.
  • Weights = parameters × bytes per parameter. Usable memory is 90% of the total, with about 1.5 GB per GPU set aside for runtime overhead.
  • Decode step ≈ (weights + KV cache read) ÷ (GPUs × bandwidth × efficiency). This is the memory-bound lower bound, so real systems are slower. It ignores communication between GPUs and assumes every sequence is at full context, which is the worst case.
  • Cost per 1M tokens = (GPUs × $/hour) ÷ (throughput × 3,600) × 10⁶.
  • For mixture-of-experts models such as Mixtral, all experts must be in memory, but with large batches nearly all of them are read every step. That is why total, not active, parameters are used here.

Prefill (processing the prompt) is compute-bound and is not modelled here. It determines time to first token. See the numbers page.