Start here

This site is for engineers preparing for interviews where they will be asked to design a system around a large language model: AI engineer, ML engineer, applied AI, forward deployed engineer, and Staff or Principal platform roles that now include an LLM round.

It is not a list of 200 questions and answers. It is a small number of designs, worked through properly, so you can learn a method you can apply to a problem you haven’t seen.

What’s here

• The framework. Seven steps for any LLM design, with a 45-minute time plan. Read this first.
• System designs. Full worked designs. Each one states its assumptions, does the cost and latency math, draws the architecture, goes deep on the parts that matter, and ends with what separates a Senior answer from a Staff one.
• Problem solving. The practical coding rounds AI teams now run instead of puzzles: caches, rate limiters, attention and the KV cache from scratch. Python, with tests.
• Numbers. The constants you should be able to quote without looking them up.
• Tools. Calculators for the arithmetic, so you can check your back-of-the-envelope work.

A two-week plan

Days 1–2. Read the framework and the numbers. Then do the three numbers exercises at the bottom of this page on paper.

Days 3–6. One design per day. Read the problem statement only, close the page, and design it yourself for 45 minutes, out loud, with a timer. Then read the worked design and write down what you missed. Your misses are your study list.

Days 7–10. One coding problem per day, timed at 45 minutes, with tests. Then do the follow-ups at the end of each problem, because that is where interviews are decided.

Days 11–14. Redo the two designs you did worst on, without notes. The second attempt is where the framework becomes automatic.

Three exercises to start with

1. A product has 200,000 daily users who each ask 8 questions a day. Each question uses 3,000 input tokens and 300 output tokens on a model priced at $3 and $15 per million. What does it cost per month?
2. How much memory does the KV cache of Llama 3.1 70B need for 32 concurrent conversations at 16K tokens of context, in BF16?
3. You have 4 million documents averaging 2,500 tokens. You chunk at 500 tokens and embed at 1,024 dimensions in FP16. How big is the vector index, and does it fit on one machine?

Answers

1. 1.6M questions a day. Input: 1.6M × 3,000 = 4.8B tokens = $14,400. Output: 1.6M × 300 = 480M tokens = $7,200. About $21,600 a day, or roughly $650,000 a month, which is why the designs here spend time on routing and caching.
2. 320 KB per token × 16,384 tokens × 32 sequences ≈ 172 GB of KV cache, more than the 141 GB of weights. Context, not model size, is often what fills the GPU.
3. 20 million chunks × 1,024 × 2 bytes ≈ 41 GB plus graph overhead. It fits in memory on one large node, but you’d shard or replicate for availability and headroom.

Who writes this

I’m Sunil Pandey, a Staff Software Engineer at ServiceNow. I work on enterprise AI platforms for multi-agent orchestration, retrieval and LLM evaluation. Every page here shows its assumptions and arithmetic, so you can check the reasoning instead of taking it on trust.