Tools

Tools

Calculators for the arithmetic in LLM system design. Use them to check your back-of-the-envelope work after you've done it by hand, not instead of doing it by hand. In the interview you won't have them.

• KV cache and serving calculator

  Pick a model, a GPU, a context length and a concurrency. See whether it fits, the decode speed, and the cost per million tokens.

Planned: a RAG latency budget planner and an API cost and routing calculator.