LLM System Design
About
About
Sunil Pandey, Staff Software Engineer at ServiceNow, working on AI platforms for multi-agent orchestration, retrieval and LLM evaluation. Why I write LLM System Design.
I’m Sunil Pandey, a Staff Software Engineer at ServiceNow, with about twelve years of experience building production software.
What I work on
I work on enterprise AI platforms: multi-agent orchestration, retrieval-augmented generation, and the evaluation and observability that keep LLM features reliable in production. The themes that shape this site most:
- Agents that take real actions, where orchestration, safety and human approval matter as much as the model.
- Retrieval under a latency budget, where answers have to arrive inside a live workflow.
- Working across model providers, behind one platform.
- Evaluation as infrastructure: LLM judges, regression gates and production monitoring.
- Architecture standards for AI components, and mentoring the engineers who build them.
Before AI platforms
Before ServiceNow I spent several years building web, mobile and backend systems at Deloitte Consulting, SSA Infosystem and Infosys, working with:
- Web: React, Angular, AngularJS, TypeScript, JavaScript
- Mobile: React Native, offline-first architecture, encrypted on-device storage
- Backend: Java, Spring, Node.js, Express, REST and SOAP services
- Data: DB2, Oracle, MongoDB, SQLite, MariaDB
- Performance: query optimisation and multithreaded batch processing
Why this site exists
Most LLM interview material online falls into one of two kinds: long lists of questions with one-paragraph answers, or architecture diagrams with no numbers on them. Neither prepares you for a Staff-level round, where you’re expected to drive the conversation, quantify every decision, and say where your own design will break.
I wanted the resource I’d have liked to have: a few systems, worked through properly, with the arithmetic shown and the trade-offs argued. The designs are generalised interview problems. Nothing here describes any employer’s internal systems, and no confidential detail is used.
Also by me
- LLM Inference Math for Interviews: 52 worked back-of-the-envelope problems on GPU memory, KV cache, latency, throughput and cost.
Get in touch
I’m happy to hear corrections, disagreements and requests for designs. I fix mistakes and date the change. The quickest way to reach me is LinkedIn or X.
The views on this site are my own and not those of my employer.