Design enterprise RAG with a sub-second latency budget

The prompt: “Design an AI assistant that suggests answers to support agents while they work a ticket, using the company's knowledge base and past tickets. It must feel instant.”

This is the most common LLM design question, and the one where the gap between a Senior and a Staff answer is widest. Almost everyone can draw “embed, retrieve, generate”. The interview is decided by the latency budget, access control, freshness and evaluation, in that order.

1. Clarify, then commit to numbers

Questions worth asking, and the answers I’ll assume:

2. Success metrics

• Suggestion acceptance: the share of generated answers the agent inserts or uses. This is the task metric.
• Retrieval recall@5 on a labelled set: is the right article in the top five?
• Groundedness: the share of answer sentences supported by a cited chunk.
• TTFT p95 ≤ 1 s, and passive results ≤ 300 ms.
• Cost per accepted answer, not per request.
• Zero cross-permission leaks. This is a hard invariant, not a rate.

3. Back-of-the-envelope

Tokens per generated answer. System instructions ~400, ticket context and question ~600, five retrieved chunks at ~400 each = 2,000. So ~3,000 input tokens and ~250 output tokens.

Index size. 40M chunks × 1,024 dimensions × 2 bytes (FP16) ≈ 82 GB of vectors, plus graph overhead. Too big for one comfortable node once you add replicas, but small enough that sharding by tenant is straightforward.

Cost. At an illustrative $3 / $15 per million input/output tokens for a frontier model:

• per generated answer: 3,000 × $3/M + 250 × $15/M ≈ $0.0128
• generating for all 5M requests: ~$64K a day, about $1.9M a month

That number decides the design. Section 6 brings it down by roughly 8×.

4. The happy path

1. The gateway authenticates the agent and resolves their tenant and permission groups once. Everything downstream receives them; nothing downstream re-derives them.
2. The input check (injection and abuse classifier) runs in parallel with query embedding, not before it.
3. Hybrid retrieval: vector search and keyword search run in parallel inside the tenant’s index, both filtered by the agent’s permission groups. Results are merged with reciprocal rank fusion.
4. A cross-encoder reranks the top ~40 down to 5.
5. For a passive trigger, stop here and return the articles. No model call.
6. For an active question, the router picks a model, the prompt is assembled with the 5 chunks and their IDs, and the answer streams back with citations.

The latency budget

Write this table on the board. It is the centre of the interview.

Two consequences come straight out of the table:

• There is no room for an LLM query-rewrite step. Rewriting the question with a model costs 200–400 ms. Instead, embed the ticket’s structured fields alongside the question, and only rewrite for multi-turn follow-ups, with a small fast model.
• The model’s first-token time is the budget. So the prompt should be as short as it can be (five chunks, not twenty), and a model whose p99 tail is unpredictable needs a fallback, covered in section 6.

5. Deep dives

The three planes that matter most here are data (access control and freshness), retrieval quality, and evaluation. Orchestration is trivial: it’s a single model call.

Access control and tenancy

Getting this wrong is the one failure that ends the project, so design it first.

• Isolation by tenant. Each large tenant gets its own index namespace or shard. Small tenants share a pooled index partitioned by tenant ID. A query can only ever reach one tenant’s data, and that is enforced in the retrieval service, not in the prompt.
• Permissions inside a tenant. Each chunk stores the permission groups of its source document. The query carries the agent’s groups.
• Filter during search, not after. If you retrieve the top 40 and then drop the ones the agent can’t see, a restricted agent can get zero results. Use the index’s filtered search so permissions are applied while traversing the index.
• Caches carry permissions too. A cache key must include the tenant and a hash of the permission groups. Never share a semantic cache entry between users with different access.
• Permission changes. When a document’s permissions change, its chunks must be updated through the same change pipeline as content edits, with the same 5-minute target.

Retrieval quality

• Why hybrid. Support content is full of exact tokens: error codes, product names, article IDs. Dense vectors blur these; keyword search nails them. Dense search handles paraphrase. You need both.
• Chunking. Split articles by section, not by fixed length, and prefix each chunk with the article title and section heading so a chunk makes sense on its own. Resolved tickets are chunked as “problem + resolution” pairs.
• Reranking is worth its 70 ms. A cross-encoder reads the query and passage together and reliably beats raw vector similarity at choosing the final five. It is the best quality per millisecond in the pipeline.
• Abstain. If the best reranked score is below a threshold tuned on the evaluation set, don’t generate. Show the top articles and say there’s no confident answer. A wrong answer delivered confidently costs more trust than no answer.

Freshness and re-indexing

• Change capture. Publishing an article or resolving a ticket emits an event. A queue feeds chunking and embedding workers, which upsert into the index. Five minutes is easy at this volume; the hard parts are deletes and permission changes, which must also flow through.
• Changing the embedding model. You can’t mix vectors from two embedding models in one index. Build a new index alongside the old one, backfill it, compare retrieval quality on the evaluation set, then switch reads over per tenant. Plan for this from day one; it happens every year or so.

Evaluation

There is a gift in this domain: natural labels. When agents resolve a ticket, they often link the knowledge article that solved it. Those (ticket, article) pairs are a ready-made retrieval test set, per tenant, refreshed every day.

• Offline: recall@5 and MRR on that set for every change to chunking, embeddings or reranking. Groundedness and correctness of generated answers via an LLM judge, calibrated against a few hundred human-graded examples.
• Online: acceptance rate, edit distance between suggested and sent text, and how often agents open a cited article.
• Gate: no change to the retrieval pipeline ships if recall@5 drops on the held-out set.

6. Failure, scale and cost

Failure modes

What breaks at 10×

At 3,000 QPS peak, the first thing to break is provider rate limits and spend, not your own services. The fix is the cost work below plus negotiated capacity, or self-hosting the small model. Retrieval scales horizontally by adding replicas per shard.

The cost table

The small model is priced at an illustrative $0.25 / $1.25 per million tokens, about $0.001 per answer. The routing rule is learnable: send short, well-covered questions (high rerank score, a single obviously relevant article) to the small model, and escalate long, ambiguous or low-confidence ones. Validate it with the same evaluation set. The accepted trade-off: a small quality drop on the easy 75%, which you must measure, in exchange for an 8× cost reduction.

7. Critique

The two weakest parts of this design:

1. The judge-based groundedness metric. It is only as good as its calibration, and it drifts when the underlying model changes. I’d budget ongoing human grading, a few hundred examples a month, to keep it honest.
2. Hedged requests double cost in the tail. They’re capped, but under a provider incident the cap is what stands between you and a spending spike. I’d want a circuit breaker that turns hedging off when the provider’s error rate crosses a threshold.

With more time I’d look at per-tenant fine-tuned rerankers for the largest customers, and caching generated answers for frequent questions within a tenant and permission group.

What a strong answer looks like at each level

Follow-up questions to practise

• A tenant with 30M documents joins. What changes?
• Legal asks you to guarantee a deleted document stops appearing within one hour, everywhere. How?
• The agent can now ask the assistant to take an action, like closing the ticket. What does that change? (See the multi-agent service desk design.)
• How would you detect that the knowledge base itself is out of date?