Mixture-of-Experts is routinely sold as doing more with less: activate 37B of 671B parameters per token and pay for a 37B model. For training FLOPS that framing is roughly right. For serving memory at long context it is wrong in the way that causes OOMs at 2 a.m. The experts are not the wall. The KV cache is.
The arithmetic nobody puts on the slide
KV cache per token is set by depth, width, and precision — not by expert count:
bytes_per_token = 2 (K and V) × n_layers × d_model × bytes_per_weight
Take a public 70B-class dense shape (80 layers, 8192 width, fp16) as a worked example: 2 × 80 × 8192 × 2 bytes ≈ 2.5 MB per token. At 128k context that is ~320 GB of KV cache — before weights, before activations, before batching. Four H100s of HBM just for one request's memory of the conversation.
Now make it MoE. Routing tokens to 8 of 256 experts changes none of the four terms above. KV cache scales with layers × width, and MoE models are typically deeper than same-quality dense models. The sparsity dividend applies to matmuls; the memory tax applies to every token you have ever seen in the session.
The counterweight is real — price it, don't assume it
Production MoE serving stacks blunt this three ways, and each has a cost:
| Technique | What it saves | What it costs |
|---|---|---|
| Grouped-query attention (GQA) | KV heads ÷ 4–8 (public shapes) | Small quality delta to eval |
| Multi-head latent attention (MLA) | Order-of-magnitude KV compression | Custom kernels, harder portability |
| Prefix caching / disaggregated prefill | Recompute vs HBM tradeoff | Extra orchestration, tail latency |
The honest capacity plan prices all three against your p99 context, not the median. A fleet sized for 8k medians OOMs on the first 1M-context enterprise pilot — and the overage GPUs are bought at emergency prices.
Worked sizing rule
For each candidate model, compute KV bytes/token from its published config, multiply by (p99 context × target concurrency), and divide into per-GPU HBM (vendor-published: 80GB H100, 141GB H200, 192GB B200). That quotient is your memory GPU count. Compute your FLOPS GPU count separately. Buy the max of the two. In our experience reviewing serving postmortems, teams that skip the first calculation discover it in production.
Decision impact
Before signing any inference capacity: demand the vendor's KV-cache accounting at your p99 context and concurrency, in writing. If the quote only shows tokens/sec at 2k context, it is a benchmarketing number. Model the power and capex envelope yourself with the cluster modeler first.