<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>The Raw Logs — Essays</title><link>https://www.therawlogs.com/essays</link><description>Long-form technical analysis from The Raw Logs.</description><language>en</language><lastBuildDate>Tue, 15 Sep 2026 19:58:25 GMT</lastBuildDate><atom:link href="https://www.therawlogs.com/rss/essays.xml" rel="self" type="application/rss+xml"/>    <item><title>The OOM Horizon: memory wall economics in Mixture-of-Experts serving</title><link>https://www.therawlogs.com/essays/oom-horizon-moe-memory</link><guid>https://www.therawlogs.com/essays/oom-horizon-moe-memory</guid><pubDate>Fri, 11 Sep 2026 00:00:00 GMT</pubDate><description><![CDATA[MoE sparsity cuts FLOPS per token, not bytes per token. At long context the KV cache — not the experts — decides how many GPUs you buy.

Mixture-of-Experts is routinely sold as doing more with less: activate 37B of 671B parameters per token and pay for a 37B model. For training FLOPS that framing is roughly right. For **serving memory at long context** it is wrong in the way that causes OOMs at 2 a.m. The experts are not the wall. The KV cache is. ## The arithmetic nobody puts on the slide KV cache per token is set by depth, width, and precision — not by expert count: ``` bytes_per_token = 2 (K and V) × n_layers × d_model × bytes_per_weight ``` Take a public 70B-class dense shape (80 layers, 8192 width, fp16) as a worked example: 2 × 80 × 8192 × 2 bytes ≈ **2.5 MB per token**. At 128k context that is ~320 GB of KV cache — before weights, before activations, before batching. Four H100s of HBM just for one request's memory of the conversation. Now make it MoE. Routing tokens to 8 of 256 experts changes none of the four terms above. KV cache scales with layers × width, and MoE models are typically *deeper* than same-quality dense models. The sparsity dividend applies to matmuls; the memory tax applies to every token you have ever seen in the session. ## The counterweight is real — price it, don't assume it Production MoE serving stacks blunt this three ways, and each has a cost: | Technique | What it saves | What it costs | |---|---|---| | Grouped-query attention (GQA) | KV heads ÷ 4–8 (public shapes) | Small quality delta to eval | | Multi-head latent attention (MLA) | Order-of-magnitude KV compression | Custom kernels, harder portability | | Prefix caching / disaggregated prefill | Recompute vs HBM tradeoff | Extra orchestration, tail latency | The honest capacity plan prices all three against your **p99 context**, not the median. A fleet sized for 8k medians OOMs on the first 1M-context enterprise pilot — and the overage GPUs are bought at emergency prices. ## Worked sizing rule For each candidate model, compute KV bytes/token from its published config, multiply by (p99 context × target concurrency),…

Full text → https://www.therawlogs.com/essays/oom-horizon-moe-memory]]></description></item></channel></rss>