Eight compounding reductions, none of them magic.

Mixture of experts

Route each token to a small subset of expert feed-forward blocks, so parameter count and per-token compute decouple.

Parameter footprint scales with the number of experts; FLOPs per token scale only with the number activated. DeepSeek-V3 activates 37B of 671B parameters and spends 250 GFLOPS per token against dense LLaMA-3.1-405B's 2,448, a 9.8x reduction, while scoring higher on most benchmarks. The sparsity ratio has climbed roughly 16x in three years, from 4 (Mixtral) to 64 (DeepSeek-V4-Pro). Baseline for this factor is a dense model of equal or better benchmark quality on the same hardware. Two caveats bound the factor from above: sparsity does not reduce memory, since a 671B model still needs 671B parameters resident, and at small batch a mixture of experts reads k/N of the weights to do k/N of the work, so its arithmetic intensity matches dense while it additionally pays routing and all-to-all cost. The gain is entirely a large-batch phenomenon.

Speculative decoding and multi-token prediction

A cheap drafter proposes several tokens and the target model verifies them all in one forward pass, converting idle FLOPs into throughput.

Decode reads the whole weight matrix from HBM to do a trivial amount of arithmetic, so verifying eight tokens costs almost exactly what verifying one costs. Rejection sampling makes the output distribution provably identical to plain sampling, so this is not a quality trade. The governing condition is that acceptance rate must exceed the drafter's relative cost; below that, speculation makes the system slower, which is what happens to EAGLE-1 at batch 24 and above. Baseline for this factor is the same model and hardware with speculation disabled at production concurrency, not at batch 1. Paper figures of 4x to 6.5x are single best cells at batch 1 against unoptimized baselines and are not used here.

FP8 and NVFP4 / MXFP4 low-precision serving

Store and compute weights at four to eight bits instead of sixteen, cutting the bytes that decode is bound by and the FLOPs that prefill is bound by.

FP8 is settled and effectively lossless: across the Llama-3.1 family W8A8-FP recovers 99.3% to 100.1% of BF16 on Open LLM Leaderboard averages. FP4 is the live frontier and is format-sensitive. NVFP4's 16-element blocks with FP8 scales beat the open MXFP4 standard's 32-element blocks with power-of-two scales by roughly 10 percentage points of relative recovery before format-specialized methods close the gap. Baseline for this factor is FP8 weights on the same silicon, which is what production actually serves today, so the factor isolates the precision step from the hardware step. It therefore assumes Blackwell-class FP4 tensor cores. Nobody quantizes everything: the universal production recipe is FP4 for the large mixture-of-experts and feed-forward GEMMs, FP8 for attention and KV cache, and BF16 for a handful of sensitive layers.

Attention architecture and KV cache management

Shrink the bytes each sequence occupies in HBM, because KV cache size is what caps batch size and batch size is the denominator of cost per token.

Total HBM is weights plus activations plus KV cache. Weights are fixed, so the KV cache sets the maximum concurrency, and cost per token is inversely proportional to achievable concurrency. Multi-head latent attention gets DeepSeek-V3 to 70.3 KB per token against LLaMA-3.1-405B's 516 KB, though normalising per layer the honest architectural gap is 3.6x rather than 7.3x. Baseline for this factor is a modern serving stack that already has PagedAttention and grouped-query attention, both of which are universal in 2026, so the 8x MHA-to-GQA reduction and PagedAttention's 2-4x against 2023 baselines are already spent and are not counted here. The marginal levers that remain are FP8 or NVFP4 KV cache and latent-attention-class compression. FlashAttention makes attention compute fast but does not reduce KV cache size, so its wins land in training and prefill rather than in decode cost.

Continuous batching, prefix caching, and CPU overhead removal

Schedule at the iteration rather than the request, reuse the KV of any prefix already computed, and stop the CPU from being the bottleneck.

Continuous batching was the largest single historical win in this field, up to 23x over static batching, but it is table stakes and is inside the baseline rather than counted here. Baseline for this factor is stock vLLM with continuous batching, PagedAttention and chunked prefill already on. What remains is prefix caching, cache-aware routing, CUDA graphs and scheduler overhead removal. Prefix caching is the highest-leverage item and its value is a property of the traffic, not of the system: measured hit rates run 1.7% on code generation, 78% on chat, and near 100% on classification. DeepSeek's production 56.3% hit rate was worth 36% of their total serving cost. Cache investment should be sized to the input:output ratio, since at r = 0.125 even a 90% hit rate is worth only 4.7%.

Prefill/decode disaggregation and goodput-aware scheduling

Run the compute-bound prefill phase and the bandwidth-bound decode phase in separate GPU pools, each with its own parallelism strategy and SLO.

Prefill wants FLOPs and decode wants bytes per second, so co-locating them means each interferes with the other and neither gets its preferred parallelism. Naive hybrid batching inflates time-between-tokens by up to 28.3x. Baseline for this factor is a co-located deployment with chunked prefill already enabled, which is the honest comparison since chunked prefill is the alternative solution to the same problem and is already in the serving baseline. This factor is expressed on goodput, meaning requests per second that meet both a time-to-first-token and a time-per-output-token target, not raw throughput; a system with 10 rps of throughput can have 3 rps of goodput. Disaggregation is not always a win: under tight-TTFT and loose-TPOT workloads one 2025 measurement puts aggregation ahead at 97% against 42% SLO attainment, which is why the upper end of the range is 1.00.

Model routing, cascades, and semantic caching

Send each request to the cheapest model that will answer it correctly, escalating only when a scorer says the cheap answer is not good enough.

Two very different families of numbers exist here and conflating them is the main credibility risk. Single-benchmark headlines (85% on MT-Bench, 98% on HEADLINES) come from settings where the weak model was already close to the strong one: on MT-Bench, Mixtral-8x7B alone already scores 89% of GPT-4, so the router closes 6 of the 11 points of gap. Baseline for this factor is serving every request on the single best model, and the factor used is the broad multi-domain number rather than any headline. The documented failure mode is routing collapse: as budgets rise, deployed routers converge on the strongest model, sending roughly 100% of queries to GPT-4 where an oracle would use it for under 20%, and prior routers reach GPT-4-level performance only at over 93% of GPT-4's cost. Semantic caching is a separate mechanism with a real correctness cost: at a 0.8 similarity threshold, 2.7% to 7.5% of cache hits returned a wrong answer.

Capacity sourcing across tiers and terms

Buy the same silicon at the clearing price rather than the posted price, and match contract term to the shape of the load.

The identity is cost = 277.78 x ($/hr) / (tokens/sec/GPU), so price and throughput are separate multiplicative terms and this is the one factor in the stack that is close to genuinely independent of the others. Identical H100 silicon rents from $0.45 to $12.29 per GPU-hour, a 27x raw spread and 6.2x on non-interruptible capacity, while the all-in cost to produce that hour is roughly $1.45. The spread persists because hyperscaler on-demand is an administratively-set posted ceiling rather than a clearing price, and because resale is contractually prohibited at most venues, so the classic arbitrage is not executable where the spread is widest. Baseline for this factor is a neocloud on-demand rate, which is where the unit economics cases in fleet.ts are already priced. From that starting point the available lever is tier and term, roughly 1.2x to 2.7x, not the full hyperscaler-to-marketplace spread. Utilization is a larger lever still and is deliberately excluded here to avoid double-counting it against the serving and scheduling factors.