Evidence at a glance
Lever One: Compile Kernels for the Real Shapes
NVIDIA published a set of MoE inference tuning results on its developer blog. The test subject is Ling-3.0-flash from Ant Group's inclusionAI: a 124B-parameter mixture-of-experts model with 5.1B active parameters per token and a native 256K context. The hardware is not a data-center rack but a single DGX Spark, built on the GB10 Grace Blackwell Superchip with 128 GB of unified memory, 273 GB/s of memory bandwidth, and roughly 1 petaFLOP of sparse FP4 compute, rated for models up to 200B parameters. Batch size is fixed at one throughout.
That configuration needs to be stated explicitly because it defines what every number downstream means. DGX Spark is a desktop system serving single-user, single-request, local inference rather than multi-tenant throughput. Its 128 GB of unified memory is enough to hold 124B FP4 weights, but 273 GB/s sits orders of magnitude below the multi-TB/s HBM of data-center GPUs. So the question stops being how fast the math runs and becomes how many bytes each generated token must read. NVIDIA's profiling gives a blunt answer: decoding is memory-bandwidth bound, and reading model weights dominates execution time.
Two operations stand out: the MoE expert computation and the LM head vocabulary projection. The former activates only a subset of experts per token through routing; the latter reads the entire vocabulary matrix on every generated token. Those two observations map onto two levers — move fewer bytes per decode step, and produce more tokens per verification round. The three optimizations are organized around exactly those levers.
Lever Three: More Tokens per Verification
The LM head is a 157,184 × 2,560 matrix stored in BF16. Every generated token reads all of it. At batch size one with bandwidth as the binding constraint, this is not a negligible constant but a steady per-step tax.
NVIDIA's response is restrained: quantize only this layer to FP8 online at load time, leaving every other weight untouched. Read volume halves outright. It is enabled by setting SGLANG_ENABLE_FP8_LM_HEAD=1, and it is mutually exclusive with the baseline's --enable-fp32-lm-head flag, so the two cannot be enabled together. That detail is worth noting because it implies the path is a branch in the framework rather than an orthogonal switch.
Combined with Humming, synthetic throughput reaches 43.94 tokens/s — 21.7% over baseline and a further 11.4% over Humming alone. Read together, the two gains are clearly additive: one reduces per-step kernel overhead, the other reduces per-step weight reads, and their cost models do not overlap.
Results on Real Traffic: Where 42.5% Comes From
The first two optimizations make each step cheaper; the third changes how many steps there are. Ling-3.0-flash ships with a built-in NEXTN prediction layer, so no separate draft model is needed: the draft proposes three tokens and the target model checks them in a single four-position verification window.
The cost structure has to be spelled out. One verification costs about 2.26× a normal decode step, and the main cost is still MoE rather than attention: verifying four tokens can activate up to 32 experts instead of eight. Whether MTP pays off therefore depends almost entirely on acceptance length per round. On synthetic traffic, an average of 3.90 tokens is accepted per round, pushing throughput to 102.88 tokens/s — a 185.1% gain over baseline. That number looks spectacular, but it comes from the workload class most favorable to speculative decoding.
NVIDIA itself warns against extrapolating synthetic gains to other scenarios, and gives the reasons plainly: synthetic outputs are more repetitive, sampling differs, and output length distributions differ. Acceptance length is the one quantity in this stack that is highly input-distribution dependent, and it is the most easily misread number in the whole write-up.
Stacking Is Not Automatic: The Silent Fallback Trap
To produce a more realistic result, the team ran all 1,319 GSM8K questions with 5-shot prompting and chain-of-thought, still at batch size one. The key change here is acceptance length: it drops from 3.90 on synthetic traffic to 2.68. MTP's contribution is cut down accordingly, but remains positive.
| Metric | Baseline | All three stacked | | --- | --- | --- | | Decode throughput | 37.81 tok/s | 53.89 tok/s (+42.5%) | | Mean end-to-end latency | 15.21 s | 11.49 s (−24.5%) | | TPOT | 26.45 ms | 18.56 ms (−29.8%) | | P50 / P95 latency | 8.36 / 50.20 s | 6.11 / 38.60 s | | GSM8K accuracy | 96.89% | 96.82% |
The row worth reading is not throughput but the last one. Accuracy moves from 96.89% to 96.82%, a difference NVIDIA describes as not statistically detectable. That means FP8 vocabulary quantization stacked with speculative decoding bought no measurable quality loss on this benchmark. P95 latency falling from 50.20 s to 38.60 s shows the gains are not confined to the mean — the tail shortens too, which matters more for interactive use.
One discrepancy is worth flagging: the Chinese page lists the accuracy delta as +0.08, but the two figures are 96.89% and 96.82%, so the direction should be downward, and the English table records the difference as negative. This is a translation or table-formatting inconsistency rather than a substantive one, but when citing, follow the sign in the English table.
What to Take Away, What to Doubt
The three optimizations do not overlap in cost model — Humming and the FP8 LM head cut per-step cost, MTP cuts the number of steps — so in principle they stack. That does not make stacking free. NVIDIA states a specific condition: with both FP8 LM head and MTP enabled, the draft must share the full LM-head module. If it does not, the framework silently falls back and the draft interprets the FP8 weights as BF16.
The consequence is severe: acceptance length collapses to 1.00, meaning every drafted token per round is rejected and decoding degrades to one token at a time while the extra verification cost is still paid. Throughput then drops below the unoptimized baseline. This is not a discount on performance; it is slower than doing nothing. For an engineer who just followed a checklist and turned on three variables, this is the hardest class of problem to diagnose, because the configuration says optimizations are enabled while the result is negative.
NVIDIA's advice is plain but effective: verify at startup that each optimization is actually active, and it points to the Ling cookbook notebook for the correct way to enable both together. That is a general requirement for anyone deploying inference: when optimizations carry implicit module-sharing contracts, an enabled flag is not evidence that an optimization engaged — you need a runtime check.
If you remember only one number, 42.5% is not the most informative. More useful is the division of labor: profile first to confirm the bottleneck is bandwidth, then attack kernel overhead, weight reads, and step count separately, then validate on real traffic and state plainly that synthetic gains do not extrapolate. The gap between +185.1% and +42.5% is not a failure but the normal price of speculative decoding, whose payoff is inherently determined by the input distribution: repetitive content accepts at a high rate, chain-of-thought output that requires careful reasoning accepts at a lower one.
In the broader picture, none of this is new. MoE is a repeatedly validated architectural choice, and speculative decoding has long been standard in mainstream inference stacks. What carries information is how they combine on a desktop system with 273 GB/s of bandwidth, and what debugging cost that combination imposes. For anyone planning to serve large models on a desktop workstation, three judgments are actionable: confirm whether the model ships its own prediction layer, since otherwise MTP requires a separate draft model and the cost structure changes; treat FP8 LM head as a single-layer change that targets the fixed per-token read cost, which pays off clearly at small batch; and add assertions to your startup script so each optimization is proven to engage.
The boundaries deserve stating. Every number comes from a single-user scenario at batch size one, the evaluation set is GSM8K alone, and the claim of no detectable accuracy change holds only within that scope. As batch si