Evidence at a glance
From an Inference Kernel to a Trainable Path
Prime Intellect released Prime Mega MoE, a training extension of the Mega MoE fused kernel in DeepSeek’s open-source DeepGEMM library. DeepSeek’s implementation had a forward path only; Prime Intellect added backpropagation so the kernel can be used for training, not just inference. The target is not a new model, but the sequence of dispatching tokens to experts, computing on them, and collecting the results in a mixture-of-experts model.
The release is worth examining not because “3.11x” applies to total training time, but because it addresses a familiar source of MoE idling: GPUs can wait for tokens to travel to experts on other devices. Speeding up the forward pass alone does not remove the waits in the backward path. To benefit training, both directions matter. Prime Mega MoE’s central claim is that it brings overlapping execution into the full training process.
Fusion Changes Where the Waiting Happens
A conventional MoE execution typically involves dispatching tokens, computing on experts, and combining the results. During dispatch, GPUs sort tokens and wait for transport; after computation, results must be sent back. When those stages run serially, gaps open between communication and computation. DeepSeek’s Mega MoE puts expert-parallel dispatch, two expert linear layers, SwiGLU, and result combination into one fused call. As one token group travels, the kernel computes on another group that has already arrived.
Fusion here is more than reducing the number of calls. The implementation also uses GPU-side scratch buffers to avoid some intermediate tensors, while Prime Intellect’s new backward kernel applies the same overlapping-execution idea. Together, these choices aim to reduce waiting and intermediate data movement, rather than change MoE routing. The payoff therefore depends on whether computation can cover communication, and whether the model’s expert workload and token distribution suit this execution path.
End-to-End Gains Are Smaller Than the Layer Peak
Prime Intellect’s full-step measurements are more useful for judging practical gains than the single-layer peak. The tests used 8×H200 nodes, FP8, 16,384-token sequences, and full activation checkpointing, with results averaged over steps 6 to 20. On 32 H200s, Qwen3-30B-A3B took 2.08 seconds with Prime Mega MoE, versus 2.78 seconds for the prime-rl baseline and 3.07 seconds for TorchTitan. On 128 H200s, Qwen3-235B-A22B took 9.65 seconds, compared with 12.72 and 14.98 seconds respectively. By the publisher’s comparisons, full steps were about 32% to 34% faster than prime-rl and 48% to 55% faster than TorchTitan.
The single-layer numbers are larger, but they should not be confused with training-step speedups. In a one-node, 8×H200 test with intra-node EP=8, the Qwen3-30B-A3B forward pass fell from 12.0 milliseconds to 3.9 milliseconds, or 3.11x. For Qwen3-235B-A22B, the backward pass fell from 30.5 milliseconds to 24.6 milliseconds, a 1.24x gain. Forward gains are substantial, while backward gains are smaller; the new kernel does not accelerate every stage equally. A full training step includes other costs, so the step-level results are the appropriate measure—not the peak forward-pass figure.
The Performance Numbers Have Clear Boundaries
The published tests cover Qwen3-30B-A3B and Qwen3-235B-A22B, with 128 experts and top-8 routing; the reported expert parallelism is intra-node EP=8. The results are self-reported by Prime Intellect and do not establish that cross-node communication, other model structures, or different GPU clusters will see the same gains. In particular, the value of communication overlap varies with topology and workload. A kernel benchmark alone cannot predict training throughput after deployment.
The compatibility requirements also affect migration. The kernel requires gated SwiGLU without expert biases, a hidden size divisible by 256, and an expert FFN size divisible by 128. Hopper supports BF16 and blockwise FP8, while Blackwell supports BF16 and MXFP8. In prime-rl, the dispatch type can be set to `mega_moe` and a precision mode such as `fp8` selected; the configuration also offers an option to reserve 16 SMs for NCCL. For technical leads, the sensible decision is not to copy the published speedup, but to check model structure, precision path, and communication layout, then verify throughput and stability with their own training-step benchmarks.