The Bottleneck in Video DiTs Is No Longer a Single Operator
A video Diffusion Transformer flattens the spatial and temporal tokens of a clip into one long sequence, then runs full self-attention at every layer. A five-second, 720p Wan2.2-14B clip contains roughly 70,000 tokens, and the cost of attention grows quadratically with sequence length. In one MiniMax-H3 test, attention consumed about two thirds of each denoising step on a single B200 running BF16 FlashAttention-4.
Low-bit Tensor Cores can already accelerate the two matrix multiplications inside attention: QK and PV. That does not make the entire attention block low bit, however. The softmax between them still runs in FP32, and on H200 and B200 systems its exponentiation and subsequent FP8 conversion can become the longest pipeline stage. VC-Attention therefore does not start by merely pushing 8-bit computation down to 4 bits. It addresses quantization error and the softmax stage that low-bit matrix hardware leaves untouched.
Value Is the Error Source Earlier Methods Underestimated
Earlier low-bit attention methods generally focused on Queries and Keys, smoothing or rotating them to reduce outliers. In Wan2.2 analysis, however, Value still accounted for 82% of output error after Q/K processing. Value outliers do not have a stable channel or spatiotemporal location, and simply rotating V does not solve the problem: rotating V changed the error by only 0.2%.
V-Smooth uses data rearrangement instead of another fixed rotation. It performs online clustering within each batch and attention head, then places similar Value tokens together. Each 128-token hardware block subtracts its own mean, and only the residual is quantized: E4M3 for the 8-bit path and NVFP4 for the 4-bit path. The mean is not discarded. It is restored using the row sum already maintained by online softmax, avoiding a second pass or an additional buffer.
The important idea is not the mean by itself, but embedding statistical correction into state that softmax already maintains. With sequence-order blocks, the mean removes about 8% of block energy on average; after sorting, that figure reaches 36%. The cost is 0.125 extra bits per value element for the stored mean, plus the computation required for online clustering.
Softmax Has to Be Designed for Low-Bit Execution Too
The other half of VC-Attention is ExpCast-FP8. An E4M3 byte is close to a logarithmic representation of its stored value, allowing the kernel to map a log-domain score directly to an FP8 probability code. One fused multiply-add replaces FP32 exponentiation and the format conversion that follows. The report says the direct path writes the same byte as the conventional path across 79.6% of each doubling interval, and is usually only one code away elsewhere.
This is not an exact replacement under all conditions. The report gives a per-row total-variation bound below 3.64%, while the measured average across 204.8K Wan2.2 attention rows is 1.6%. More importantly, ExpCast-FP8 applies only to the 8-bit path. NVFP4 has no corresponding single affine mapping from logarithmic values to codes, so 4-bit execution does not receive this additional softmax acceleration.
V-Smooth is not free preprocessing either. Clustering runs only during the first 25% of denoising steps and reuses its permutation across four adjacent steps, yet it still consumes 3% to 4% of attention time on average. On B200, handwritten CuTe/CUDA fusion reduced one V-Smooth call from 42.2 milliseconds to 4.8 milliseconds. That makes implementation quality part of the method’s viability, not an afterthought.
A Faster Kernel Does Not Automatically Mean Faster Generation
Relative to BF16 FlashAttention-4, the report gives attention-kernel speedups of 1.46 to 1.59 times on data-center Blackwell and Hopper GPUs, and 2.3 to 3.6 times on workstation GPUs. End-to-end video generation improves much less: about 1.13 to 1.19 times on data-center GPUs and 1.36 to 1.70 times on workstation cards. In one RTX 5090 test, the attention kernel reached up to 3.58 times the speed, while end-to-end generation improved by about 1.70 times.
That gap is the central architectural lesson. A complete generation pipeline also includes other operators, denoising steps, and data movement, while attention accounts for only part of the total. The GPU’s existing baseline kernel also changes the comparison. The materials note that SageAttention2 lacks a Blackwell-specific kernel on B200 and instead uses an implementation written for earlier GPUs. A speedup against a particular baseline therefore cannot be interpreted independently of hardware, baseline maturity, and attention’s share of runtime.
For an engineering lead, VC-Attention is better treated as a backend candidate to validate than as a universal acceleration switch. It should first be benchmarked end to end on long-sequence video DiTs, with separate comparisons of the 8-bit and 4-bit paths, clustering overhead, and output quality. The materials do not provide a publicly deployable kernel or explain why the kernel is not open source. Those are deployment boundaries that reported speedup numbers cannot resolve.