Evidence at a glance
The mechanism in one line
Compress the visual or contextual input before the main reasoning path.
Route or verify the expensive step instead of repeating the full path.
Translate the mechanism into a bounded deployment or evaluation check.
Long Histories Cost More Than Their Length Suggests
Researchers from Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, an SGLang-based serving system for agents whose histories grow through repeated reasoning and tool use. It does not provide new model weights. Instead, it changes how the serving system manages KV caches. The motivation is specific: repeatedly reading an agent’s history can make cache reads a significant part of the cost.
The paper’s cost analysis attributes 82.4% of total API cost to KV-cache reads. A common response is to summarize or trim text after the context reaches a threshold. But changing a prefix can invalidate later cache states, forcing retained content to be prefilled again, while summarization can erase details from intermediate turns. AKV keeps the original text trajectory and rejects the assumption that a long history must be rewritten.
It Changes What Attention Can See, Not the Conversation
AKV defines three cache operations. Append adds states for new tokens, drop removes selected states from future attention, and repos rearranges the positions of retained states. Repos maps retained keys to consecutive positions and adjusts their positional encoding through RoPE rotation; values remain unchanged, so hidden states do not have to be recomputed. In other words, drop does not delete messages. The client can still submit the full history, while the system changes which KV states remain visible.
This allows retained states to be reused rather than recomputed after a text-prefix change. In an ablation, adding repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%. That is not evidence that repositioning always improves performance. It is a result for one model and task, and the interaction between cache operations and positional encoding is one of the mechanisms that must be validated.
Less Recomputation Requires More Careful State Management
Cache engineering moves part of the problem from model-call policy into the serving system. The same token sequence can produce different cache states after different drop or repos operations. Ordinary prefix matching may therefore reuse an incorrect state. AKV extends SGLang’s radix tree to track these operations and reuse relationships, and frees dropped KV only after no active request depends on it.
A second mechanism, segment-batch attention, restores evicted states in one prefill pass. This makes AKV more than a client-side switch for trimming history: the server must understand operation history, cache lifetimes, and recovery paths. The code is Apache-2.0 licensed, and the API is stateless and OpenAI-compatible. Clients can pass operations through drop_message and reposition fields, use turn-, token-, or model-based policies, or let the model initiate a drop through CacheTool.
The Average Improves, but Not Every Task Does
The paper evaluates MiniMax-M2.7, GPT-OSS-120B, and MiroThinker-1.7-Mini across BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510. Against Standard, a threshold-triggered summarization baseline, AKV reports a 27.7% average cost reduction and a 1.7-point average score increase. Against Text Engineering, the reported average cost reduction is 21.9%. These figures use different baselines and should not be collapsed into a single unconditional savings claim.
Individual results show why the scope matters. On SWE-bench Verified, GPT-OSS-120B’s per-task cost fell from 20.0 to 9.7 US cents while its score rose from 40.0 to 43.2. But MiniMax-M2.7 scored 58.9 on BrowseComp-Plus, below Text Engineering’s 62.3. Long runs show another pattern: at 96K tokens and above on BrowseComp-Plus, GPT-OSS-120B scored 37% with AKV, compared with 24% for Standard and 32% for Text Engineering. The evidence makes cache engineering worth testing, not a guarantee of higher accuracy on every task.
Evaluate the Whole Serving Path Before Deployment
AKV’s performance claims also have clear limits. The serving experiments used eight NVIDIA A800 80GB GPUs. The paper reports peak gains of up to 1.18x in output throughput and 1.51x in P95 request latency, but those figures compare each method at its own peak-throughput setting. They are not universal speedups under a fixed workload. Cost figures are API-equivalent estimates based on published prices, with repositioned tokens charged at the cache-hit rate, a deliberately conservative pricing assumption.
A useful pilot should measure more than token charges. Track task scores, cache-hit and recovery overhead, end-to-end latency, and server-side cache occupancy and eviction behavior, segmented by model and task. Check that the serving stack can correctly track cache-operation history, and assess how submitting full message histories affects actual billing and transfer costs. The available material reports the authors’ experiments on specific models, benchmarks, and hardware, and does not mention independent replications. AKV is best treated as a serving-architecture option to test, not a replacement already proven to work everywhere.