Evidence at a glance
The mechanism in one line
Compress the visual or contextual input before the main reasoning path.
Route or verify the expensive step instead of repeating the full path.
Translate the mechanism into a bounded deployment or evaluation check.
Why Static Optimization Fails in Inference Services
In large language model inference, operator performance is highly sensitive to input data distribution. Together AI notes that traditional kernel tuning is typically based on pre-set random datasets or fixed-length prompts, which deviates significantly from production traffic patterns. In some scenarios, request length distributions fluctuate from 70K to 97K tokens across different regions, while other clients have seen P90 input lengths grow from 18K to 33K in just three days. When kernels are pre-compiled against static data, these sudden, long-context loads often cause execution efficiency to crater. For instance, in an MoE model, optimization schemes tuned on random data not only lost their edge under 70K context but often regressed due to branch mispredictions and inefficient memory access patterns.
A more critical bottleneck is the 'startup tax' which creates a paralysis in performance tuning. Standard high-performance engines like vLLM or SGLang require a full cold start of the engine to switch kernels or re-test parallel layouts. Given the multi-gigabyte weight loading times, this often takes hundreds of seconds, making it practically impossible for engineers to iterate on performance under real-time traffic. Together Compiler (TKCC) aims to solve this by decoupling the optimization loop, allowing experiments to run alongside production serving in the same instance, thereby tackling the performance drift that inevitably follows deployment.
Operator Hot-Rewriting and Tiered Measurement Mechanisms
The core logic of Together Compiler is to maintain a continuous compilation loop: coding agents automatically generate improved operator schemes and inject them into the live serving engine via RPC. Unlike traditional approaches, TKCC avoids engine restarts by utilizing internal hooks to swap function pointers for operators in real-time, subsequently triggering a re-capture of CUDA graphs. This architecture dramatically boosts experiment throughput; Together reports that it can perform 252 times more experimental trials in the same timeframe compared to engine-restart baselines, compressing the feedback loop from hours to minutes.
To balance experimental fidelity and speed, the system employs a 'two-tier screening' strategy. The Fast Tier tests candidate code on small 12-layer model segments, quickly filtering out underperformers. Schemes that pass are then routed to the Slow Tier, which executes on the full 60-layer model under real-world traffic distributions to confirm global performance gains. By adopting this approach, TKCC tested 8.1x more candidates than a traditional slow-only tier, successfully uncovering solutions that were 7.2% faster. This mechanism ensures that code generated by agents is not just theoretically faster, but robustly tuned for the nuances of actual inference environments.
Managing the Search Space: Parallel Layouts and Cost Modeling
Beyond operator rewriting, TKCC introduces a cost model to automate decisions regarding parallel layouts. In inference scaling, the combinatorial configuration of Expert Parallelism (EP), Tensor Parallelism (TP), and Data Parallelism (DP) directly dictates memory usage and communication overhead. This model does not rely simply on theoretical hardware specs; it integrates micro-benchmarks, observed imbalances in MoE expert loads, and real-world request length statistics. When testing the Kimi K3 model, this cost model automatically chose a parallel layout distinct from traditional baselines, resulting in up to a 24% boost in TPS.
It is worth noting that in the case of MiniMax M3.1 Flash, TKCC switched the parallel strategy from TP4 to TP8. While this improved throughput, it implies that when comparing against the baseline, TKCC was not just optimizing code, but re-allocating how cluster resources were utilized. This reveals a critical engineering takeaway: the optimization frontier for agents should not be constrained to 'kernel tuning' but must encompass system-level parameters of the inference cluster. By treating both model architecture and compute resource distribution as a joint search space, TKCC demonstrates greater adaptability across diverse hardware targets (such as NVIDIA B200 and AWS Trainium2), automatically identifying optimal parallel dimensions tailored to the underlying architecture.
Boundaries, Trade-offs, and Engineering Judgments
While TKCC achieves impressive performance gains (up to 2.6x TPS), the operational complexity it introduces is non-trivial. Embedding agents deep into the production loop effectively makes the operator generation process a core component of the infrastructure. In Together's experiments, naive schemes generated entirely by agents outperformed human-curated expert knowledge bases in certain scenarios, proving that in highly automated environments, manual heuristics can inadvertently become bottlenecks that limit innovation in the search space. However, this also demands a robust regression testing framework, as agent-generated operators carry the theoretical risk of producing non-deterministic behavior.
Current evaluation suggests that TKCC is best suited for inference services with highly volatile traffic and intense sensitivity to throughput. For scenarios where stability is paramount and traffic patterns are relatively flat, the additional compute overhead and potential unpredictability introduced by continuous compilation might outweigh the benefits. From a systems perspective, TKCC represents an evolutionary path for future inference engines: transitioning the compilation layer from a 'static offline tool' to a 'real-time sensing service.' For engineering leaders, the challenge lies in building a verification framework that balances automated agent-driven optimization with service reliability, ensuring that this continuous compilation loop does not evolve into an opaque and untraceable performance trap.