Xiaohongshu, Peking University, Shanghai Jiao Tong University Unveil HYPIC to Slash First Token Latency by 3.25x in Hybrid Attention Models

The primary focus of large model services is transitioning from single-turn conversations to retrieval-augmented question answering, multi-document summarization, and long-range agents. These emerging workloads share a key trait: a single request's prompt often comprises dozens to hundreds of semantically independent segments, including retrieved documents, skill descriptions, memory files, and previous conversation rounds, all assembled into ultra-long contexts ranging from tens of thousands to hundreds of thousands of tokens. At this scale, the prefill phase dominates the computational cost per request, becoming the most visible expense for service providers. More critically, cache misses can cause tail-first token delay (TTFT) to surge to several seconds, severely degrading user experience.

The industry has adopted two cost-reduction strategies. The first is position-agnostic caching (PIC), which relaxes strict prefix constraints, allowing each semantically independent segment to be cached once and then spliced into any prefix, perfectly aligning with the prompt composition of RAG and agents. Its core mechanisms are splice, which directly concatenates along the token axis, and correction, which recalculates a few tokens to restore cross-segment context. The second approach is the hybrid attention model, which replaces most full attention layers with linear attention, reducing quadratic complexity to linear and compressing unbounded history into a fixed-size cyclic state. Recent production models like MiniMax-M1, Ring-2.5, Qwen3.5, and Kimi-Linear linearize over 75% of their layers, retaining only a few full attention layers, forming the mainstream hybrid design. For instance, Qwen3.5-35B-A3B features 30 linear layers out of 40 total.

The core contradiction lies here: existing PIC operates on per-token key-value (KV) caches, while linear attention layers only expose a per-request cyclic state, lacking per-token hooks for splice or correction. Consequently, most layers in hybrid models fall outside the capabilities of existing PIC systems. Until now, no system provided position-agnostic caching for hybrid attention large models.

Developed by the Xiaohongshu large model inference team in collaboration with Peking University and Shanghai Jiao Tong University, HYPIC is the first service system to implement PIC on hybrid attention large models. Testing across four production-level hybrid attention models and five workloads shows that HYPIC reduces first token delay by an average of 3.25 times and increases sustainable QPS by 1.66 times under the same SLO, with task quality differing by only 1.71 points from full recomputation. The system's core idea is to handle linear and full attention layers separately within the hybrid model, accelerating cold requests through layer-level parallelism via three interlocking mechanisms.

For linear layers, HYPIC caches "transition operators," enabling constant-time state combination. The most straightforward PIC straw man solution is to directly add the final states of two segments with zero initial values—valid in naive linear attention but failing in advanced linear attention with decay, gating, or delta erasure (e.g., RetNet, Mamba2, GLA, DeltaNet, GDN, KDA). The critical missing quantity is the accumulated transition operator T_C of a segment, i.e., the product of all token transition matrices within it. The actual final state should be S(C1·C2) = T(C2) · S(C1) + S(C2), while simple addition omits the T(C2) term, causing structural error. Tests showed the slow decay head in RetNet exhibited an error of 22% of the state norm at 256 tokens. HYPIC’s key insight is that T_C and the zero-initial-state final state S(C|0) are determined solely by a segment's internal tokens, independent of the prefix. Thus, during the first prefill of a segment, HYPIC caches the pair (T_C, S(C|0)), and upon reuse, applies left multiplication of T_C followed by addition to nearly exactly reconstruct the final state under any prefix, naturally covering the entire linear attention family. Tests on Qwen3.5-35B-A3B showed combined states differed from full recomputation by only 6×10⁻⁵, within FP16 noise. For variants with causal convolution or RoPE, HYPIC uses two patches—convolution state warm-up and state re-rotation—to achieve strict alignment.

For full attention layers, HYPIC uses "seam windows" to fix cross-segment attention. Although a few full attention layers still require PIC in hybrid models, previous selective recomputation cannot be directly migrated because downstream linear layers retain only the final state, making it impossible for non-terminal tokens to pass upward through the full attention layer. Storing cyclic state per token or recomputing from scratch are infeasible due to storage or computation constraints. The team observed that bias after KV concatenation is highly concentrated at the beginning of each reused segment, with the rest almost unaffected, consistent with the attention sink phenomenon in full attention models, which also holds in hybrid attention models. Based on this, HYPIC constructs a seam window of w tokens at the beginning of each internal segment, excluding these tokens from caching. Upon reuse, they are recomputed under the complete prefix to repair cross-segment attention, with w=8 by default; since actual segment length is usually greater than 512 tokens, the seam window constitutes a very small portion, making recomputation cost manageable. To support recomputation of the initial KV, linear attention layers compute the T and S of the seam window during reuse, advancing the running state while forwarding per-layer outputs of each seam token to the upper full attention layer.

The third component is segment-level parallelism, transforming "long cold requests" into accelerated workloads. Cache misses are inevitable due to continuous corpus updates and low-frequency document removals, with long cold requests being the main cause of tail latency. Existing systems treat the entire prompt as a single entity, serially prefilling all cold segments on a single instance, with TTFT growing as O(n·|C|); although tensor parallelism and other intra-instance parallelisms can speed up individual forward passes, their scalability is limited, with a 100k-token request on TP-8 still taking 17.7 seconds. However, PIC has already made each segment self-contained—its prefill results depend only on its internal tokens. HYPIC proposes inter-instance segment-level parallelism: the Router detects whether each segment hits the cache, then distributes cold segments in parallel to multiple scatter workers, and a combine worker assembles the results of each segment into a complete running state, reducing cold prefill time from O(n·|C|) to O(⌈n/m⌉·|C|+c). To fully utilize this path, HYPIC uses a greedy LPT (Longest Processing Time) strategy to balance workload across workers, and overlaps computation and transmission of each segment to avoid the combine worker being held back by synchronous transmission bursts; segment-level parallelism and intra-instance parallelism (TP/DP/SP) operate on orthogonal axes and can be seamlessly combined.

This system has been implemented on SGLang, with approximately 14,000 lines of Python and Triton code. It was evaluated on eight H20 nodes using four production-level hybrid attention models (Ring-mini/flash-linear-2.0, Qwen3.5-35B-A3B/122B-A10B), four public datasets (HotpotQA, TriviaQA, MultiNews, GovReport), and one production RAG trace. After full preheating, compared to Prefix Cache, HYPIC reduces p50 TTFT by an average of 3.25 times (from 2.77× to 4.05×), with almost no quality loss—16 "model × dataset" units averaged only 1.71 points behind full recomputation, and even exceeded it by 0.47 points on Qwen3.5-35B. As a comparison, a naive addition without caching transition operators lost 66.9% of the score, confirming the necessity of T_C. On the production RAG trace, under the same 1-second TTFT SLO, HYPIC increased sustainable QPS by 1.66 times (from 1.49× to 1.85×), and peak throughput per card increased by about 1.3 to 1.5 times. On the pure cold miss path, a 32k-token request's TTFT dropped from 2.83 seconds on a single worker to 0.49 seconds on eight workers, achieving a 5.7× speedup, with nearly all gains coming from the scalable scatter stage. Against uneven sharding, LPT reduced TTFT from 1.26 seconds to 0.84 seconds (3.6× vs. 2.4×). The overhead is also manageable: the one-time cost of constructing (T_C, S(C|0)) is only 5.2% to 6.7% of the main forward pass in the heaviest dense transfer model, and this cost can be amortized by reusing it multiple times.

HYPIC brings position-agnostic caching to hybrid attention large models for the first time: it enables constant-time state combination for linear layers using cached segment-wise accumulated transition operators, fixes cross-segment alignment for full attention layers with small seam windows, and turns long cold requests into accelerated workloads using segment-level parallelism. This allows long-context services like RAG and agents to enjoy the benefits of segment-level cache reuse while embracing efficient hybrid architectures. The team stated that future work will focus on exploring distributed PIC management and scheduling, and multi-level cache hierarchy management, further advancing large model inference toward faster, more cost-effective, and more scalable solutions.