Google TurboQuant Compresses LLM Cache 6x, Boosts Speed 8x, Training-Free with Zero Precision Loss

On March 26, Google Research announced the launch of TurboQuant, a new vector quantization compression algorithm that combines PolarQuant and QJL technologies. This innovation reduces the memory requirements of key-value cache (KV Cache) in large language model (LLM) inference by at least six times. On Nvidia H100 GPUs, it improves attention computation speed by up to eight times while maintaining zero precision loss across multiple long-context benchmarks. This breakthrough is expected to lower AI deployment costs and accelerate the adoption of long-context applications.

KV Cache Pain Points: High-Dimensional Vector Memory Overhead

When processing long sequences, LLMs must maintain a cache of key and value vectors. These high-dimensional vectors enable fast attention mechanism calculations without redundant computations. However, as context length increases, KV Cache memory consumption grows exponentially, becoming a major bottleneck limiting model inference efficiency and deployment scale.

While traditional vector quantization methods can compress data, they require additional storage for quantization constants such as scaling factors and zero points. These constants are typically stored in full precision, adding an extra 1–2 bits per value and partially offsetting compression benefits.

TurboQuant Core Innovation: Dual-Phase Compression with PolarQuant + QJL

TurboQuant adopts a two-phase training-free compression framework that effectively addresses the overhead of traditional quantization:

PolarQuant (Polar Coordinate Compression):

First, vectors are randomly rotated, then Cartesian coordinates (X/Y/Z, etc.) are converted into polar form (angle + radius). Because angles fall within a fixed, predictable range, this method eliminates the storage overhead required for boundary normalization in traditional quantization, enabling more efficient compression.

QJL (1-bit Error Correction, Quantized Johnson-Lindenstrauss):

After PolarQuant compression, residual errors remain. QJL uses the Johnson-Lindenstrauss transform for dimensionality reduction, then quantizes using a minimal 1-bit scheme (+1/-1 sign). By employing a special unbiased estimator, it achieves error correction during attention score calculation without additional memory overhead, ensuring the overall process remains unbiased.

Combined, TurboQuant compresses KV Cache to approximately 3 bits per value while maintaining unbiasedness and high accuracy in inner product estimation.

Benchmark Test Performance: Comprehensive Leadership, Ideal for Long Contexts

The Google team conducted extensive validation on open-source models such as Gemma and Mistral:

Additionally, on vector datasets like GloVe, TurboQuant's recall rate outperforms traditional methods such as PQ and RabbiQ.

AIbase Comment: TurboQuant requires no model retraining or fine-tuning and can be directly applied to existing LLMs. It is suitable for any scenario relying on vector quantization, including database retrieval, recommendation systems, and vector search engines. This not only enables a single consumer-grade GPU to support longer contexts (e.g., tens of thousands of tokens) but also significantly lowers the hardware threshold for enterprise-level AI services.

Industry Significance: A New Benchmark for AI Inference Efficiency

With the explosion of long-context and multimodal applications, KV Cache memory has become a core constraint in AI infrastructure. TurboQuant's "near-optimal, data-independent" quantization framework opens a new path for efficient inference. Google Research stated that this technology has been detailed in papers presented at ICLR 2026 and other conferences, with related code and implementation details expected to be gradually open-sourced.

In the future, TurboQuant is expected to be integrated into mainstream inference frameworks such as vLLM and TensorRT, further promoting the democratization and scalability of AI deployment.