Running a large language model is expensive mostly because of memory. Every time an AI model processes a long conversation or document, it stores temporary data called key-value (KV) cache - essentially the model's short-term memory of everything it has read so far. That cache eats GPU memory fast, and GPU memory is the bottleneck that determines how many users a company can serve at once.
Google just published a way to make that cache 6x smaller without losing any accuracy. The algorithm, called TurboQuant, will be presented at ICLR 2026, one of the top machine learning conferences.
How It Works (Without the Math)
Quantization is the practice of storing numbers with fewer bits. A standard AI model stores values in 16 or 32 bits of precision. TurboQuant compresses KV cache values down to just 3 bits - roughly the difference between storing a number as "3.14159265" versus "3.1." The trick is doing this without the model giving worse answers.
TurboQuant achieves this through a two-step process. First, an algorithm called PolarQuant converts the data from standard coordinates into polar coordinates (think: radius and angle instead of x and y). This makes the data more predictable and easier to compress. Second, a correction step called QJL (Quantized Johnson-Lindenstrauss) cleans up any remaining errors using a 1-bit error-correction pass.
The result: 3-bit compression that requires no retraining, no fine-tuning, and adds negligible processing overhead.
The Benchmark Results
Google tested TurboQuant on open-source models including Llama 3.1 8B, Gemma, and Mistral across six benchmark suites: LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, L-Eval, and GloVe vector search.
The headline numbers: 6x memory reduction on KV cache storage, and up to 8x speedup in computing attention scores on Nvidia H100 GPUs compared to uncompressed 32-bit keys. Across all benchmarks, accuracy stayed flat - no measurable degradation in question answering, code generation, or summarization tasks.
For context, the H100 is the GPU that most AI companies use in their data centers. An 8x speedup on that hardware translates directly to either serving 8x more users on the same machines, or cutting your GPU bill dramatically.
Practical Impact
The internet immediately compared TurboQuant to the fictional compression algorithm from HBO's "Silicon Valley," which is fun but undersells what's happening here. This is infrastructure-level optimization that could meaningfully reduce the cost of running AI services.
For anyone running AI locally on consumer hardware, better KV cache compression means longer context windows (the amount of text a model can consider at once) on GPUs with limited memory. A GPU that could previously handle 32k tokens of context might handle 192k with TurboQuant-style compression.
For cloud providers, the math is simpler: same performance, fewer GPUs, lower costs. VentureBeat estimates potential cost reductions of 50% or more for inference workloads.
The caveat: TurboQuant is a research paper, not a product. Google hasn't announced plans to integrate it into Gemini or any commercial service. But the technique works on open-source models, so expect third-party implementations to appear quickly. When the cost of running AI goes down, everyone benefits - including the companies whose tools we review every day on this site.