Google published the TurboQuant paper. Within days, someone already got it running on Apple Silicon with near-native performance.
A developer has released an open-source implementation of TurboQuant for MLX, Apple's machine learning framework for M-series chips. The benchmarks on an M4 Pro MacBook with 48GB RAM running Qwen2.5-32B tell the story: 4.6x memory compression while maintaining 98% of the speed you'd get without any compression at all. A 16K token context window that normally eats 4.2GB of memory now takes just 897MB.
Getting there wasn't straightforward. The naive implementation ran at just 28% of normal speed - technically correct output, but painfully slow. The solution was writing custom Metal kernels (low-level GPU code specific to Apple's chips) that fuse the compression and decompression steps into single operations, plus an incremental decode buffer that avoids recompressing the entire cache on every new token.
Who This Is For
Anyone running large language models locally on a Mac. The M4 Pro with 48GB of unified memory is a popular setup for local AI, but 32-billion parameter models push right up against that memory ceiling during longer conversations. Cutting cache memory by 4.6x means you can either run longer conversations, run larger models, or both - without buying new hardware.
The implementation is open-source with a full writeup covering the optimization journey from 0.28x to 0.98x native speed. For the MLX community, this is immediately usable, not a proof of concept.
The speed of this port - paper to working Apple Silicon code in days - says something about where local AI development is right now. The gap between cutting-edge research and "I can run this on my laptop" keeps shrinking.