Google Research, DeepMind and NYU have introduced TurboQuant, and this could be one of the most important infrastructure works of the year in AI.

On paper, it doesn't look too exciting: vector quantization, compression of high-dimensional vectors, theoretical guarantees. In practice, it's about two very expensive things: KV-cache in large language models and vector search.

In both cases, the old problem is the same. You need to compress the data as much as possible, but not spoil the geometry of the vectors, i.e., distances and dot products. If you spoil it, problems with attention, retrieval, ranking, and model response quality begin.

That's exactly where TurboQuant hits.

The authors claim that their method almost approaches the theoretical limit in terms of distortion rate. Simply put, the limit of how well such vectors can be compressed at a given bitrate. For an academic work, this is already a strong thesis.

Most practical schemes here have an old trade-off. Either you need heavy offline training, calibration, and codebooks, or the method is fast and convenient for online use, but the compression quality is mediocre. TurboQuant tries to shift this trade-off.

Moreover, the problem is not only about quality loss. Google in its blog separately writes about another unpleasant detail of classic vector quantization: memory overhead. You seem to save bits, but then you have to spend part of that saving on storing quantization constants in full precision for small blocks of data. According to them, such overhead can add another 1-2 bits per number. That is, part of the gain is simply eaten up by overhead.

Against this background, TurboQuant looks like an attempt to solve the entire package of problems at once: reduce memory, eliminate extra overhead, require no training or fine-tuning, and not break downstream tasks.

Importantly, the paper does not reduce everything to MSE. For modern models, this is not enough. In attention and retrieval, it is critical to preserve the inner product. The authors directly show that a quantizer optimized only for MSE can give a biased estimate of the dot product. Therefore, they have a two-step scheme: first, basic compression, then a 1-bit residual correction via QJL to remove bias.

To put this in plain language, the idea is simple: compressing a vector is not enough. You need to do it in such a way that the model doesn't start thinking about something slightly different afterward.

According to the stated results, the picture looks strong. In the paper, the authors write that for KV-cache, they achieve quality neutrality at 3.5 bits per channel and only slight degradation at 2.5 bits. In Needle-in-a-Haystack, according to their data, TurboQuant preserves the quality of the full-precision model at 4x compression. On LongBench, the 3.5-bit variant goes almost on par with the full cache on Llama-3.1-8B-Instruct.

But in the Google blog, there is perhaps an even more important thesis. There, they specifically emphasize that TurboQuant can compress KV-cache to 3 bits without training and fine-tuning and without quality degradation on their tests. Moreover, they claim that 4-bit TurboQuant provides up to 8x speedup in computing attention logits on H100 compared to 32-bit unquantized keys.

This is a direct talk about the cost and speed of serving.

Equally intriguing is the part about vector search. The authors claim that TurboQuant outperforms Product Quantization and RabitQ in recall, and indexing time is almost zero. For systems that need to store and search through huge embedding indices, this is potentially very big money.

Of course, one should not lose their head. This is a story with very strong claims, so external replication is especially important. Phrasings like zero accuracy loss, near-optimal distortion, and up to 8x always need to be checked against implementation details, baselines, and experimental settings.

But in the end, the news is indeed big.

If external verification confirms at least most of what is claimed, this will be one of the most important infrastructure works in AI in recent times.


❗️❗️❗️❗️❗️❗️❗️❗️ / Not banned in Russia