TurboQuant: A first-principles walkthrough

Eight ideas the rest of the page is built on.

Each mini-demo below covers one concept used later. Skip the ones you already know.

[0.3, −1.2]

. Geometrically it is an arrow from the origin. A

d-dimensional vectoris an arrow in $d$-space, hard to picture past 3-D, but the rules are the same.

**[0.70, 0.50]**length

0.86

Length= $\sqrt{x_1^2+x_2^2+\dots}$.

Inner product$\langle x,y\rangle = x_1 y_1 + x_2 y_2 + \dots = |x||y|\cos\theta$. The inner product reaches its largest positive value when the two arrows point in the same direction. It drops to zero when the two arrows are perpendicular. It becomes negative when the arrows point in opposite directions, with its most negative value when they point exactly opposite.

1.00‖y‖

1.00⟨x,y⟩

0.00angle

90°

Error is the distance between a guess and the truth. Scoring a guess by the signed error lets positive and negative errors cancel, which means the score does not penalise being off. Squaring forces every error to count as a positive number and gives big errors a larger penalty than small ones. The guess that minimises the mean of squared errors is the data’s average: it is the unique number that minimises the sum of squared distances to the points.

The first moment of a quantity $X$ is its mean $\mathbb{E}[X]$; the second moment is the mean of its square $\mathbb{E}[X^2]$. A zero-mean variable has a vanishing first moment because positive and negative deviations cancel. Its second moment is strictly positive whenever any deviation is nonzero, because squared values are nonnegative and cannot cancel. The MSE above is itself a second moment of the residual error. This distinction returns in §7, where the per-input gap $\tilde y - y$ averages to zero in the first moment, while its square averages to a strictly positive quantity in the second.

The average has a property we will use in §7. It lies between the data’s most extreme points, so its magnitude is smaller than at least one of them. When a quantizer compresses a whole bin of values down to the bin’s average, the stored value is smaller in magnitude than the bin’s largest values. The reconstruction is a shrunken version of the input. An inner product against a shrunken reconstruction comes out smaller than the same inner product against the input.

0.00MSE at guess

1.00MSE at mean

1.00

An estimator is a procedure that takes data and returns a guess $\hat\theta$ for an unknown truth $\theta$. Repeat it on fresh data and the guesses form a cloud. The cloud can fail in two independent ways. Variance is one: individual guesses are noisy. Bias is the other: the procedure is wrong even after averaging many guesses. An estimator with $\mathbb{E}[\hat\theta]=\theta$ is unbiased; the cloud’s centre sits at $\theta$ regardless of the cloud’s width.

The bullseye below shows both failure modes. Bias is the distance from the cloud’s centre to the crosshair. Variance is the width of the cloud. The two quantities are independent of each other. §7 runs the same bullseye against the MSE quantizer of §6, and the cloud’s centre lands away from the crosshair. §8 runs it against a different estimator whose cloud centres on the crosshair.

0mean of shots

**–**bias

–

rotation matrix$R$ spins space. The key property: $|Rx|=|x|$ and $\langle Rx,Ry\rangle=\langle x,y\rangle$. Rotation only changes the

basisthe coordinates are written in, not the geometry.

1.41 → 1.41preserved?

yes

Central Limit Theoremsays that summing enough independent random numbers produces a distribution close to a bell curve. The shape of each individual term in the sum does not affect the limit. A sum of coin flips converges to the same Gaussian shape as a sum of uniform draws or a sum of skewed draws. A rotated coordinate is one of these sums: it is a weighted combination of every coordinate of the original vector, with random weights. After a random rotation, each new coordinate is therefore approximately Gaussian, which is the property TurboQuant relies on for every input.

±1 coinconverged?

no, n too small

measure concentration, and it is the core fact TurboQuant exploits.

≈ 0.711/√d

0.71

4gap Δ

0.667max error

0.333

Vector: ordered list of numbers / arrow from the origin. Length & inner product: the norm $\sqrt{\sum x_i^2}$ and how much two vectors point the same way. MSE: average squared error. Unbiased: the average of many estimates equals the truth. Rotation: change of basis that preserves lengths and angles. CLT: sum of many independent randoms converges to a Gaussian. High-D concentration: coordinates of a random unit vector in $d$-space cluster near $\pm 1/\sqrt d$. Quantization: snap each number to one of $2^b$ levels; one extra bit quarters the squared error.

What is vector quantization, really?

You have a vector $\mathbf{x}\in\mathbb{R}^d$, say $d{=}1536$ floats from an OpenAI embedding. You want to store it using $b$ bits per coordinate (total $b\cdot d$ bits), then later recover an approximation $\tilde{\mathbf{x}}$ close to $\mathbf{x}$. Closeness is measured by

MSE distortion$D_{\text{mse}} = \mathbb{E}\big[,|\mathbf{x} - \tilde{\mathbf{x}}|_2^2,\big]$ or

inner-product error$D_{\text{prod}} = \mathbb{E}\big[,|\langle\mathbf{y},\mathbf{x}\rangle - \langle\mathbf{y},\tilde{\mathbf{x}}\rangle|^2,\big]$

The second one matters because attention scores and nearest-neighbor queries are all inner products. We would like the estimator to be unbiased: $\mathbb{E}[\langle\mathbf{y},\tilde{\mathbf{x}}\rangle] = \langle\mathbf{y},\mathbf{x}\rangle$.

MSE distortion: average squared error between the true vector and its reconstruction, primer §0.3.

Inner product $\langle y, x\rangle$: how much two vectors point the same way, primer §0.2. This is what attention computes.

Estimator: a rule (here: quantize, then decode) that returns an approximation $\hat s$ of a true number $s$.

Unbiased estimator: across many queries, the average of $\hat s$ equals $s$. Individual estimates can be noisy; the mean is on target. Primer §0.4.

The obvious quantizer

For each coordinate, pick the closest of $2^b$ evenly-spaced levels in $[-1, 1]$. That is $b$ bits per number. The same rule runs in 2D and 3D first, where the geometry is visible, before the high-dimensional version below.

First, in 2D

Drag the tip of the vector. The vector snaps to the nearest point of a $2^b \times 2^b$ grid. The green arrow shows the original input. The blue arrow shows where the input is quantized to. The red segment between them is the reconstruction error $\mathbf{x} - \tilde{\mathbf{x}}$.

Same trick in 3D

A $2^b$-level grid on three axes gives $2^{3b}$ snap points. Drag the canvas to orbit the view. The spike preset shows where the construction breaks: the input lies near one axis and falls between two grid levels, which is where the reconstruction error is largest.

Now at scale (d up to 128)

The same rule applied to every coordinate of a high-dimensional vector. You cannot see the grid anymore, but the per-coordinate errors are still there.

Select the spike input. The naive quantizer's grid is spaced evenly over $[-1, 1]$. The input has almost all of its magnitude in a single coordinate, whose value falls between the two grid levels nearest to it and so reconstructs poorly. The remaining coordinates are near zero and consume most of the levels despite carrying little of the input's information.

A fixed grid produces small reconstruction errors on inputs whose coordinates are roughly uniform in magnitude, and large reconstruction errors on inputs whose magnitude is concentrated in one or a few coordinates. Next: §2 shows how production systems handle the second case and what they pay for the fix.