Model quantization: how big AI models shrink to run in a browser

The promise of on-device AI — that a capable model runs entirely in your browser, on your own machine, without sending your data anywhere — runs straight into a hard physical fact: capable models are large, and a browser cannot download or hold an arbitrarily large one. A model that is hundreds of megabytes is a slow download and a heavy memory load; a model that is several gigabytes simply will not run in a typical browser tab at all. So the whole project of running real AI locally depends on making models small enough to be practical, and the single most important technique for doing that is quantization. If on-device AI feels almost too good to be true, quantization is a large part of how the trick is actually done.

Quantization sounds technical and a little forbidding, but the core idea is genuinely simple and worth understanding, because it explains why the privacy-first, runs-on-your-device approach is possible at all rather than just a marketing aspiration. This article explains, without assuming a machine-learning background, what a model actually is under the hood, what quantization does to it, why it works as well as it does, and what it costs. The short version is that a model is billions of numbers, quantization stores those numbers in fewer bits each, and the result is often around four times smaller for a remarkably small loss in quality — which is exactly the trade that turns an impractical model into one that fits in a browser.

To see why quantization matters so much, you have to appreciate that for on-device AI, size is not one consideration among many — it is the gatekeeper that decides whether the whole approach is viable. A model has to be downloaded to the user’s device before it can do anything, and that download competes with every other reason a user might lose patience and leave. It then has to fit in the memory the browser tab is allowed, alongside everything else the page is doing, and a model that is too large does not run slowly, it does not run. Both constraints push relentlessly in one direction: smaller is the difference between a feature that exists and one that does not.

This is why the size limits of browser AI, explored in /product-blog/how-big-are-browser-ai-models-and-why, set the boundary of what on-device AI can do, and why shrinking models is not a nice-to-have optimisation but the enabling technology. Every megabyte you can remove from a model is a faster first use, a wider range of devices that can run it, and more headroom for the model to actually be capable rather than spending its entire size budget just being loadable. Quantization is the most effective single lever on that size, which is why it sits at the heart of making real models run where the data already is, instead of shipping the data to where a big model lives.

Strip away the mystique and a neural network is, concretely, an enormous collection of numbers called weights — the values it learned during training that encode everything it knows. A capable model has millions to billions of these weights, and at full precision each one is stored as a floating-point number using a fixed number of bits, commonly thirty-two or sixteen bits each. The model’s size on disk and in memory is essentially the count of its weights times the bits used to store each weight, so the total comes out to those hundreds of megabytes or gigabytes that strain a browser. There is nothing else hiding in there driving the size; it is overwhelmingly just the weights.

Once you see a model as "a big pile of numbers, each stored in so many bits," the lever quantization pulls becomes obvious. If the size is the number of weights multiplied by the bits per weight, and you cannot easily reduce the number of weights without changing the model, then the other factor — the bits per weight — is the thing to attack. Store each of those billions of numbers in fewer bits and the whole model shrinks proportionally, with the number of weights, and therefore the model’s fundamental structure and capability, left intact. That is the entire premise of quantization: keep the numbers, spend fewer bits describing each one.

Quantization is the process of representing a model’s weights with lower precision — fewer bits each — than they were trained at. The most common form takes weights stored as sixteen- or thirty-two-bit floating-point numbers and converts them to eight-bit integers, which immediately makes each weight a quarter the size of a thirty-two-bit original, and the model roughly four times smaller overall. More aggressive schemes push to four bits or even fewer per weight, trading more quality for still-smaller size. The headline outcome is a dramatic reduction in the bytes needed to store and load the model, achieved purely by being less precise about each individual number.

The mechanism is a mapping from a wide range of high-precision values onto a small set of low-precision ones. Instead of recording each weight as a finely-detailed decimal, quantization records which of a limited number of "buckets" it falls into, storing the bucket index in a few bits and remembering how to translate those buckets back to approximate real values. It is the same idea as rounding: you lose some detail by snapping each value to the nearest available step, but you gain enormously in how compactly you can store it. The art is in choosing the buckets well so the rounding does as little damage as possible, which is where the various quantization schemes differ.

The surprising thing — the reason quantization is not just a desperate compromise but a genuinely good trade — is how little a model’s quality typically suffers from this rounding. The intuition is that a neural network is not a delicate calculation where every decimal place matters; it is a robust statistical system whose behaviour emerges from the combined effect of billions of weights, and that emergent behaviour is remarkably insensitive to small errors in any individual weight. The full precision the model was trained at turns out to be more than the model needs to do its job, so trimming it back to a coarser representation costs far less than you would naively expect.

Put another way, much of the precision in a full-precision model is not actually carrying useful information; it is detail finer than the task requires. Quantization exploits exactly that slack, throwing away the precision the model can spare while keeping the precision it actually relies on. This is why an eight-bit model is often nearly indistinguishable in quality from its full-precision original despite being a quarter of the size — the discarded bits were largely describing distinctions too fine to matter to the output. The model degrades gracefully rather than catastrophically as precision falls, which is the property that makes the whole technique viable, and it is why quantization can be aggressive before the quality cost becomes noticeable.

The most obvious payoff is size, and it is substantial: an eight-bit quantized model is roughly a quarter the download and a quarter the memory footprint of its thirty-two-bit original, which can be the difference between a model that downloads in a few seconds and runs comfortably and one that never loads at all. For on-device AI, where the model must arrive over the network and live in a constrained memory budget, that reduction is not a minor optimisation — it is what moves a model from impossible to practical in the browser. A great deal of what makes a tool feel responsive on first use traces directly back to how aggressively its model was quantized.

There is frequently a speed bonus too, because lower-precision arithmetic is often faster for hardware to perform. Integer math can run more quickly than floating-point on many processors, and moving smaller numbers around means less data shuttling through the memory system, which is often the real bottleneck in inference. So quantization can make a model not only smaller but faster to run, a rare case where the compression also improves performance rather than trading against it. The combination — smaller download, smaller memory footprint, and faster execution — is why quantization is close to a free lunch for on-device deployment, as long as the accuracy cost stays acceptable.

It would be dishonest to present quantization as entirely free, because it is not — it is a trade, and the cost is some loss of accuracy. For most models and most tasks the loss is small enough to be imperceptible in practice, which is why quantization is so widely used, but it is not always negligible. Push the precision low enough, or apply it to a model or task that happens to be sensitive, and the quality degradation becomes visible: outputs get slightly worse, edge cases get handled less well, the model’s behaviour drifts from its full-precision self. The right amount of quantization is therefore task-dependent, balancing how much size you need to shed against how much quality you can afford to lose.

This is why responsible on-device tools care about which quantization they ship and are honest about the trade. A tool that quantizes aggressively to fit a tight size budget is making a real decision about quality, and the right level depends on whether the task tolerates it — some jobs are forgiving of a slightly coarser model, others are not. The mature approach is to quantize as much as the task allows and no more, measuring the actual quality impact rather than assuming it, and to be transparent that the on-device model is a deliberately compressed version. The aim is the best quality that fits, not the smallest model regardless of output, and knowing where that balance sits for a given task is part of doing on-device AI well.

There are broadly two approaches to quantizing a model, and the difference matters for how much quality survives. The simpler is post-training quantization: take an already-trained full-precision model and convert its weights to lower precision after the fact. This is fast, requires no retraining, and works well in many cases, which makes it the common default — but because the model never "knew" it would be quantized, it can suffer more quality loss, especially at aggressive precision, since nothing in training prepared it for the coarser representation.

The more involved approach is quantization-aware training, where the model is trained, or fine-tuned, with the quantization simulated during the process, so it learns weights that survive the rounding gracefully. This costs more effort — it requires the training pipeline and the data — but it generally preserves more quality at a given precision, because the model effectively adapts to the constraint it will be deployed under. The choice between them is the usual engineering trade: post-training quantization when its quality is good enough and you want it cheaply, quantization-aware training when you are pushing precision low enough that the extra quality is worth the cost. Either way the principle is the same; the difference is how much care goes into preserving accuracy under compression.

A quantized model is only useful if the device can actually execute it efficiently, and this is where the browser’s evolving access to hardware matters. Lower-precision integer math is exactly the kind of operation modern processors and graphics hardware are good at, and the browser increasingly exposes that hardware through interfaces like WebGPU and WebAssembly, letting a quantized model run with real performance rather than as a slow emulation. The runtime that executes the model maps its quantized operations onto whatever acceleration the device offers, so the same compressed model can run respectably across a wide range of hardware.

This hardware story is the other half of why quantization unlocks on-device AI: it is not enough to make the model small if the device cannot run it quickly. The pairing of compact, quantized models with browser access to acceleration — covered from the runtime side in /product-blog/how-ai-runs-in-your-browser — is what turns "a model small enough to download" into "a model that downloads fast and runs fast on ordinary hardware." Quantization shrinks the model and, conveniently, shrinks it into exactly the integer-heavy form that the available hardware handles best, so the size win and the speed win come from the same change.

Quantization is the most impactful single technique for shrinking models, but it is not the only one, and the best on-device models usually combine several. Distillation trains a smaller "student" model to imitate a larger "teacher", producing a model with fewer weights that captures much of the larger one’s capability — attacking the other factor in the size equation, the number of weights, rather than the bits per weight. Pruning removes weights that contribute little, trimming the model’s structure directly. Each technique reduces size in a different way, and they stack: a model can be distilled to fewer weights, pruned of the redundant ones, and then quantized to fewer bits, with the reductions multiplying together.

The reason this matters is that the dramatic shrinkage that makes a genuinely capable model fit in a browser is rarely one technique alone; it is the compound effect of several, with quantization typically doing the largest share. Understanding quantization as the headline act, supported by distillation and pruning, gives an accurate picture of how an AI model that would be impractical at full size becomes something that downloads in seconds and runs on a phone. The engineering goal across all of them is the same one this whole article circles: maximum capability per byte, because in the browser the byte budget is the binding constraint and every technique that buys capability within it is worth combining.

It is worth connecting the technique back to why any of it matters, because quantization is not an end in itself — it is the enabler of a particular promise. The entire case for on-device AI, made in /product-blog/why-on-device-ai-is-private-by-design, rests on the model being small enough to run where the data already is, so that nothing has to be uploaded to a server to be processed. That promise is impossible without the shrinking that quantization provides; a model that only runs in a data centre forces the data to come to it, which is exactly the privacy compromise on-device AI exists to avoid. Quantization, in other words, is quietly a privacy technology as much as a performance one.

So the next time an in-browser tool removes a background, restores a photo, or transcribes audio entirely on your own machine, it is worth knowing that a large part of how that is even possible is the unglamorous work of storing billions of numbers in fewer bits each. Quantization takes a model that would never fit and rounds it, carefully, down to something that does — losing a little precision the model could spare and gaining the size and speed that make local AI real. It is one of those techniques that is simple at its core, surprisingly effective in practice, and almost invisible in the result, which is exactly why it is worth understanding: it is a big part of the answer to how the data can stay on your device and the AI can still work.