Model Quantization: Running LLMs on Your Laptop
Updated: 3 days ago

UIT University 365 Institute of Technology
Series AI Engineering
Duration 25 minutes
IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.
[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]
In this Lecture
The Hook: A Model That Fits in Your Bag
A 70-billion-parameter language model, stored in its original training format, needs about 281 GB on disk. That is a server.
The same model, converted to a 4-bit format, needs about 43 GB. That fits on a laptop with a decent solid-state drive, and it runs on the laptop's own memory, offline, with no API key and no per-token bill.
Nothing about the model's architecture changed. It still has 70 billion parameters, the same layers, the same attention heads, the same weights in the same places. What changed is how precisely each weight is written down.
That is quantization: storing the same numbers with fewer bits. This lecture shows you what that costs, what it buys, and how to choose a format using measured evidence rather than folklore.
Step 1: What a Model Actually Stores
Before you can compress a model, you need to know what is in it.
The parameters
A language model is a very large collection of numbers called parameters or weights. For Llama 3 8B that is roughly 8 billion numbers. For Llama 3 70B, roughly 70 billion. The number after the "B" is the parameter count, and it is the main determinant of how much memory the model needs.
The default storage format
Weights are trained and usually stored in 16-bit floating point, written as FP16 or BF16. Each parameter takes 2 bytes.
Do the arithmetic:
8 billion parameters × 2 bytes = 16 GB
70 billion × 2 bytes = 140 GB
405 billion × 2 bytes = 810 GB
That is the baseline. Everything below is about writing those same parameters in fewer bytes.
Why precision was needed in training
Training is sensitive. Gradients are small, they accumulate over millions of steps, and rounding errors compound. Sixteen bits, and in some places 32 bits for the master copy of the weights, is what makes training stable.
Inference is not like that. At inference time each weight is used once per token to do a multiply-and-add. The model does not accumulate error across steps in the same way. That asymmetry is the opening quantization walks through.
The three things you have to store
Memory for a running model is more than the weights:
Weights, the parameters themselves.
The KV cache, the stored key and value vectors for every token already generated.
Activations and working buffers, temporary values during the forward pass.
Quantization mostly attacks the first, and there are separate techniques for the second. Step 7 covers the split.
Step 2: What Quantization Does to a Number
A floating-point number versus a fixed-point number
An FP16 value spends 1 bit on the sign, 5 bits on the exponent and 10 bits on the significand. The exponent gives it a huge dynamic range: it can represent very large and very small numbers.
A 4-bit integer has 16 possible values. It cannot represent a wide range.
The trick is that you do not need a wide range for any single weight. You need a wide range across all the weights, and each weight only needs to be accurate relative to its neighbours.
The block-and-scale idea
Modern quantization splits the weight matrix into blocks, typically 32 or 256 weights each, and stores one scale and one zero point per block. Every weight inside the block is stored as a small integer, and its real value is recovered as:
real value ≈ scale × (stored integer − zero point)
A block of 32 weights at 4 bits each takes 128 bits. Add the scale (16 bits) and you have 144 bits for 32 values, which is 4.5 bits per weight. That extra half-bit is the cost of the scale, and it is why a format labelled 4-bit is often closer to 4.5 bits in practice.
Where the error comes from
Two things decide whether quantized weights still work:
The scale granularity. Smaller blocks mean a scale that better fits the local range of values, which reduces error. They also cost more overhead.
Which weights get more precision. Not all weights matter equally. The output projection and the attention projections tend to be more sensitive than the feed-forward layers. This is why the "K" formats use mixed precision inside one file: some tensors at higher bit width, some at lower.
Calibration
A second lever is calibration. Instead of quantizing each block against a naive range, you run a small amount of representative text through the original model, record which weights have the largest effect on the output, and use those statistics to choose the scales. In llama.cpp this produces an importance matrix (imatrix), and it measurably reduces the perplexity penalty at low bit widths.
The practical summary: quantization is not one operation, it is a family of choices about block size, mixed precision and calibration.

Step 3: The GGUF Formats, Decoded
GGUF is the file format used by llama.cpp and most local inference tools. A GGUF file is a single container holding the weights, the tokenizer, the architecture metadata and the chat template.
Within GGUF there are three generations of quantization schemes, and the naming tells you which one you have.
Legacy block quants
Q4_0, Q4_1, Q5_0, Q5_1, Q8_0.
Fixed block sizes, simple scaling, no mixed precision, no calibration. Old and fast to produce. Still useful as a baseline.
K-quants
Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K.
The "K" means k-quant. These use a larger super-block structure with two levels of scaling, and mixed precision across tensor types. The suffix is a size tier within the same bit width:
_S small, the most aggressive compression in that tier
_M medium, the usual default
_L large, the largest and highest quality in that tier
So Q4_K_M and Q4_K_S are both roughly 4-bit, but _M keeps a little more precision in the sensitive tensors and costs a little more disk.
I-quants
IQ1_S through IQ4_NL.
Importance-matrix quants. They depend on a calibration file, use a codebook rather than a simple scale, and reach very low bit widths. They are the smallest usable files, and they are the most sensitive to how well the calibration matched your use case.
The names you will actually type
# quantize an FP16 GGUF to 4-bit medium ./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M # with calibration statistics for a better result ./llama-quantize --imatrix model.imatrix model-f16.gguf model-Q4_K_M.gguf Q4_K_M
The number to watch
Every format has a bits per weight figure, and it is higher than its label:
Format | Bits per weight (7B-class) | Size (GiB) |
Q2_K | 2.96 | 2.8 |
Q3_K_M | 4.00 | 3.7 |
Q4_K_M | 4.89 | 4.6 |
Q5_K_M | 5.70 | 5.3 |
Q6_K | 6.56 | 6.1 |
Q8_0 | 8.50 | 8.0 |
F16 | 16.00 | 15.0 |
The gap between the label and the real figure is the scale and metadata overhead. A "4-bit" file is a 4.9-bit file. Plan your disk and memory against the second column, not the first.
Step 4: The Size and Quality Table
Here is what the compression buys you, in the numbers the tooling itself publishes.
File size, by model
Model | Original size | Q4_K_M size |
8B | 32.1 GB | 4.9 GB |
70B | 280.9 GB | 43.1 GB |
405B | 1,625.1 GB | 249.1 GB |
The 8B row is roughly a 6.5x reduction. The 70B row turns a server requirement into a workstation requirement. The 405B row stays out of reach for a laptop either way.
Quality, by format
Perplexity measures how surprised a model is by real text. Lower is better. These deltas are measured against a 7B-class baseline:
Format | Perplexity change | Read it as |
Q8_0 | +0.003 | indistinguishable |
Q6_K | +0.02 | indistinguishable in practice |
Q5_K_M | +0.06 | very small |
Q4_K_M | +0.18 | small, usually acceptable |
Q3_K_M | +0.66 | noticeable |
Q2_K | +3.52 | substantial |
The shape of that table is the whole argument. Going from 16 bits to 4 bits saves about 70% of the file size and costs a fraction of a perplexity point. Going from 4 bits to 2 bits saves a further 40% of the remaining size and costs twenty times the quality.
Speed, and the surprise
On a CPU reference benchmark for a 7B-class model, generation speed at 128 tokens was:
Format | Tokens per second |
F16 | 29.2 |
Q8_0 | 50.9 |
Q6_K | 58.7 |
Q4_K_M | 71.9 |
Q2_K | 79.9 |
Smaller files are also faster on CPU, because the bottleneck is memory bandwidth rather than arithmetic. Reading 4.6 GB per pass is quicker than reading 15 GB. Quantization is not purely a quality sacrifice; in the CPU regime it is often a straight win.

Step 5: Why Perplexity Is Not Enough
Perplexity is a proxy. It averages over a whole corpus, and it does not tell you how a specific capability degrades.
A 2026 study evaluated thirteen GGUF configurations of one instruction-tuned 8B model on five downstream benchmarks rather than perplexity alone. The results show the problem clearly.
Configuration | Size reduction | GSM8K (math) | Average score | Perplexity |
F16 baseline | none | 77.63 | 69.47 | 7.32 |
Q4_K_M | 69.4% | 77.41 | 69.15 | 7.56 |
Q3_K_M | 75.0% | 73.16 | 68.07 | 7.96 |
Q3_K_S | 77.2% | 68.31 | 65.49 | 8.96 |
Read the top two rows together. At 4-bit medium, the model loses about 0.3% of its average benchmark score and effectively nothing on multi-step arithmetic, while the file shrinks by nearly 70%.
Then read further down. Q3_K_S saves only eight more percentage points of size than Q3_K_M, and loses nine points of arithmetic reasoning to do it. That is a bad trade for any task that involves calculation.
Three findings from that study are worth carrying into your own work:
Degradation is task-dependent, not uniform. Multi-step arithmetic and instruction-following are the most sensitive. Knowledge questions and commonsense questions degrade less.
Format matters as much as bit width. Two formats with almost the same measured perplexity can differ meaningfully on arithmetic and instruction-following, because the quantization error interacts with the computation pattern the task requires.
A well-calibrated 5-bit format can match or slightly beat the 16-bit baseline on some benchmarks. The study reported several 5-bit configurations scoring above F16 on the arithmetic benchmark. Quantization is not monotonically destructive; it is a perturbation that can go either way on a given metric.
The practical rule that follows: measure on your own task, not on a leaderboard. A model that scores well on general knowledge can still fail on the arithmetic your application depends on.
Step 6: Your First Local Model, Step by Step
This is a concrete sequence you can run today on a laptop with 16 GB of memory.
1. Check what you have
Memory: at least 8 GB free, 16 GB comfortable. A 7B model at Q4_K_M needs about 4.6 GiB for weights plus context.
Disk: at least 10 GB free.
CPU: any modern laptop will run a 7B model and give readable output. Tokens per second will be modest.
GPU: optional. A discrete GPU with 8 GB of video memory can hold a 4-bit 7B model entirely.
2. Get a model in GGUF format
Prefer a GGUF file from a publisher you trust, or convert and quantize yourself from the original weights. The published reference tables use a 7B-class dense model, so a matching file is the easiest way to see the numbers in this lecture reproduce.
3. Pick a format
Start at Q4_K_M. It is the balance point the tooling itself recommends, and Table 4 above shows why: most of the quality, about 30% of the size.
Move to Q5_K_M if you have the memory and your task is arithmetic-heavy. Move to Q3_K_M only if you are memory-constrained and have tested your task.
4. Run it
llama-cli -m model-Q4_K_M.gguf -p "Explain what a KV cache is in two sentences"
5. Find the real limit
Increase the context length until it fails. Watch memory as you go. The point at which it fails is your KV cache limit, and it is usually much smaller than the model's advertised maximum context.
6. Measure, do not guess
llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -ngl 99 -r 5 -o json
Record prompt-processing speed and generation speed. Then compare two formats on the same hardware and the same prompt. You now have your own data instead of someone else's table.
Step 7: What Else Eats Your Memory
Quantizing the weights gets the model loaded. It does not get it running well at long context. Three other costs matter.
The KV cache
Every generated token adds a key vector and a value vector for every layer. The cache grows linearly with the number of tokens in context.
For a model with 8,192 hidden dimensions and 128,000 tokens of context, the cache can run to several gigabytes on its own. This is why a 4-bit model that fits comfortably also runs out of memory at 32,000 tokens of context.
The controls are:
KV cache quantization. Store cached keys and values at 8-bit or 4-bit instead of 16-bit. Halves or quarters the cache. Quality impact grows with context length, so test it.
Sliding-window attention. Keep only the most recent tokens in the cache, with a summary of what was dropped.
Cache eviction. Drop the least useful stored tokens by an attention-based score.
Shorter contexts. The cheapest fix and the one people skip.
The runtime overhead
The inference engine, the tokenizer, the chat template and the working buffers all take memory. Budget a gigabyte or so of headroom above the weights.
The context-window claim versus the context-window reality
A model card may state a 128,000-token context. The memory to hold it is a separate question from the training that supported it. Always test at the context length you actually intend to use.

Step 8: Choosing a Format for Your Hardware
A decision table is more useful here than a rule.
Your situation | Format | Reasoning |
16 GB laptop, general chat and writing | Q4_K_M | the balance point, about 4.6 GiB for a 7B model |
16 GB laptop, arithmetic or code tasks | Q5_K_M | arithmetic is the most sensitive capability; the extra 0.8 GiB is cheap |
8 GB laptop or a small tablet | Q3_K_M | the safest 3-bit choice; accept a visible quality drop |
Disk or memory is the hard constraint | Q2_K or an IQ2 format | usable only when nothing else fits |
Quality must be provably preserved | Q6_K or Q8_0 | near-baseline perplexity, larger files |
Serving many users from a GPU | A GPU-native format | GGUF targets CPU and consumer hardware; server stacks use their own schemes |
Two principles sit behind the table.
First, decide from your task backwards. If your application does arithmetic, instruction-following or structured output, you are in the sensitive region and 4-bit medium is your floor. If it does summarisation and drafting, you have more room.
Second, calibrate when you compress hard. At 4-bit and above, calibration is a refinement. At 3-bit and below it is close to mandatory. An imatrix built from text that resembles your workload is the difference between a usable 3-bit model and a useless one.
Step 9: The Limits of the Laptop
Quantization changes the economics of running a model. It does not change what the model knows.
What local inference gives you
Privacy by construction. No prompt leaves the machine. For regulated data this is often the whole argument.
No per-token cost. Marginal cost becomes electricity.
No rate limit and no deprecation. The file you have keeps working.
Full control of the sampling. Temperature, top-p, stop sequences and the system prompt are yours to set.
What it costs you
A quality gap you did not choose. A frontier hosted model and a 4-bit 8B model on your laptop are not comparable tools. Be honest about which tasks the small model can carry.
A context ceiling. The long-context work you do in a hosted model usually does not survive the move.
Operations you now own. Model updates, prompt templates, evaluation and monitoring all become your job.
Slower iteration. Generation speed on a CPU is measured in tens of tokens per second, not hundreds.
Where the local model wins
The pattern that works is a division of labour: use the local model for the high-volume, low-stakes, private work, such as classifying, extracting fields, rewriting, and first-pass drafting, and use a hosted frontier model for the low-volume, high-stakes work, such as architecture decisions and difficult reasoning. You keep the privacy and cost benefits where they matter and you spend the frontier model where it earns its price.
This is the CI-First position applied to hardware. The human decides which tool carries which task, and the decision is made on measured quality, not on enthusiasm for either option.
Step 10: Quantizing the Cache, Not Just the Weights
Everything so far has been about the weights, and the weights are a fixed cost. You pay for them once, when the model loads. The cache is different: it grows with every token you put in context, and on a long session it can outgrow the weights themselves. That is the failure people actually hit, so it deserves its own step.
The cache formula
Every generated token adds one key vector and one value vector for every layer. The storage per token is:
2 x layers x kv_heads x head_dim x bytes_per_element
Take a model with 32 layers, 8 key and value heads, and a head dimension of 128. That gives 2 x 32 x 8 x 128 = 65,536 values for every token in context. At 16-bit storage that is 131,072 bytes, or 128 KiB per token. Multiply it out:
Context length | Cache at 16-bit | Cache at 8-bit | Cache at 4-bit |
8,192 tokens | 1 GiB | 0.5 GiB | 0.25 GiB |
32,768 tokens | 4 GiB | 2 GiB | 1 GiB |
131,072 tokens | 16 GiB | 8 GiB | 4 GiB |
Now add the 4.6 GiB of a 4-bit 8B model. At 32,000 tokens the cache is most of your budget. At 128,000 tokens it is four times the size of the weights. This is why a model that loads in two seconds can still die halfway through a long document, and why the context number on the model card is not a promise about your laptop.
What a quantized cache buys
The block-and-scale idea from Step 2 applies here too, with one difference. The weights are static: they are known before the model runs, and the scales can be chosen carefully offline. The cache is computed on the fly from activations, which carry outliers, so the same bit width means more error here than it does on a weight matrix.
The effect on memory is direct: 8-bit halves the cache, 4-bit quarters it. The effect on speed is worth as much. Long-context decoding is memory-bandwidth bound, the same as weight reading, so moving 1 GiB per step instead of 4 GiB is faster as well as smaller.
Keys and values are quantized separately, and in practice keys are the more sensitive of the two, because attention scores are computed from them directly. Many builds let you set a different type for each, which is the shape to prefer if you are chasing memory.
Turning it on
In llama.cpp the cache types are set at load time, not at compile time, through the --cache-type-k and --cache-type-v options. An 8-bit cache for both is the usual first move:
llama-server -m model-Q4_K_M.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0
Flag names in this project move between releases, so run the help output of your own build before you commit a command to a script. Once the cache is quantized, the setting applies to the sessions you start afterwards.
When it hurts
Three cases where a quantized cache costs more than it saves:
Long-context retrieval. When the task is finding one precise fact in a 60,000-token document, the model is matching a query against keys. Rounding the keys makes that matching less exact. Test with the real document, not a chat prompt.
Short prompts. At 2,000 tokens the whole cache is a few tens of megabytes. There is nothing to gain and one more variable to explain when something goes wrong.
Very long generation with a small context. The cache is bounded by context length, not by how long the session runs, so the number to watch is the context you set, not the tokens you generate.
The measurement is cheap and it is the same one you will use for the weights: run your most context-heavy question twice, once with an unquantized cache and once with the quantized one, and compare the answers rather than the memory readout. If the answer survives, keep the setting.
A budget you can write down
The point of the arithmetic is to have a plan before you download anything. Here is the same plan for two machines.
Line item | 16 GB laptop | 32 GB workstation |
Weights, 8B at Q4_K_M | 4.6 GiB | 4.6 GiB |
Runtime and working buffers | 1.0 GiB | 1.5 GiB |
Cache at 32k, 8-bit | 2.0 GiB | 2.0 GiB |
Cache at 32k, 16-bit | 4.0 GiB | 4.0 GiB |
Total, 8-bit cache | about 7.6 GiB | about 8.1 GiB |
Total, 16-bit cache | about 9.6 GiB | about 10.1 GiB |
Both machines fit this model. Only one of them fits it with 64,000 tokens of context and room to spare, and the difference is decided by the two cache lines, not by the weights line.

Step 11: Mixture of Experts Changes the Memory Math
A mixture-of-experts model splits each feed-forward block into many separate expert networks and routes each token through a small number of them. That design produces two parameter counts instead of one, and both of them matter.
Two numbers, not one
Take a published model of this kind, with 46.7 billion parameters in total and about 12.9 billion of them active for any single token. Memory follows the total, because every weight has to be somewhere the model can read it. Speed follows the active count, because that is the work actually performed per token.
That single split explains almost every surprise on this class of model. A file that is three times the size of a dense 8B model can still answer faster on a CPU, and a machine with plenty of disk and no memory still cannot run it at all.
What fits where
Apply the bits-per-weight figure from Step 3. At 4.89 bits per weight, 46.7 billion parameters come to about 28.5 GB, and the published four-bit files for this model land in the region of 26 to 29 GB depending on the exact release.
Machine | 46.7B MoE at 4-bit | Dense 8B at 4-bit |
8 GB memory | does not fit | fits, about 4.6 GiB |
16 GB memory | does not fit | fits with room for a long context |
32 GB memory | fits at a short context with an 8-bit cache | fits comfortably at a long context |
64 GB memory | comfortable, long context | not the limiting factor |
The comparison is the decision. The larger total brings knowledge and reasoning that an 8B dense model does not have, and it costs three to four times the memory. For drafting and extraction the smaller model wins on every axis except quality. For a hard reasoning task with a documented value behind it, the larger one earns its memory.
Partial offload works differently here
On a dense model, moving work to the GPU means dividing the layers, and the more layers you offload the better it runs. On a mixture-of-experts model that arithmetic changes, because only a few experts are active per token. The attention blocks and the shared layers are used by every token, while the expert tensors sit idle most of the time. The practical technique is to keep the always-used parts on the GPU and let the expert tensors stay in system memory.
The effect is that a modest GPU becomes useful. A laptop with 8 GB of video memory and 32 GB of system memory can serve a model that will never fit in the video memory alone, because the part that must be fast is small and the part that must merely be present is large. The flag that assigns expert tensors to the CPU has changed name between releases, so read the help output of your own build rather than copying a command from an older guide.
A draft model as an accelerator
There is a second way to make a large model faster without touching its weights. Speculative decoding pairs a small draft model with the large target model. The draft proposes several tokens; the target checks them in a single pass and keeps the longest prefix it agrees with. Under standard speculative sampling the output distribution is the target model's own, so this is a speed technique, not a quality trade.
It works when the draft is right often enough to pay for the extra work. Pair a small draft with a larger target from the same family and the same tokenizer, so that both speak the same token vocabulary. If the acceptance rate is low, you have added overhead and removed nothing, so measure the tokens per second before you keep the pairing.
A download plan for three machines
Machine | Model class | Format | Weight file | What it is for |
8 GB | 3B dense | Q4_K_M | about 2 GiB | chat, classification, field extraction |
16 GB | 8B dense | Q4_K_M | 4.6 GiB | the general balance point |
16 GB | 8B dense | Q5_K_M | 5.3 GiB | arithmetic and structured output |
32 GB | 14B dense | Q5_K_M | about 9 GiB | quality with a long context |
32 GB | 46.7B MoE | Q4_K_M | 26 to 29 GB | hard reasoning at a short context |
The rule
Four numbers decide whether a model runs on your machine, and they are decided in this order:
Total parameters decide whether the file loads at all.
Active parameters decide how fast it answers.
Bits per weight decide the file size you download.
Cache type and context length decide whether it survives a long session.
Write those four down before you start a download, and you will stop discovering your machine's limits halfway through a task.
Feynman Summary: Explain It Like You Are 12
Imagine you have a very long shopping list with millions of prices written with lots of decimal places, like 12.4938372 euros.
You need to carry that list in a small notebook. So you round every price to the nearest whole euro. The list is now much smaller, and it prints much faster, and most of the time you still know what things cost.
Rounding one price is a tiny mistake. Rounding millions of them adds up, but not as much as you would think, because the mistakes go in both directions and mostly cancel out.
Now imagine going further: rounding to the nearest ten euros. The notebook gets even smaller, but now a 14-euro item and a 6-euro item both say 10, and some of your totals come out wrong. That is the difference between 4-bit and 2-bit quantization. A small squeeze is nearly free. A big squeeze breaks the arithmetic.
The clever part of modern quantization is that it does not round the whole list the same way. It looks at small groups of prices together, and it spends more precision on the items that change the total the most.
Mindmap: The Complete Picture

The mindmap shows the full structure of what you learned: what a model stores, where the memory goes, how block-and-scale quantization works, the three families of GGUF format, the size and quality evidence, the task-dependent degradation, the four steps to running a local model, and the KV cache cost that limits it.

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.
[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]
Practical Exercise: Quantize and Measure
Objective
Measure the quality and speed cost of quantization on your own hardware, using two formats of the same model.
Steps
Download or produce two GGUF files of the same model: one Q8_0 and one Q4_K_M.
Write five prompts that represent your real workload. Include at least one that requires arithmetic, one that requires following a strict output format, and one that requires recalling a fact.
Run all five prompts against the Q8_0 file and save the outputs verbatim.
Run the same five prompts against the Q4_K_M file and save the outputs verbatim.
Score each pair yourself, blind if possible: same quality, slightly worse, clearly worse.
Benchmark both files:
llama-bench -m model-Q8_0.gguf -p 512 -n 128 -r 5 -o json llama-bench -m model-Q4_K_M.gguf -p 512 -n 128 -r 5 -o json
Record four numbers: file size each, generation speed each, and the count of prompts that degraded.
Now raise the context length in a conversation until memory fails, for each file. Record both limits.
What to Look For
Expect roughly a 2x file-size difference and a visible speed difference in favour of the smaller file.
Expect the arithmetic prompt to degrade first, if any does.
Expect the context limit to be far below the model's advertised maximum, and expect it to be similar for both files, because it is dominated by the KV cache rather than the weights.
If nothing degraded on your five prompts, your task sits in the insensitive region, and you can compress further with confidence.
Glossary
Term | Definition |
Parameter | A single learned number in a model. A "7B model" has about 7 billion of them. |
FP16 / BF16 | 16-bit floating-point storage formats, the usual baseline for trained weights. Each parameter takes 2 bytes. |
Quantization | Storing weights at lower precision than they were trained in, to reduce size and memory use. |
Block | A group of consecutive weights that share one scale and one zero point. |
Scale and zero point | The two numbers that let a small stored integer be converted back to an approximate real value. |
Bits per weight | The real storage cost per parameter, including scales and metadata. Always higher than the format's label. |
GGUF | A single-file model format carrying weights, tokenizer, architecture metadata and chat template, used by llama.cpp and most local inference tools. |
K-quant | A llama.cpp format family using two-level block scaling and mixed precision across tensor types. The `_S`, `_M` and `_L` suffixes are size tiers. |
I-quant (IQ) | An importance-matrix-dependent format family using a codebook, reaching the lowest bit widths. |
Importance matrix (imatrix) | Calibration statistics, gathered by running representative text through the model, that guide the choice of quantization scales. |
Perplexity | A measure of how surprised a model is by real text. Lower is better. Used as a proxy for quantization quality. |
KV cache | Stored key and value vectors for already-generated tokens, so they are not recomputed. Grows linearly with context length. |
KV cache quantization | Storing cached key and value vectors at 8-bit or 4-bit to reduce memory during long-context generation. |
Memory bandwidth bound | A regime where generation speed is limited by how fast weights can be read from memory rather than by arithmetic. This is why smaller files can be faster. |
Downstream benchmark | A task-based evaluation, such as arithmetic or instruction-following, as opposed to a distribution-based proxy such as perplexity. |
CI-First | The U365 principle that the human is the ruler and orchestrator, and AI is the amplifier. |
KV cache size per token | The memory one token of context adds to the cache: 2 x layers x kv_heads x head_dim values. Determines how fast the cache grows with context length. |
Quantized KV cache | Storing cached keys and values at 8-bit or 4-bit instead of 16-bit. Halves or quarters the cache, with more error than the same bit width on weights because activations carry outliers. |
Mixture of experts (MoE) | A model design that splits each feed-forward block into many expert networks and routes each token through a few of them. Total parameters set the memory need; active parameters set the speed. |
Total parameters against active parameters | For a mixture-of-experts model, the count that must be stored against the count used for one token. A 46.7B total model with 12.9B active needs memory for 46.7B and computes like a 12.9B model. |
Speculative decoding | Pairing a small draft model with a larger target model. The draft proposes tokens and the target verifies them in one pass. Speed at no change to the output distribution under standard speculative sampling. |
Quiz: TEST YOUR UNDERSTANDING
1. A 70-billion-parameter model in FP16 needs about 140 GB. What is the main reason a 4-bit version needs roughly 43 GB?
A) The 4-bit version has fewer parameters
B) Each parameter is stored in fewer bits, with a small overhead for scale and metadata
C) The 4-bit version removes some layers
D) The tokenizer is compressed
2. Why is a format labelled "4-bit" often closer to 4.9 bits per weight in practice?
A) Because 4-bit means 4.9 by definition
B) Because each block needs a scale and a zero point, and some tensors are kept at higher precision
C) Because the tokenizer is counted in the average
D) Because of file-system block size
3. Which capability degrades first when you compress a model aggressively?
A) General knowledge recall
B) Commonsense reasoning
C) Multi-step arithmetic reasoning
D) Tokenizer coverage
4. Why can a smaller quantized file generate tokens FASTER on a CPU?
A) Fewer parameters means less arithmetic
B) CPU generation is memory-bandwidth bound, and a smaller file reads faster
C) Quantized weights use a faster instruction set
D) The KV cache is disabled
5. You have 16 GB of memory, you run mostly arithmetic and code tasks, and you want the highest practical quality. Which format?
A) Q2_K
B) Q3_K_S
C) Q4_K_M
D) Q5_K_M or Q6_K, if the file fits
6. A model has 32 layers, 8 key and value heads, a head dimension of 128, and you store the cache at 16-bit. About how much cache does each token of context add?
A) 4 KiB
B) 32 KiB
C) 128 KiB
D) 1 MiB
7. Why does the same bit width cost more accuracy in the cache than in the weights?
A) The cache is stored in a different file format
B) Cache values are computed on the fly from activations, which carry outliers
C) The cache is not quantizable at all
D) The tokenizer recomputes the cache
8. A model lists 46.7 billion total parameters and 12.9 billion active parameters. What does each number tell you?
A) Total sets the file and memory need; active sets the work per token
B) Both describe file size only
C) Total sets the speed; active sets the disk use
D) Neither matters on a laptop
9. Under standard speculative sampling, what happens to the output distribution when a draft model proposes tokens?
A) It shifts toward the draft model's style
B) It is unchanged, because the target model verifies each proposal
C) It becomes deterministic
D) It loses long-range coherence
Answers: 1-B, 2-B, 3-C, 4-B, 5-D, 6-C, 7-B, 8-A, 9-B.
Related Resources
U365 INSIDE Publications
How LLMs Actually Work: Transformers in 20 Minutes: the architecture you are quantizing
AI Code Generation: Copilot, Cursor, and Beyond: the tooling layer above the model
External Resources
llama.cpp quantization tool documentation: the GGUF format list, sizes and perplexity deltas: github.com/ggml-org/llama.cpp/tree/master/tools/quantize
Which Quantization Should I Use? (2026): a unified downstream evaluation of llama.cpp GGUF formats: arxiv.org/abs/2601.14277
llama.cpp (ggml-org): the reference local inference engine: github.com/ggml-org/llama.cpp
Llama 3 herd of models (Meta, 2024): architecture and training details: arxiv.org/abs/2407.21783
Related U365 Lectures (Coming Soon)
Lecture 8: AI Safety and Alignment: Why Hallucinations Happen (UIT, AI Foundations)
U.Copilot for This Lecture
Discuss this lecture with U.Copilot, your AI chat companion trained on this content.
Copy and paste the following prompt into the U.Copilot chat on university-365.com:
You are U.Copilot for Lectures, an AI chat companion trained on University 365 lecture content. You are helping a Fellow who just completed the lecture "Model Quantization: Running LLMs on Your Laptop" from the AI Engineering series at the U365 Institute of Technology (UIT). Your role is to help the Fellow run a local model well. You can: - Explain block-and-scale quantization, bits per weight, and why the label understates the real cost - Compare GGUF format families: legacy quants, K-quants with their _S, _M and _L tiers, and I-quants - Walk through the size, perplexity and speed tables in the lecture and what each implies - Explain why degradation is task-dependent, and why perplexity is not a sufficient predictor - Help the Fellow choose a format from a described machine: available memory, disk, GPU, and the task - Explain the KV cache and why context length, not weights, is usually the binding limit Always maintain the U365 CI-First approach: encourage the Fellow to measure on their own hardware and task rather than trusting a general recommendation. Use the UP-Context-Engineering Method: ask about the Fellow's machine, memory and intended task before recommending a format.
Next Steps
Now that you understand what quantization costs and buys, here is what to do next:
Run the practical exercise with Q8_0 and Q4_K_M of the same model, and keep your four numbers.
Test your real context requirement by growing a conversation until memory fails, rather than trusting the model card.
If you compress below 4-bit, build an importance matrix from text that resembles your workload.
Write down which of your tasks the local model can carry and which must stay on a hosted model.
Take the next lecture in this series, "AI Safety and Alignment: Why Hallucinations Happen", to understand why even a correctly quantized model still produces confident errors.
Local inference is a control decision before it is a cost decision. Once you can run the model yourself, you decide what leaves your machine.
IMPORTANT NOTICE
This lecture is published by University 365 as part of its INSIDE Publications Hub. The content is free to read for all visitors. Lectures in this series may be part of a structured academic program leading to a Micro-Credential for your Career (MCC). To enroll in an academic program, visit university-365.com/tuition.
This content is for educational purposes. While we strive for accuracy, AI is a fast-moving field. Benchmark figures move with every release; verify current numbers against the primary sources for professional applications.
Copyright University 365, Inc. All rights reserved. This content is protected under University 365's copyright policies. For permissions or inquiries, contact uda@university-365.com.
Published by the Department of Academics, University 365.
Lecture delivered by the University 365 Institute of Technology (UIT).
Sam Utteker, Dean of Technology, UIT
Signed for the academic year 2026.









Comments