GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8_0 vs IQ4_XS, What the Letters Mean, Which Quant Fits Your RAM, and Why Small Models Break the Rules

GGUF quantization stores a model's weights in fewer bits so it fits in your RAM or VRAM. The name tells you how. In Q4_K_M, "Q4" means about 4 bits per weight, "K" means the k-quant method (blocks of 256 weights with their own scales), and "M" means a medium mix that keeps some sensitive tensors at higher precision. For most people, Q4_K_M is the default download. On Llama 3 8B, llama.cpp's own table puts it at 4.58 GB, against 16 GB at 16-bit, with a perplexity cost of +0.18. Q5_K_M (+0.06) and Q6_K (+0.02) are visibly closer to the original, Q8_0 (+0.003) is practically lossless, and Q2_K (+3.52) loses twenty times more than Q4_K_M. The part almost nobody explains is that Q4_K_M is not really 4-bit. It averages 4.89 bits per weight on Llama 3.1 8B. On a small model it can be far more: quantizing the 135-million-parameter SmolLM2 to Q4_K_M produced a file at 6.17 bits per weight, because 180 of its 272 tensors had rows of 576 values, which is not a multiple of the 256-value blocks k-quants need, and fell back to bigger formats. This page explains every quant type you will see on Hugging Face (Q, K, IQ, UD, _S/_M/_L/_XL, imatrix), compares them head to head, shows real measurements from quantizing one model nine ways, and tells you exactly which file to download for your RAM.

Ethan was setting up a local model for the shop and opened its download page: twenty-two files, from 11 GB to 80 GB, named things like UD-IQ2_XXS and Q5_K_XL. "Which one do we want?" Jake asked. "The biggest one that fits," Ethan said, "minus room for the conversation." That is the short answer. Here is the long one, which is how you know what "fits" means and what you lose at each step down.

New to local AI? GGUF is the file format used by llama.cpp and the apps built on it: Ollama, LM Studio, Jan, Unsloth and others. A model's "weights" are billions of numbers. Stored at full 16-bit precision, an 8-billion-parameter model needs about 16 GB. Quantization stores each number in fewer bits, accepting a small loss of accuracy in exchange for a much smaller file that runs faster. Our run AI locally series covers the basics of running models, and our app comparison helps you pick where to run them. This page is about choosing the file.

What GGUF quantization does, in plain English

Imagine a list of prices written to the cent: $12.47, $3.99, $105.12. If you round them to the nearest dollar, the list takes less space and is still useful, but some detail is gone. Quantization does that to model weights. Instead of storing each weight as a 16-bit number, it stores it as, say, a 4-bit number plus shared information (a scale and offset) for each small group of weights, so the rounded values land as close as possible to the originals.

Two things make this work far better than plain rounding. First, weights are quantized in blocks, typically 32 or 256 values, each with its own scale, so a block of tiny weights and a block of large ones are each rounded on their own terms. Second, not all tensors are equal. Some (like parts of attention, or the output layer) hurt quality much more when squeezed, so good quantization schemes give those a few extra bits and squeeze the rest harder. That mix is what the _S, _M and _L suffixes describe.

The result: a model at 4 to 5 bits per weight keeps most of its ability at a third of the size, and runs faster too, because generating each token means reading every weight from memory, and fewer bytes means less reading. On Llama 3.1 8B, text generation runs about 2.5 times faster at Q4_K_M (72 tokens per second) than at F16 (29 tokens per second) on the same hardware, in the llama.cpp project's benchmark table.

How to read a quant name: Q4_K_M, IQ3_XXS, UD-Q4_K_XL and friends

• The number (Q2 to Q8) is the approximate bits per weight for most tensors. Q4 is about 4 bits, Q8 about 8. The true average is higher, because of block scales and higher-precision tensors.
• Q4_0, Q4_1, Q5_0, Q5_1, Q8_0 are the legacy quants: simple blocks of 32 weights with a scale (_0) or a scale and minimum (_1). Q8_0 is still excellent. The 4- and 5-bit legacy types are mostly superseded by k-quants, but they are fast on some hardware.
• K (Q2_K to Q6_K) marks the k-quants: super-blocks of 256 weights with smaller sub-blocks and their own scales, which give better quality per bit.
• _S, _M, _L (small, medium, large) are k-quant mixes: how many sensitive tensors get bumped to a higher type. Q4_K_S is the leanest mix, and Q4_K_M upgrades more tensors, at a little more size and noticeably better quality. When a type has no suffix (Q4_K), it is an alias for _M.
• IQ (IQ1_S to IQ4_XS) marks the i-quants: more complex encodings that squeeze more quality out of very low bit counts. The lowest ones (IQ1, IQ2) require an importance matrix to create. _XXS, _XS, _S, _M again describe size within the family, and _NL means "non-linear".
• UD- (as in UD-Q4_K_XL) is Unsloth Dynamic: Unsloth's own per-layer choice of types, calibrated with an importance matrix. _XL suffixes are Unsloth's labels for its larger mixes, not a standard llama.cpp type.
• F16, BF16, F32 are unquantized, at 16 or 32 bits. BF16 is the usual "original" GGUF.
• MXFP4, NVFP4 are 4-bit floating-point formats. MXFP4 is used for some MoE models, and NVFP4 targets recent NVIDIA GPUs.

Every quant type compared: size and quality cost

This is llama.cpp's own reference table (build b11827), shown by llama-quantize --help. Sizes are for an 8-billion-parameter Llama 3 model, and the quality cost is the increase in perplexity (lower is better, and +0.00 would be identical to the original):

The i-quants are listed by bits per weight instead: IQ1_S 1.56, IQ1_M 1.75, IQ2_XXS 2.06, IQ2_XS 2.31, IQ2_S 2.5, IQ2_M 2.7, IQ3_XXS 3.06, IQ3_XS 3.3, IQ3_S 3.44, IQ3_M 3.66, IQ4_XS 4.25 and IQ4_NL 4.50. On the project's Llama 3.1 8B measurements, IQ4_XS comes to 4.46 bits per weight (4.17 GiB) against Q4_K_M's 4.89 (4.58 GiB), which is about 9 percent smaller.

The shape of the curve is the lesson. From Q8 down to Q4_K_M, each step costs very little. Below Q4, losses grow fast: Q3_K_M loses almost four times what Q4_K_M does, and Q2_K twenty times. "So going from 8 to 4 bits is cheap, and going from 4 to 2 is expensive," Jake said. "Like trimming a beard," Ethan said. "The first inch is free."

Measured: one small model quantized nine ways

To see the numbers on a real file, here is SmolLM2-135M, a 135-million-parameter model from Hugging Face, quantized from its BF16 GGUF with llama.cpp build b11827. Perplexity was measured on the first 60,000 characters of Alice's Adventures in Wonderland (Project Gutenberg), 20 chunks of 512 tokens:

Three things stand out, and all of them matter when you download real models:

• Small models break the bit-count rule. Q4_K_M came out at 6.17 bits per weight, not 4.9. SmolLM2's hidden size is 576, and k-quants work in blocks of 256, so llama.cpp warned "ncols 576 not divisible by 256 (required for type q4_K) -> falling back to q5_0" and did that for 180 of 272 tensors. Big models (hidden sizes like 4096 or 5120) do not hit this, so their files match the table above. Small models often do.
• Small models lose more per bit. On this 135M model, Q4_K_M raised perplexity by 5.6 percent, Q5_K_M by 4.1 percent and Q6_K by 1.0 percent. Large models usually lose less at the same quant, because they have more redundancy, so for small models each extra bit matters more. Legacy Q4_0 lost 30.5 percent, more than Q3_K_M's 23.1 percent, which is the clearest sign of how much better k-quants are.
• Q8_0 really is near-lossless. Q8_0 changed perplexity by just 0.03 percent (12.392 against 12.389), far inside the measurement's own margin of error of about plus or minus 0.41.

The practical rule follows: for models under about 3 billion parameters, prefer Q8_0 or Q6_K. They are small anyway, and the savings from 4-bit are tiny compared to the quality you give up. Save the aggressive quants for big models, where gigabytes are at stake and the model has more redundancy to spare.

How big is your model at each quant? A size table

File size is roughly parameters × bits per weight ÷ 8. Using the bits per weight measured on Llama 3.1 8B for each type (Q4_K_M 4.89, Q5_K_M 5.70, Q8_0 8.50 and so on), here is what you can expect for common model sizes. Real files vary by a few percent, because each architecture has different tensor shapes, and multimodal models add a separate projector file:

Read it with your memory in mind, not just your disk. If you have a 24 GB graphics card, a 27B model fits at Q4_K_M (16.5 GB) with room for context, but not at Q8_0 (28.7 GB). If you have 16 GB, a 14B model at Q4_K_M (8.6 GB) or Q6_K (11.5 GB) is comfortable, and a 27B model only fits at Q3 or with part of it in system RAM. A 70B model needs about 43 GB at Q4_K_M, which means two 24 GB cards, a 48 GB workstation card, or a Mac with 64 GB or more. For mixture-of-experts models, the whole file must fit in memory, but only the active experts are read per token, so speed behaves like a much smaller model while the size table still applies in full.

How much RAM or VRAM a quant needs

A model needs its file size in memory plus room to work. Use this estimate:

1. Start with the file size of the quant (for split files, add all parts).
2. Add the KV cache, which is memory for the conversation. It grows with context length and depends on the model's design. For a typical 8B model it is around 1 GB at an 8K context at 16-bit precision, and several GB at 32K or more. Many apps let you store it at 8 bits (q8_0), which halves it.
3. Add 1 to 2 GB for runtime buffers.
4. Leave room for your operating system and other apps: 3 to 4 GB on Windows, a bit less on Linux.
5. Compare with your free memory. On a GPU, that is its VRAM. With CPU and GPU together, it is VRAM plus system RAM, with anything that does not fit on the GPU running slower from RAM. On a Mac, it is unified memory, of which macOS lets the GPU use a large share but not all.

Worked example: a 27B model at Q4_K_M is about 16 to 17 GB, plus around 2 GB of cache at a moderate context, plus 1.5 GB of buffers, so about 20 GB. That fits a 24 GB GPU, and it does not fit a 16 GB GPU without spilling into system RAM. If your app reports that a model "requires more system memory" than is available, drop one quant step or shorten the context. Our Ollama memory guide and LM Studio load-failure guide walk through it.

Where the bits go: the arithmetic behind Q8_0 and Q4_0

You can check the bit counts yourself with simple arithmetic. A Q8_0 block holds 32 weights at 8 bits each (32 bytes) plus one 16-bit scale (2 bytes), so 34 bytes for 32 weights: 34 × 8 ÷ 32 = 8.5 bits per weight. That matches the 8.50 bits per weight measured on Llama 3.1 8B exactly. A Q4_0 block holds 32 weights at 4 bits (16 bytes) plus the same 2-byte scale: 18 bytes, or 4.5 bits per weight. The scale is the price of accuracy. It lets each little group of 32 weights use the full range of its 4 or 8 bits, instead of one range for the whole tensor.

K-quants spend those overhead bits more cleverly. Their 256-weight super-blocks hold smaller sub-blocks, each with its own scale and offset, and those scales are themselves quantized. You get finer adjustment for less overhead, which is why Q4_K_M beats Q4_0 so clearly at a similar size. The mix adds the rest of the average. In SmolLM2's 30 layers, the Q4_K_M log shows the attention value tensors (attn_v) and feed-forward down tensors (ffn_down) of 14 layers set to Q6_K, with everything else at Q4_K. Those upgraded tensors are what lift Q4_K_M to an average of 4.89 bits on Llama 3.1 8B, and they are the difference between _M and _S.

That is also why small models break the rule. When a tensor's rows are not a multiple of 256, k-quant blocks do not fit, and llama.cpp falls back to a legacy type with a higher bit count, such as Q5_0 in place of Q4_K. That one rule explains SmolLM2's 6.17 bits per weight at "Q4".

Do smaller quants run faster?

Usually, yes. Generating text is limited by how fast your machine can read the weights from memory, since every token reads every active weight once. Fewer bytes mean faster tokens. In llama.cpp's Llama 3.1 8B table, text generation went from 29 tokens per second at F16 to 51 at Q8_0, 72 at Q4_K_M and 90 at Q2_K_S on the same hardware. So 4-bit is roughly two and a half times as fast as 16-bit there.

Two caveats apply. First, prompt processing (reading your input) depends more on raw compute, and it changes much less between quants: it ranged from about 710 to 920 tokens per second across all types in the same table. Second, more complex formats can be slower to decode on some hardware. Some i-quants and k-quants have optimized code paths only for certain CPUs or GPUs. If speed matters to you, try two candidate quants in your own app and compare tokens per second. Most apps show it, and llama-bench measures it directly.

Which quant should you download? A rule for every machine

• Plenty of memory (file at Q8_0 fits with room to spare): take Q8_0 for small and medium models, or Q6_K if Q8 is tight. There is little reason to run BF16 locally.
• Normal case: Q4_K_M, or Unsloth's UD-Q4_K_XL where offered. If you have a little headroom, Q5_K_M is a cheap quality bump.
• Tight memory: IQ4_XS (about 9 percent smaller than Q4_K_M, with similar quality when made with an imatrix), then Q3_K_M or IQ3_M.
• Very tight: 2-bit (Q2_K, IQ2_M, UD-Q2_K_XL) only for large models, and expect weaker reasoning. A smaller model at Q4 often beats a bigger model at Q2.
• Coding and agents: stay at 4-bit or above. Low-bit quants break tool calling and long reasoning first.
• Small models (under about 3B): Q8_0 or Q6_K. The savings below that are small and the losses large.

Two tie-breakers help. If two options fit, prefer the bigger model at a moderate quant over a smaller model at a high one, for example a 27B at Q4_K_M over a 14B at Q8_0, because size usually buys more ability than precision. And never take the biggest file that technically loads, because you will have no room left for context, and long conversations will fail or crawl.

Quick picks for common machines

Putting the size table and the memory rule together, here are sensible starting points. Each leaves room for a working context, and each assumes a dense model unless noted:

• 8 GB GPU or 16 GB laptop: an 8B model at Q4_K_M (about 4.9 GB). Q5_K_M (5.7 GB) if you keep the context short.
• 12 GB GPU: an 8B model at Q8_0 (8.5 GB), or a 14B model at Q4_K_M (8.6 GB).
• 16 GB GPU: a 14B model at Q6_K (11.5 GB), or a 27B model at IQ3 with some layers in system RAM.
• 24 GB GPU: a 27B model at Q4_K_M (16.5 GB) or Q5_K_M (19.2 GB). A 32B model at Q4_K_M (19.6 GB) fits with a modest context.
• 32 GB system RAM, no big GPU: mixture-of-experts models with few active parameters run well from RAM at Q4, because only a few gigabytes are read per token.
• Mac with 36 to 48 GB unified memory: a 27B or 32B model at Q6_K or Q8_0, or a large MoE model at Q4.
• 64 GB or more: a 70B model at Q4_K_M (about 43 GB), or smaller models at Q8_0 with long contexts.

If your machine sits between two rows, take the lower row's model at the higher row's quant. A model that fits with room to think beats a bigger one that barely loads.

These picks are for chatting, writing and everyday coding help. For agents that read whole repositories or long documents, move one row down, because their contexts grow large, and the KV cache then claims the room these figures leave free. For short, single questions you can often go one row up.

Head to head: Q4_K_M vs Q5_K_M, Q6_K, Q8_0, Q4_K_S, Q4_0 and IQ4_XS

Q4_K_M vs Q5_K_M

Q5_K_M is about 16 percent larger (5.33 against 4.58 GB on Llama 3 8B) and cuts the perplexity cost by two thirds (+0.06 against +0.18). If it fits with your context, take it, especially for coding and reasoning. If it would push you into CPU offload, stay at Q4_K_M, because the speed loss is worse than the quality gain.

Q4_K_M vs Q6_K and Q8_0

Q6_K is 34 percent larger than Q4_K_M and Q8_0 about 74 percent larger, for nearly lossless quality. They are worth it on small models and when memory is plentiful. For a 70B model they cost tens of gigabytes, where Q4_K_M is the sane choice.

Q3_K_M vs Q4_K_M

Q3_K_M saves about 18 percent of size (3.74 against 4.58 GB on Llama 3 8B) but loses almost four times as much quality (+0.66 against +0.18 perplexity). Use it only when Q4_K_M will not fit, and prefer an imatrix-made IQ3_M or IQ3_S at similar size where available.

BF16 or FP16 vs Q4_K_M

BF16 and FP16 are the unquantized originals, about 3.5 times the size of Q4_K_M. Locally, they make sense only for fine-tuning, for making your own quants, or for small models where size does not matter. For chatting and coding, Q8_0 gives you near-identical output at half the size of BF16.

IQ3_XXS vs Q4_K_M

IQ3_XXS is about 3.06 bits per weight, roughly a third smaller than Q4_K_M. It is a good way to fit a much larger model than you otherwise could. On a 24 GB card, a 32B model at IQ3_XXS fits where Q4_K_M would not, but expect noticeably weaker results than Q4_K_M on the same model.

Q4_K_M vs Q4_K_S

Q4_K_S saves about 5 percent of size (4.37 against 4.58 GB) but raises the quality cost by half (+0.27 against +0.18). Choose it only if those few hundred megabytes decide whether the model fits.

Q4_K_M vs Q4_0

Q4_0 is the older legacy format. It is slightly smaller (4.34 GB) but loses nearly three times as much (+0.47). Pick Q4_K_M unless your hardware or app specifically runs Q4_0 faster, which some ARM CPUs and older tools do.

Q4_K_M vs IQ4_XS

IQ4_XS is about 9 percent smaller at similar quality when it is made with a good importance matrix. On machines where every gigabyte counts, it is the better 4-bit choice. On some older or CPU-only setups, i-quants decode more slowly than k-quants, so check your speed.

Q4_K_M vs UD-Q4_K_XL

UD-Q4_K_XL is Unsloth's dynamic 4-bit, usually a little larger than Q4_K_M because more layers get extra bits, and calibrated with an importance matrix. Where both exist, it is generally the better choice at the same nominal level.

How to tell which quant a file or model is

• The file name is the usual clue, as in model-Q4_K_M.gguf. Treat it as a hint, because anyone can rename a file.
• The metadata is authoritative. Every GGUF stores a general.file_type number. In a Q4_K_M file it reads 15, which llama.cpp's own type list maps to Q4_K_M, along with 7 for Q8_0 and 30 for IQ4_XS. Python's gguf package reads it: gguf.GGUFReader("model.gguf").fields["general.file_type"]. Hugging Face also shows each GGUF's metadata in its file viewer.
• Ollama: run ollama show MODEL, which prints the quantization next to the parameter count and context length.
• LM Studio shows the quant on each model card and in the model picker.
• The quantize log, if you made the file yourself, prints the final size and bits per weight.

Checking takes seconds and saves real confusion. It is how you find out that a model page's "Q4" upload is actually a mixed or dynamic quant, or that the file you downloaded last month is not the one you think it is. "So the name is the label on the jar," Jake said, "and the metadata is what's inside." "Exactly," Ethan said. "Read the inside."

What an importance matrix (imatrix) is, and why some files require one

An importance matrix records which weights matter most, measured by running the model over sample text before quantizing. The quantizer then spends its precision where it counts. It improves every quant type, and at very low bits it is mandatory. Asking llama.cpp for IQ2_XXS without one stops with:

 ERROR: this quantization requires an importance matrix!
        - offending tensor: blk.3.ffn_down.weight
        - target type: iq2_xxs

The difference it makes is easy to measure. Using an importance matrix built from a different part of the same book (so the test text was never seen), the same SmolLM2-135M quants improved sharply: Q4_K_M went from +5.6 to +3.7 percent, IQ4_XS from +17.9 to +9.3 percent, and Q2_K from +42.7 to +24.7 percent. IQ2_XXS, which cannot be made without one, landed at +61.2 percent. File sizes stay essentially the same, so the imatrix is quality gained for free. That is why most quant uploaders now use one, and why an i-quant without an imatrix (like the plain IQ4_XS in the table above) performs worse than its reputation.

The calibration text matters. An imatrix built from chat and code helps chat and code, and quant makers such as Unsloth publish theirs. For downloads, you rarely need to think about it: reputable uploaders use one for every i-quant and usually for k-quants too.

Unsloth Dynamic (UD) quants explained

Unsloth's UD files go further than standard mixes. Instead of a fixed recipe (bump these tensor types, squeeze those), Unsloth chooses a quantization type for every layer it can, based on measurements, and calibrates with its own importance matrix built from chat, coding and multilingual text. Its Dynamic 3.0 method is plain post-training quantization, with no extra training (no QAT). Unsloth publishes the imatrix it uses. In practice:

• UD-Q4_K_XL is the go-to 4-bit where it exists, slightly larger than Q4_K_M.
• UD-Q2_K_XL is the smallest size Unsloth still treats as usable for real work on large models.
• UD-IQ1 and UD-IQ2_S and below are for squeezing huge models onto small machines for short, simple questions only. At those sizes, tool calling and multi-step work break down, models tend to loop, and they are not suitable for agents.

Because UD sizes vary per model, compare the actual file sizes on the download page rather than assuming "Q4 means 4.6 GB per 8B".

GGUF quants vs AWQ, GPTQ, FP8, NVFP4, MXFP4 and QAT

• AWQ and GPTQ are 4-bit methods for GPU servers such as vLLM, stored as safetensors rather than GGUF. A GGUF Q4_K_M and an AWQ 4-bit are broadly similar in size and quality. Pick by the runtime you use: GGUF for llama.cpp, Ollama and LM Studio, and AWQ or GPTQ for vLLM and similar servers.
• FP8 is 8-bit floating point, used on recent data-center GPUs. It is comparable in quality to Q8_0, but not a GGUF format.
• NVFP4 is NVIDIA's 4-bit floating-point format for its newest GPUs, appearing now in some GGUF files and Ollama tags. On supported NVIDIA cards it can be fast at 4-bit quality. On other hardware, regular Q4 files are the safer choice.
• MXFP4 is an open 4-bit floating-point format used by some MoE models, and llama.cpp lists it as MXFP4_MOE.
• QAT (quantization-aware training) means the model was trained to expect quantization, so its 4-bit version loses less. A QAT 4-bit build, where a vendor offers one, usually beats a plain post-training Q4 of the same model.

What perplexity tells you, and what it does not

Perplexity measures how surprised a model is by real text: lower means it predicts the next word better. Comparing a quant's perplexity with the original's on the same text is the standard way to measure what quantization cost, and it is the number in every table on this page. It is useful because it is cheap, repeatable and sensitive to small changes.

It also has limits you should know before you trust any single number. Perplexity is an average over many tokens, so a quant can match it closely and still slip on rare skills such as arithmetic, long reasoning chains or tool-call formatting. That is why quant makers increasingly report KL divergence (how far the quant's whole probability distribution drifts from the original's) and top-1 agreement (how often it picks the same next token). Some also run multi-token checks that compare whole short continuations. A good rule follows from this. Use perplexity to rank quants of the same model, never to compare different models, and when two quants are close, test the one you plan to use on your actual tasks.

The size of the test matters too. The SmolLM2 numbers above used 20 chunks of 512 tokens, and llama.cpp reported an uncertainty of about plus or minus 0.41 perplexity on each. That is enough to rank Q8_0, Q6_K, Q4_K_M and Q2_K clearly, but not to call a winner between two quants a fraction of a point apart.

Quantizing mixture-of-experts (MoE) models

MoE models, which hold many "expert" sub-networks and use only a few per token, change the quant decision in two ways. First, most of their size sits in the experts, and each expert is used for only some tokens, so experts tolerate harder squeezing than the always-used attention and shared layers. Good quant mixes for MoE models keep the shared parts at higher precision and push the experts lower. Dynamic quants do this per layer. Second, the whole file must fit in memory even though only a small fraction is read per token. A big MoE model at 3-bit can therefore be fast but still need a lot of RAM.

Practically, for MoE models, prefer the uploader's dynamic or MoE-aware quants (UD files, or ones labeled with MoE mixes), and consider keeping expert weights in system RAM with GPU offload for the rest. llama.cpp's --cpu-moe and --n-cpu-moe options do exactly that. That lets a larger quant fit than you would expect from your VRAM alone.

Quantizing the KV cache too

The weights are not the only thing you can shrink. The KV cache, the model's working memory for the conversation, is stored at 16-bit by default, and for long contexts it can rival the model in size. llama.cpp lets you store it at 8 bits with -ctk q8_0 -ctv q8_0 (cache type for keys and values), which halves its memory with a small quality cost. Lower types such as q4_0 exist too, at a larger cost. Ollama and LM Studio expose the same choice in their settings.

When memory is tight, the order of moves that usually costs you least quality is this. Shorten the context to what you need, then store the KV cache at q8_0, then drop the weights one quant step. All three save memory, but the first two usually cost you less quality than the third.

Download only the quant you need

Model repositories often hold every quant, sometimes over 100 GB in total, so download one file, not the whole repository:

• Hugging Face CLI: hf download unsloth/MODEL-GGUF MODEL-Q4_K_M.gguf --local-dir models/ fetches a single file. For split files, list each part, or use a pattern.
• Ollama can pull a specific GGUF quant from Hugging Face by tag, as in ollama run hf.co/USER/REPO:Q4_K_M, or use the quant tags on its own library pages. Run ollama show afterwards to confirm.
• LM Studio lists every quant of a model with its size and marks the ones likely to fit your hardware, so pick from that list rather than downloading the default blindly.
• Check the size before you click. If the file is more than about 80 percent of your free RAM or VRAM, go one step down, or plan to shorten the context.

Ethan's rule for the shop PC ended up as a single sentence taped to the monitor: "Largest quant that leaves six gigabytes free." Jake added a second line below it: "And ask before downloading 80 GB."

Make your own quant with llama-quantize

1. Get a high-precision GGUF. Download the BF16 GGUF if the model page offers one, or convert the original model with llama.cpp's convert_hf_to_gguf.py script.
2. Optionally build an imatrix with llama-imatrix -m model-bf16.gguf -f calibration.txt -o imatrix.gguf, using text like what you will ask the model.
3. Quantize with llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M, adding --imatrix imatrix.gguf if you built one.
4. Check the log. It prints the original and quantized sizes with bits per weight, and any fallback warnings.
5. Measure with llama-perplexity -m model-Q4_K_M.gguf -f test.txt, and compare against the BF16 file on the same text.

Two cautions. Always quantize from 16- or 32-bit, never from an already-quantized file (--allow-requantize exists, and its own help text warns it can severely reduce quality). And quantizing loads the whole model into memory, so making a 70B quant needs a machine that can hold the 16-bit model. If you cannot, Hugging Face's GGUF-my-repo space builds quants without any local setup.

Common GGUF problems and fixes

• "unknown model architecture" means the app's llama.cpp is too old for this model. Update the app (or its runtime in LM Studio).
• The model loads but answers are garbage: a broken or mismatched file, the wrong chat template, or a quant too low for the model. Re-download, use the bundled template, or step up a quant.
• Out of memory, or very slow: the quant plus context does not fit. Drop a quant step, shorten the context, or store the KV cache at 8 bits.
• Split files (model-00001-of-00003.gguf) need every part in the same folder. Point the app at the first part.
• "requires an importance matrix" when making IQ1 or IQ2 quants: build one with llama-imatrix first.
• Fallback warnings such as "not divisible by 256": normal for small models. Expect a larger file than the table suggests.

Troubleshooting by symptom

My Q4 file is bigger than the table says

The model has tensor sizes that k-quants cannot block evenly, or the uploader kept some tensors at higher precision. Check the quantize log or the file list for actual sizes.

The 2-bit model loops or ignores tools

That is typical of very low-bit quants. Step up to at least Q3 or UD-Q2_K_XL for large models, or 4-bit for agents.

IQ quants run slower than K quants

Some CPUs and older GPUs decode i-quants more slowly. Try the Q4_K_M of the same model and compare tokens per second.

A bigger quant made answers worse

That is usually a different file, template or sampling setting, not the quant. Compare with the same prompt, settings and app version.

I cannot tell which quant Ollama downloaded

Run ollama show MODEL. It lists the quantization, parameters and context of the installed model.

GGUF quantization: frequently asked questions

Ethan downloaded the Q4_K_M, checked that it left six gigabytes free with the context he needed, and deleted the 80 GB BF16 file he had started downloading by mistake. The model ran at a comfortable speed on the first try, with room for long repair notes in the context. "Twenty-two files," Jake said, "and we wanted one." "We always want one," Ethan said. "The trick is knowing which one before the download finishes."

Revision note. Written October 22, 2026, with llama.cpp build b11827's quantization table and fresh measurements on SmolLM2-135M. "Don't bite off more than you can chew," the proverb says, and that is the whole art of picking a quant.