Transformers GGUF “Dequantizing the whole model” Fix: PyTorch Version Test on Apple Silicon

If Transformers prints Dequantizing the whole model, because no GGUF matmul kernel is published for this device while loading a GGUF, do not immediately conclude that your Mac cannot run GGUF. The warning means the current environment did not match a usable packed GGUF matmul kernel, so Transformers is falling back to full-model dequantization. The fastest check is to compare your PyTorch version with the kernel builds Hugging Face currently publishes. On the same M1 Pro, our PyTorch 2.14.0 environment hit the fallback while PyTorch 2.13.0 removed the warning. A compatible PyTorch pin is a useful containment step; the durable fix is to lock a currently supported kernel/PyTorch combination and regression-test it. If efficient local inference is your only goal, llama.cpp remains the simpler baseline.

This conclusion comes from a local control, not from rephrasing the launch post. On September 26, 2026, I ran the same Qwen3.5-0.8B Q4_K_M GGUF on an Apple M1 Pro with 16 GB unified memory and macOS 26.6.2 through three paths: Transformers with PyTorch 2.14.0, Transformers with PyTorch 2.13.0, and llama.cpp through its Python binding with Metal GPU offload.

The result: this was a version-compatibility failure, not an M1 Pro limitation

Path	PyTorch	Packed GGUF	Load time	Median 128-token generation	Measured current MPS allocation
Transformers	2.14.0	No, full dequantization	9.08s	29.24 tok/s	2879.97 MB
Transformers	2.13.0	Yes	8.60s	80.55 tok/s	509.47 MB
llama.cpp binding	—	Native GGUF	0.43s	104.95 tok/s	—

Changing only PyTorch from 2.14.0 to 2.13.0 moved the Transformers median from 29.24 to 80.55 tok/s, about 2.75x faster. Measured current MPS allocation dropped from about 2.88 GB to 0.51 GB, roughly 82.3% lower.

More importantly, the full-model dequantization warning disappeared in the 2.13.0 environment.

That rules out the tempting but incorrect diagnosis that the warning simply means “M1 Pro is too old for Transformers GGUF.” The same hardware restored the packed path when the PyTorch/kernel compatibility boundary changed.

What does the warning actually mean?

Hugging Face’s September 2026 direct-GGUF work is useful precisely because supported quantized matrix multiplications can stay packed and run through published ggml-backed kernels instead of first rebuilding ordinary high-precision weights.

That is why this warning is operationally important:

Dequantizing the whole model, because no GGUF matmul kernel is published for this device.
It will need several times the memory the file does, and load more slowly.

It is not a cosmetic warning. It says the expected quantized matmul kernel was not selected, so Transformers is dequantizing the whole model.

Two consequences follow:

1. The memory advantage can collapse. A Q4 file on disk does not guarantee Q4-like runtime weight memory if the loader dequantizes it.
2. Throughput can drop sharply. You are no longer measuring the packed GGUF execution path you thought you enabled.

The current Hugging Face guidance also makes the compatibility requirement explicit: the direct path depends on kernel builds that match the device and supported PyTorch versions. “Apple Silicon is supported” does not imply that every newly released PyTorch build already has a matching published kernel.

Fastest way to verify the root cause

Before reinstalling half the environment, capture the versions:

python -c "import torch, transformers, kernels; print(torch.__version__); print(transformers.__version__); print(kernels.__version__)"

Then load the same GGUF once and inspect the log.

If the full-dequantization warning appears, check whether your exact PyTorch version is covered by a currently published Hugging Face GGUF kernel before changing the model or blaming the GPU.

Our controlled matrix was:

Apple M1 Pro / 16 GB
macOS 26.6.2
Python 3.10.2
Transformers 5.18.0.dev0
kernels 0.17.1

Fallback: PyTorch 2.14.0
Packed path: PyTorch 2.13.0
GGUF: Qwen3.5-0.8B Q4_K_M

At the time of this test, 2.13.0 hit the packed path while 2.14.0 fell back to full dequantization.

Fix it by matching a published kernel, not by worshipping one version number

To reproduce our known-good control in an isolated environment:

python -m venv .venv
.venv/bin/python -m pip install "torch==2.13.0"
.venv/bin/python -m pip install accelerate kernels "git+https://github.com/huggingface/transformers.git"

Then load the original GGUF again.

But PyTorch 2.13.0 is not a forever recommendation. It is the compatible version verified on September 26, 2026 in this exact lab.

The durable procedure is:

1. check which PyTorch versions have a published GGUF/ggml kernel for your device;
2. pin one of those versions;
3. run a minimal load test;
4. confirm the full-dequantization warning is gone;
5. benchmark latency, memory and correctness on your own workload;
6. move forward when a newer PyTorch version gains a compatible published kernel.

That is safer than “always install the newest PyTorch,” and more future-proof than copying a 2.13 pin forever.

Why are there still fallback warnings after the main warning disappears?

The PyTorch 2.13.0 run still reported that some Qwen3.5 operations, including selected causal-convolution and gated-delta-rule operations, had no optimized kernel mapping and were using reference PyTorch implementations.

Those warnings are not the same failure mode.

Full-model dequantization means the central GGUF weight execution path did not match. An operator-level fallback means packed GGUF is working, but some model operations still lack their own optimized kernel.

The numbers show the difference clearly. Even with those residual operator fallbacks, the 2.13.0 path recovered from roughly 29 tok/s to roughly 81 tok/s.

After the fix, how far is Transformers from llama.cpp?

The same GGUF was also run through llama-cpp-python 0.3.35 with Metal GPU offload.

Median generation throughput:

• Transformers + PyTorch 2.13: 80.55 tok/s
• llama.cpp binding: 104.95 tok/s

On this 0.8B, fixed-128-token workload, llama.cpp was about 1.30x faster.

Do not generalize that into “llama.cpp is always 30% faster.” Model size, quant type, context length, batch size, Apple Silicon generation and kernel coverage can all move the result. Hugging Face’s launch article also uses llama-bench for its llama.cpp table, while our local comparison includes prompt processing in both Python paths, so I am not mixing their numbers with ours.

The useful engineering conclusion is narrower:

• use the Transformers route when you want its Python ecosystem, generate(), processors or a unified workflow around existing Transformers code;
• keep llama.cpp as the efficiency baseline when local GGUF inference is the primary product requirement;
• if Transformers suddenly becomes much slower or much more memory-hungry, verify that it has not silently entered full dequantization.

Test methodology

To keep vendor benchmark data separate from XBSTACK measurements, this result comes from an independent local experiment.

Controlled variables:

• one M1 Pro 16 GB machine;
• one Qwen3.5-0.8B Q4_K_M file;
• same Transformers commit and kernels 0.17.1 in both Transformers runs;
• only PyTorch changed between 2.14.0 and 2.13.0;
• llama.cpp used the same GGUF;
• one warm-up before timing;
• three different short prompts;
• exactly 128 generated tokens per timed run;
• median of the three generation rates;
• current MPS allocation recorded for the Transformers paths.

This is not a model-quality benchmark and not a universal runtime leaderboard. It answers one search task:

Is the full-dequantization warning a real performance/memory problem, and what changes when the PyTorch/kernel compatibility is corrected on the same Mac?

In this lab, the answer is unambiguously yes.

When should you not apply this fix mechanically?

Do not downgrade PyTorch just because you saw the words GGUF when:

• the full-dequantization warning is not present;
• your quant type is not currently supported by the packed Transformers kernel path;
• you are not on an applicable device;
• your application requires newer PyTorch functionality;
• your real target is server throughput and a different runtime should be evaluated;
• Hugging Face has already published a compatible kernel for the newer PyTorch version.

A version pin is a diagnostic and compatibility control, not an architecture principle.

FAQ

Why does a Q4_K_M file use much more memory than expected?

The file format on disk and the runtime execution format are different concerns. If Transformers cannot select the matching quantized matmul kernel, it can dequantize the whole model and lose much of the expected runtime memory advantage.

Should every M1/M2/M3 Mac pin PyTorch 2.13?

No. This test proves only that 2.13 worked and 2.14 fell back in this September 26 environment on an M1 Pro. The long-term rule is to match the PyTorch version to currently published kernels.

How do I know the fix worked?

The most direct signal is that the full-dequantization warning disappears. Then rerun the same prompt and check throughput and MPS memory so you do not replace one silent fallback with another.

Can the fixed Transformers path replace llama.cpp?

It depends on the objective. Our corrected Transformers path improved to roughly 81 tok/s, while the same llama.cpp binding workload reached roughly 105 tok/s. Transformers brings ecosystem integration; llama.cpp still provides a mature efficiency-first GGUF runtime.

Related hands-on labs

If the broader problem you are solving is runtime compatibility rather than this one GGUF warning, these XBSTACK labs use the same reproduce-control-verify method:

• AI Tools Lab: real-project tests and reproducible failure analysis
• n8n distroless glibc compatibility failure on ARM64 / Apple Silicon
• Codex response.failed + SSE idle timeout: same-fixture runtime boundary test

Sources and reproducibility

• Hugging Face — Transformers now runs llama.cpp quants
• Transformers — GGUF quantization documentation
• Independent reproduction: note.com — Transformers GGUF / PyTorch compatibility
• Independent reproduction: Takuya GenAI — Transformers GGUF guide

The local experiment is stored under experiments/transformers-gguf-dequantization-repro/. The public version matrix, raw JSON and reproduction steps are available at xbstack/transformers-gguf-dequantization-repro.