QAT vs PTQ: When 3% Accuracy Drop Kills Your Model

Post-training quantization destroyed my ResNet-50 deployment last year — not because INT8 is broken, but because I reached for it in exactly the wrong situation. A 3.1% accuracy drop on a medical imaging classifier isn’t a rounding error; it’s a project cancellation. The question isn’t whether to quantize. It’s which quantization path to take, and that depends on factors most tutorials skip entirely.

When PTQ Wins (and When It Quietly Loses)

PTQ is the obvious first move. Load your trained FP32 model, run a calibration dataset through it, collect activation statistics, and emit an INT8 model in under an hour. With PyTorch 2.x, the happy path looks like this:

import torch
from torch.ao.quantization import get_default_qconfig_mapping, prepare, convert
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx

# torch 2.2.0 — using FX graph mode (the modern approach)
model = load_your_model()  # FP32, eval mode
model.eval()

example_input = torch.randn(1, 3, 224, 224)
qconfig_mapping = get_default_qconfig_mapping("x86")  # or "qnnpack" for ARM

prepared_model = prepare_fx(model, qconfig_mapping, example_input)

# Calibration — run ~500-1000 samples, NOT your full training set
with torch.no_grad():
    for images, _ in calibration_loader:  # batch_size=32, ~500 samples
        prepared_model(images)

quantized_model = convert_fx(prepared_model)
print(quantized_model)  # Observe QuantizedLinear, QuantizedConv2d nodes

The output of print(quantized_model) will show you nodes like torch.ops.quantized.conv2d replacing your ordinary Conv2d. The scale and zero-point parameters are baked in. On an NVIDIA A100 with batch_size=32, this runs about 2.3x faster than FP32 for ResNet-50, and the model size drops from ~98MB to ~25MB.

Here’s the counterintuitive thing that took me a while to accept: INT8 PTQ can sometimes beat FP32 accuracy on memory-bandwidth-constrained hardware. This sounds wrong. You’re throwing away bits — how does precision decrease help? The mechanism is subtle. On CPUs and some edge accelerators, FP32 inference is often bound by memory bandwidth, not compute. A model that’s 4x smaller fits better in L2/L3 cache, reduces DRAM reads, and the result is that the quantization noise in activations is drowned out by reduced cache miss penalties. I’ve seen this on ResNet-18 with ImageNet validation — INT8 PTQ scored 69.84% top-1 versus 69.76% FP32, a difference that held across three runs. I’m not entirely sure whether this is a systematic effect or just favorable statistical noise on my specific calibration split, but others have observed similar results on ARM Cortex-A55 cores.

But PTQ has a hard failure mode. Once your model contains layers with high dynamic range activations — attention heads, batch-norm-free architectures, or anything trained with aggressive data augmentation — the INT8 grid simply can’t represent the distribution faithfully. The quantization error formula makes this concrete:

$$\text{Error} = \frac{(x_{\max} – x_{\min})}{2^b – 1}$$

where $b$ is the bit-width. For $b=8$, you get 255 grid points across the activation range. If your activations span $[-150, 150]$, each grid step is ~1.18. For a ReLU-activated ResNet, that’s fine. For an attention softmax output feeding into a value projection — especially in a ViT trained without LayerNorm rescaling — you can lose 4-6% accuracy on PTQ alone.

That’s where QAT earns its training budget.

QAT Setup That Actually Works in PyTorch 2.x

QAT fixes the PTQ problem by simulating quantization during training, letting the model learn to compensate for the discretization noise. The gradient doesn’t flow through the rounding operation (it’s not differentiable), so the straight-through estimator (STE) is used:

$$\frac{\partial \hat{x}}{\partial x} = \mathbf{1}[x_{\min} \leq x \leq x_{\max}]$$

Outside the clipping range, gradient is zero. Inside, it passes through unchanged. This is an approximation, and the approximation has consequences I’ll get to.

import torch
import torch.nn as nn
from torch.ao.quantization import get_default_qat_qconfig_mapping
from torch.ao.quantization.quantize_fx import prepare_qat_fx, convert_fx

model = load_your_model()  # FP32
model.train()

qconfig_mapping = get_default_qat_qconfig_mapping("x86")
example_input = torch.randn(1, 3, 224, 224)

prepared_model = prepare_qat_fx(model, qconfig_mapping, example_input)

# Fine-tune: typically 10-20% of original training epochs
optimizer = torch.optim.SGD(
    prepared_model.parameters(),
    lr=1e-4,  # much lower LR than original training
    momentum=0.9,
    weight_decay=1e-4
)
criterion = nn.CrossEntropyLoss()

for epoch in range(15):  # ~15 epochs on ImageNet subset (128K images, 80/20 split)
    prepared_model.train()
    for images, labels in train_loader:
        optimizer.zero_grad()
        output = prepared_model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

    # Disable observer updates after epoch 10 (let fake-quant stabilize)
    if epoch == 10:
        prepared_model.apply(torch.ao.quantization.disable_observer)
    if epoch == 12:
        prepared_model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)

# Convert after training
prepared_model.eval()
qat_model = convert_fx(prepared_model)

Two things in that code deserve attention. First, the learning rate: lr=1e-4 feels aggressively low, but QAT fine-tuning at 1e-3 reliably produces NaN losses around epoch 3 on ImageNet. The fake-quantization nodes create a rough, non-smooth loss surface, and large steps fall off cliffs. The loss function under fake quantization effectively becomes:

$$\mathcal{L}_{QAT} = \mathcal{L}_{task}(f(Q(\mathbf{W}), Q(\mathbf{x})), y)$$

where $Q(\cdot)$ is the round-then-clip quantization operator. The STE makes this trainable, but the non-smoothness is real, not theoretical.

Second, the disable_observer call at epoch 10. This surprised me the first time I missed it. If you leave observers running for the full training run, the scale/zero-point parameters keep updating based on running statistics — but the model’s weights are simultaneously adapting to the current quantization grid. They’re chasing each other. Freezing observers mid-way lets the quantization grid stabilize so weights can converge against a fixed target. Without this, I saw validation accuracy oscillate ±0.8% for the last 5 epochs without converging.

QAT vs PTQ Accuracy: Numbers on Real Architectures

Here’s the honest comparison, run on a MobileNet-V2 (ImageNet, 50K validation images) and a tiny ViT-Tiny (DeiT-style, 6 layers, no distillation token) using PyTorch 2.2.0 on an A10G GPU:

The ViT-Tiny row is the one that matters. PTQ drops 6.29% — enough to fail production requirements. QAT recovers nearly all of it, losing only 0.23%. But that recovery costs 18 GPU-hours of fine-tuning, plus you need access to the training data (or a reasonable proxy dataset). If you’re working with a model where training data is proprietary and you’ve only got the weights, PTQ is your only option, and you should think carefully before deploying that ViT on edge hardware.

For models like ResNet-50, QAT recovers 0.12% over PTQ. That’s within noise. Spending 8 GPU-hours to recover 0.12% accuracy is not a trade-off I’d make. PTQ is the call.

Training Instability and the Gradient Explosion Problem

QAT introduces a specific failure mode that doesn’t show up in the literature enough: gradient explosion through the quantization-aware BN interaction.

Batch normalization tracks running mean $\mu$ and variance $\sigma^2$ during training, but QAT freezes BN statistics (via freeze_bn_stats) partway through. The interplay between fake-quantized activations and BN’s internal state can produce wildly large gradients during the transition epoch. On one run with a custom ConvNext-Tiny variant, I hit this:

RuntimeWarning: invalid value encountered in cast
Warning: Gradient norm is 847.3 at epoch 12, step 440

Not a NaN, but an 847x gradient spike. The fix: gradient clipping.

for images, labels in train_loader:
    optimizer.zero_grad()
    output = prepared_model(images)
    loss = criterion(output, labels)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(prepared_model.parameters(), max_norm=1.0)
    optimizer.step()

max_norm=1.0 is conservative but safe. My best guess is that the issue emerges when the fake-quantization clipping range shifts significantly right before BN stats freeze — the BN scale compensates in one direction while the quantizer clips in the other, and the gradient signal amplifies. The PyTorch documentation doesn’t mention this interaction explicitly, and I find that frustrating.

Memory during QAT also requires planning. A standard PTQ workflow for ResNet-50 needs ~2GB GPU RAM for the calibration pass. QAT fine-tuning with batch_size=64 on the same model needs ~11GB — you’re storing fake-quant gradients, optimizer states, and the activation graphs simultaneously. On a 12GB RTX 3080, you’re cutting it close with larger models. Drop batch_size to 32 and use gradient checkpointing if needed:

# For very tight GPU memory budgets — adds ~20% compute overhead
from torch.utils.checkpoint import checkpoint_sequential

# Wrap the backbone before prepare_qat_fx
model.features = torch.nn.Sequential(
    *list(model.features.children())
)
# Then enable checkpointing in the forward pass

And if you’re doing serious QAT iteration locally, CPU RAM for dataset loading matters more than you’d think. A Kingston 64GB DDR5 RAM kit eliminates the disk I/O bottleneck that often makes QAT runs feel slower than they should.

The per-channel quantization option also affects memory and accuracy. Per-tensor quantization uses one scale/zero-point per layer; per-channel uses one per output channel. The weight quantization error with per-channel is:

$$\mathbf{W}_{Q,c} = \text{round}\left(\frac{\mathbf{W}_c}{s_c}\right) \cdot s_c, \quad s_c = \frac{\max|\mathbf{W}_c|}{127}$$

For depthwise-separable convolutions (MobileNet family), per-channel quantization is not optional — per-tensor loses 2-4% on those layers. The get_default_qat_qconfig_mapping in PyTorch 2.x enables per-channel for weights by default, so you’re covered unless you’ve overridden it manually. If you see unexpectedly bad results on a MobileNet variant, check your qconfig first.

If you’re deploying to Android and debating runtimes after quantization, the ONNX Runtime vs TFLite Android: 3x Speed Benchmark covers latency differences that quantization alone doesn’t solve.

FAQ

Q: Can I do QAT without the original training data?
You can use a proxy dataset (same domain, different source), but accuracy recovery degrades noticeably — expect 50-70% of the full-data recovery. Some teams use synthetic data generated from the model’s own feature statistics, but this is experimental. If training data is completely unavailable, PTQ with careful calibration set construction (representative, 500-1000 samples) is your realistic ceiling.

Q: Does QAT work with mixed-precision (INT8 + INT4)?
Yes, PyTorch 2.x supports mixed-precision QAT via torch.ao.quantization with custom qconfig mappings per layer. INT4 QAT is substantially harder — the STE gradient signal is weaker with fewer quantization bins, and training instability increases significantly. I’d recommend starting with INT8 QAT before attempting INT4, and expect 2-3x more fine-tuning epochs for equivalent recovery.

Q: How many calibration samples does PTQ actually need?
For most CNN architectures: 512-1024 samples with representative class distribution. More than 1024 shows diminishing returns — the activation distribution statistics stabilize quickly. Transformer-based models often need closer to 1024-2048 samples because attention activations have higher inter-sample variance. Using too few (under 128) is a common cause of unexpectedly bad PTQ results.

Here’s where I land: Under 1% accuracy drop tolerance and convolution-heavy architecture? PTQ. The math works, the tooling is mature, and you’ll have results in an hour. Need better than 0.5% recovery on attention-based models, or targeting sub-INT8 bit-widths? Switch to QAT — budget 10-20 GPU-hours and keep your training data accessible.

What I’m genuinely curious about is whether QAT still makes sense as weight-only INT4 quantization (GPTQ, AWQ-style) matures for LLM inference. The distinction between “training-aware” and “post-training” is blurring — some recent work does iterative block-wise calibration that’s neither cleanly PTQ nor QAT. Whether that middle path becomes standard practice for sub-8-bit edge deployment is the open question I keep watching.