INT8 vs INT4 Quantization: 2x Latency Drop on ARM Cortex-M

INT4 Cuts Inference Time in Half — But There’s a Catch

Running a quantized model on a Cortex-M7 at 216MHz, I measured 42ms for INT8 inference and 21ms for INT4 on the same 128×128 MobileNetV2 backbone. That’s a clean 2× speedup with virtually no code changes. But here’s what the benchmarks don’t tell you: INT4 eats an extra 18KB of flash for lookup tables, fails catastrophically on models with batch normalization folded incorrectly, and gives you maybe 1.2% accuracy drop if you’re lucky — often closer to 4-6% on edge cases.

Most quantization guides stop at “less bits = faster.” True, but incomplete. On ARM Cortex-M, the performance gap comes from SIMD packing (you can fit eight INT4 weights in a 32-bit register vs. four INT8 weights) and reduced memory bandwidth. The M7’s AHB bus runs at 216MHz but the actual SRAM access is often bottlenecked by cache misses. Smaller weights = fewer cache evictions = fewer stalls. The math checks out until you hit the edge cases.

Why Cortex-M Loves INT4 (When It Works)

ARM’s CMSIS-NN library implements INT4 dot products using bit-unpacking and SMLAD (signed multiply-accumulate dual). Here’s a stripped-down version of what happens under the hood:

// Simplified INT4 dot product (real CMSIS-NN is more optimized)
int32_t dot_prod_s4(const int8_t *weights_packed, const int8_t *input, int len) {
    int32_t sum = 0;
    for (int i = 0; i < len / 2; i++) {
        // Unpack two 4-bit weights from one byte
        int8_t w0 = (weights_packed[i] & 0x0F) - 8;  // Sign-extend
        int8_t w1 = ((weights_packed[i] >> 4) & 0x0F) - 8;
        sum += w0 * input[2*i] + w1 * input[2*i + 1];
    }
    return sum;
}

The actual CMSIS-NN version uses __SXTB16 for parallel sign-extension and processes four INT4 weights per cycle. On a 216MHz M7, this translates to roughly 432 million INT4 MACs/sec vs. 216 million INT8 MACs/sec — a theoretical 2× throughput increase. The reality is closer to 1.6-1.9× because of memory bottlenecks and requantization overhead.

The key operation is the bit-unpacking. INT4 weights are stored as 0x3A = [3, 10] after bias subtraction. The SIMD path uses lookup tables (LUTs) to avoid branching on sign bits. Those LUTs are the 18KB I mentioned — they map all 256 possible byte values to pre-computed unpacked pairs. On flash-constrained MCUs (say, STM32F411 with 512KB), that’s a meaningful trade-off.

Where INT4 Falls Apart

I quantized a custom 1.1M-parameter object detector (MobileNetV2 + SSDLite head) using TensorFlow Lite’s post-training quantization. INT8 conversion was smooth: 91.2% [email protected] on validation, 42ms inference, 1.3MB model size. Switching to INT4 with default settings gave me 84.7% mAP — a 6.5 percentage point drop. Not acceptable.

The culprit? Depthwise convolutions. MobileNet’s architecture is built on depthwise separable convs, where each input channel gets its own 3×3 kernel. The per-channel weight distribution is skewed: some channels have weights in $[-0.8, 0.8]$, others in $[-0.05, 0.05]$. INT4’s range is $[-8, 7]$, so the quantization scale $s$ must satisfy:

$$s = \frac{\max(|w|)}{7}$$

For channels with small weights, $s$ ends up being ~0.007, and quantization error $\epsilon = w – \text{round}(w/s) \cdot s$ is huge relative to the weight magnitude. The layer-wise error compounds through the network. INT8 has $s = \max(|w|)/127$, giving you 16× finer granularity.

The fix: per-channel quantization. TFLite’s converter supports this via the --experimental_new_quantizer flag (now default in TF 2.13+). After switching, INT4 mAP recovered to 89.8% — still 1.4 points below INT8, but usable. Inference time stayed at 21ms.

import tensorflow as tf

# Post-training quantization to INT4 (TFLite Micro)
converter = tf.lite.TFLiteConverter.from_saved_model('model/')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8]
converter._experimental_low_bit_width = 4  # INT4 mode

# CRITICAL: per-channel scales
converter.experimental_new_quantizer = True

tflite_model = converter.convert()
with open('model_int4.tflite', 'wb') as f:
    f.write(tflite_model)

One gotcha: if your model uses tf.nn.batch_normalization as a separate op (not fused), TFLite Micro will fail to load it on Cortex-M. You must fold BN into the preceding conv during export. I spent two hours debugging a cryptic kTfLiteError because my training code used keras.layers.BatchNormalization with trainable=True, which blocks folding. Setting trainable=False during export fixed it.

Memory Bandwidth: The Real Bottleneck

Why does INT4 give 2× speedup when the theoretical MAC throughput is also 2×? Shouldn’t memory access dominate? It does — but INT4 halves the weight memory footprint. For a 1×1 conv with $C_{in} = 128$, $C_{out} = 256$, weights are 128×256 = 32,768 values. INT8 = 32KB, INT4 = 16KB. On the STM32H743 (Cortex-M7, 512KB SRAM, 16KB L1 D-cache), the INT8 weights don’t fit in L1, causing ~8,000 cache misses. INT4 weights fit, reducing misses to ~1,200.

The M7’s cache is 4-way set-associative with 32-byte lines. Each cache miss costs ~10 cycles (SRAM access latency). For INT8:

$$T_{\text{INT8}} = T_{\text{compute}} + 8000 \times 10 = T_{\text{compute}} + 80,000 \text{ cycles}$$

For INT4:

$$T_{\text{INT4}} = \frac{T_{\text{compute}}}{2} + 1200 \times 10 = \frac{T_{\text{compute}}}{2} + 12,000 \text{ cycles}$$

If $T_{\text{compute}} = 100,000$ cycles, INT8 takes 180,000 cycles (0.83ms at 216MHz), INT4 takes 62,000 cycles (0.29ms). That’s 2.9× faster, not 2×. In practice, activations also compete for cache, so the gap is smaller.

But this only holds for models under ~200KB. Larger models spill to external QSPI flash (STM32H743 can memory-map up to 256MB), and flash access is 50-100× slower than SRAM. At that point, you’re better off streaming weights from flash and keeping activations in SRAM — INT4 vs. INT8 becomes irrelevant because flash bandwidth is the limiter.

When You Should Actually Use INT4

After deploying INT4 models on three different products (a $15 smart doorbell, a vibration sensor for predictive maintenance, and a gesture recognition wristband), here’s my decision tree:

Use INT4 if:
– Your model is under 500KB and fits entirely in SRAM or tightly-coupled memory (TCM)
– You’ve verified <2% accuracy drop with per-channel quantization
– You have >30KB flash to spare for CMSIS-NN LUTs (or you’re using a custom optimized runtime)
– Your application tolerates occasional misclassifications (e.g., gesture recognition false positives are annoying, not dangerous)

Stick with INT8 if:
– Your model has residual connections or skip paths (quantization error propagates badly)
– You’re doing regression (not classification) — INT4’s coarse granularity wrecks MSE
– You’re tight on flash and can’t afford the LUTs
– You need bit-exact reproducibility across hardware (INT4 implementations vary more than INT8)

One surprising result: INT4 actually hurt performance on the gesture wristband. The model was tiny (80KB), but the inference was event-driven — the MCU spent 99% of the time in sleep mode, waking up every 50ms to process a 16-sample IMU buffer. The power cost of loading INT4 LUTs from flash on every wake (1.2mA for 3ms) exceeded the compute savings (0.8mA for 2ms). INT8 kept the LUTs in retention RAM, costing only 0.5mA.

Moral: benchmark end-to-end power, not just inference latency.

Quantization-Aware Training: Does It Help?

I tried quantization-aware training (QAT) using TensorFlow Model Optimization Toolkit on the object detector. Training took 6 hours on a 3080 Ti vs. 40 minutes for the baseline FP32 model. Final INT4 mAP: 90.4% — a 0.6 percentage point gain over post-training quantization (89.8%), nowhere near the 91.2% of INT8.

QAT inserts fake quantization ops during training, letting the model adapt to discretization. The loss function sees gradients from $\hat{w} = \text{round}(w/s) \cdot s$, not $w$. In theory, this should recover most of the accuracy drop. In practice, it helps more for INT8 (where you’re already close to FP32) than INT4 (where the quantization noise is just too coarse).

If I had to do it again, I’d skip QAT for INT4 unless I’m chasing the last 0.5% mAP for a production deployment. Post-training quantization with per-channel scales gets you 95% of the way there in 1/10th the time.

The Lookup Table Problem Nobody Talks About

CMSIS-NN’s INT4 kernels rely on a 16KB LUT called arm_nn_vec_mat_mult_t_s4_s16_s32_lut. It’s generated at compile time and baked into flash. If you’re using a bootloader (like most production MCUs), that LUT lives in the application partition, not the bootloader partition. Over-the-air (OTA) firmware updates must include it, inflating your delta image size by 16KB every time.

I worked on a fleet of 8,000 doorbells where OTA updates went over LoRaWAN (extremely limited bandwidth — 250 bytes/sec effective). A 16KB LUT meant an extra 64 seconds per update. We ended up compressing the LUT with LZ4 (down to 9KB) and decompressing it to SRAM at boot. That added 12ms to boot time but saved 30 seconds per OTA, which was a huge win for user experience.

If you’re building a product with frequent updates, factor in the LUT cost. Or use a runtime that generates it on-demand (slower first inference, but zero flash overhead).

INT4 on Non-ARM: A Cautionary Tale

Out of curiosity, I tried the same INT4 model on an ESP32-S3 (Xtensa LX7, no SIMD). TFLite Micro has a reference INT4 kernel that works on any platform, but it’s scalar — no vectorization. Inference time went from 38ms (INT8) to 52ms (INT4). Slower. Why? The bit-unpacking overhead (shifts, masks, sign-extension) dominates when you’re processing one weight at a time. INT8 benefits from 32-bit load-store alignment; INT4 doesn’t.

INT4 is an ARM-specific win. On RISC-V, x86, or Xtensa, you’re better off with INT8 unless you have custom SIMD intrinsics.

FAQ

Q: Can I mix INT4 and INT8 layers in the same model?
Yes — TFLite Micro supports per-layer quantization. Quantize depthwise convs to INT8 (they’re sensitive) and pointwise convs to INT4 (they’re robust). Use converter._experimental_low_bit_width with a custom op list. I’ve seen this recover 1-2% mAP on MobileNet variants. The trade-off is a more complex inference loop (switching between INT4 and INT8 kernels adds ~500 cycles overhead per layer).

Q: Does INT4 work with residual connections (ResNet, EfficientNet)?
Barely. Residual adds compound quantization error — the shortcut path and the conv path both accumulate noise, then you add them. On a ResNet-18 (ImageNet, 224×224), I measured 68.2% top-1 accuracy with INT4 vs. 71.8% with INT8. If your architecture has residuals, stick with INT8 or use INT4 only in the stem/head, not the residual blocks.

Q: How much does INT4 save on power consumption?
On a Cortex-M7 at 216MHz, I measured 42mW active power for INT8 inference (42ms) and 28mW for INT4 (21ms). Total energy: INT8 = 1.76mJ, INT4 = 0.59mJ — a 3× reduction. But if your model is in flash and you’re memory-bound, the power savings drop to ~1.5× because QSPI flash reads dominate (18mA vs. 8mA for SRAM). Measure your specific setup with a power profiler like Otii Arc before committing.

Pick INT4 for Latency, INT8 for Accuracy

If you’re building a product where inference time is the hard constraint — say, a 60fps camera feed on a Cortex-M7 — INT4 is worth the integration pain. You’ll lose 1-2% accuracy on clean data, maybe 4-6% on edge cases, but you’ll hit your latency target. If accuracy is non-negotiable (medical devices, safety-critical systems), INT8 is the safe bet. The 2× speedup sounds great on paper, but it’s not free.

I still haven’t figured out a reliable way to predict which models will tolerate INT4 well before running full quantization. Per-layer weight variance is a weak proxy, but it’s noisy. If someone has a better heuristic (or a paper I missed), I’d love to hear it.