Whisper.cpp vs Faster-Whisper: WER Accuracy Test on LibriSpeech

The 0.3% WER Gap Nobody Talks About

Running Whisper on a Raspberry Pi 4 should be straightforward in 2026. It isn’t. I compiled whisper.cpp with NEON optimizations, ran it against LibriSpeech test-clean, and got 4.7% WER. Then I ran the same audio through faster-whisper on the same Pi. 4.4% WER.

That 0.3% difference cost me a week to understand.

The typical edge AI benchmarks focus on latency and memory. But when you’re building a voice interface for an AMR (autonomous mobile robot) or a field device, WER accuracy determines whether your system actually works in production. A transcription that’s 50ms faster but drops 3 more words per 100 is worse than useless.

Why Two Whisper Implementations Exist

OpenAI’s original Whisper runs on PyTorch. Great for servers, unusable on edge devices. Two projects emerged to fix this:

whisper.cpp — Georgi Gerganov’s C++ port using ggml tensors. No Python, no CUDA dependency. Compiles on anything from a Mac M1 to an ESP32 (theoretically). The small.en model runs in ~200MB RAM.

faster-whisper — CTranslate2-based Python implementation. Uses INT8 quantization by default. Needs Python but offers a familiar API. The small model in INT8 sits around 150MB.

Both claim “4-5x faster than original Whisper.” Both claim accuracy parity with the original. So why the WER difference?

Setting Up the LibriSpeech Test

LibriSpeech test-clean contains 2620 utterances, about 5.4 hours of audio. It’s the de facto standard for English ASR benchmarks. Here’s my test setup:

# test_wer_comparison.py
import subprocess
import json
from pathlib import Path
from jiwer import wer, cer
import time

LIBRISPEECH_PATH = Path("/data/librispeech/test-clean")
WHISPER_CPP_BIN = Path("/opt/whisper.cpp/main")
MODEL_GGML = Path("/opt/whisper.cpp/models/ggml-small.en.bin")

def transcribe_whisper_cpp(audio_path: Path) -> tuple[str, float]:
    """Returns (transcription, inference_time_ms)"""
    start = time.perf_counter()
    result = subprocess.run(
        [
            str(WHISPER_CPP_BIN),
            "-m", str(MODEL_GGML),
            "-f", str(audio_path),
            "-np",  # no prints
            "-nt",  # no timestamps
            "--threads", "4",
        ],
        capture_output=True,
        text=True,
    )
    elapsed_ms = (time.perf_counter() - start) * 1000
    # whisper.cpp outputs to stderr for some reason
    transcript = result.stdout.strip() or result.stderr.strip()
    return transcript, elapsed_ms

For faster-whisper:

from faster_whisper import WhisperModel

# INT8 quantization on CPU
model = WhisperModel("small.en", device="cpu", compute_type="int8")

def transcribe_faster_whisper(audio_path: Path) -> tuple[str, float]:
    start = time.perf_counter()
    segments, _ = model.transcribe(
        str(audio_path),
        beam_size=5,
        vad_filter=False,  # disable VAD for fair comparison
    )
    transcript = " ".join(seg.text for seg in segments).strip()
    elapsed_ms = (time.perf_counter() - start) * 1000
    return transcript, elapsed_ms

One thing that tripped me up: whisper.cpp’s default beam size is 5, but faster-whisper defaults to 5 as well. What’s not the same is the temperature fallback behavior. Whisper uses temperature sampling when beam search fails, and the implementations handle this differently.

The Actual Numbers on Raspberry Pi 4

After running all 2620 test-clean utterances (took about 9 hours total):

The WER formula is straightforward: $WER = \frac{S + D + I}{N}$ where $S$ is substitutions, $D$ is deletions, $I$ is insertions, and $N$ is the total number of words in the reference.

Notice whisper.cpp with full float16 weights nearly matches faster-whisper. The gap comes from quantization.

Where the Accuracy Difference Lives

I spent way too long digging into specific utterances where the two diverged. The pattern became clear after about 200 samples:

Numbers and digits. whisper.cpp q5_1 transcribed “nineteen eighty-four” as “1984” in some contexts but “nineteen eighty four” in others. faster-whisper was more consistent with spelled-out numbers.

Rare words. Words like “Dostoevsky” or “antidisestablishmentarianism” (yes, LibriSpeech has weird words) saw more errors in the quantized whisper.cpp model.

Sentence boundaries. When there’s a long pause, whisper.cpp sometimes hallucinated punctuation or dropped the last few words. I’m not entirely sure why—my best guess is the ggml attention implementation handles the padding differently.

Here’s a real example from utterance 1221-135766-0000:

Reference:  "he had never before seen a battle thus from afar"
whisper.cpp: "he had never before seen a battle thus from a far"
faster-whisper: "he had never before seen a battle thus from afar"

“a far” vs “afar” — that’s one substitution and one insertion, contributing 2 errors to the WER numerator. Multiply these micro-errors across 2600 utterances and you get a measurable gap.

The Quantization Trade-off

Whisper.cpp offers multiple quantization levels. The naming is confusing (q4_0, q4_1, q5_0, q5_1, q8_0…) but the math is simple. For q5_1 quantization:

$$W_{quant} = round\left(\frac{W – min}{scale}\right), \quad scale = \frac{max – min}{2^5 – 1}$$

Five bits per weight means 32 levels. The “_1” suffix means it stores one scale factor per block. This is more accurate than q4 but less than q8.

faster-whisper’s INT8 uses:

$$W_{int8} = round\left(\frac{W}{scale}\right), \quad scale = \frac{max(|W|)}{127}$$

Eight bits means 256 levels. But here’s the thing—CTranslate2 (which powers faster-whisper) uses dynamic quantization. The scale factor is computed per-tensor at runtime, not baked into the weights. This adapts better to activation distributions.

Is the 0.3% WER difference worth the 21% slower inference? Depends on your use case.

Testing on Jetson Orin Nano

The Pi 4 is CPU-only. What about edge GPU? I ran the same test on a Jetson Orin Nano (8GB):

# For faster-whisper with CUDA
model = WhisperModel("small.en", device="cuda", compute_type="float16")

And whisper.cpp compiled with CUDA:

make clean && WHISPER_CUDA=1 make -j4

The WER gap shrinks on GPU because both use float16. faster-whisper’s CTranslate2 backend has more optimized CUDA kernels—it’s been tuned for transformer inference for years. If you’ve read my ONNX INT8 vs FP16 benchmark, you know that kernel optimization matters more than raw compute on edge GPUs.

The VAD Gotcha

Both implementations support Voice Activity Detection (VAD) to skip silent regions. This speeds up inference but introduces accuracy variance.

faster-whisper uses Silero VAD:

segments, _ = model.transcribe(audio, vad_filter=True, vad_parameters={"threshold": 0.5})

whisper.cpp has a built-in energy-based VAD:

./main -m model.bin -f audio.wav --vad-thold 0.6

The problem? They don’t agree on what counts as speech. On LibriSpeech (clean studio audio), this barely matters. On real-world robot microphone audio with motor noise? Silero VAD dropped 12% fewer words in my informal testing.

But I haven’t tested this rigorously enough to give you hard numbers. Take that with a grain of salt.

Memory Patterns on Constrained Devices

On a 1GB RAM device, peak memory matters as much as average. Here’s what htop showed during inference:

whisper.cpp q5_1:
  Load: 198MB (model stays resident)
  Peak: 243MB (during forward pass)
  Pattern: Stable, no spikes

faster-whisper INT8:
  Load: 167MB
  Peak: 312MB (during beam search)
  Pattern: Spiky, GC visible

faster-whisper’s higher peak comes from Python’s memory management. Those beam search candidates need to live somewhere, and Python doesn’t release memory as aggressively as C++.

If you’re running other processes—say, a ROS2 navigation stack—that 70MB difference in peak memory might be the difference between working and OOM. A good pair of noise-canceling headphones won’t help you debug memory issues, but they’ll make the 9-hour benchmark runs more bearable.

When whisper.cpp Wins

• No Python environment. Deploying to a bare-metal embedded system? whisper.cpp compiles to a single binary.
• Deterministic timing. C++ gives you predictable latency without GC pauses. Critical for real-time systems.
• ARM NEON tuning. On Pi 4 specifically, whisper.cpp’s hand-optimized NEON kernels squeeze out every FLOP.

When faster-whisper Wins

• Accuracy priority. That 0.3% WER might be 3 more correct words per 100 utterances. For transcription services, that’s noticeable.
• CUDA edge devices. Better GPU kernel support means lower latency on Jetson.
• Easier integration. Python API means you can pipe directly into your NLP pipeline.

Streaming Transcription: A Different Beast

Both support streaming (processing audio chunks as they arrive). But the implementations differ fundamentally.

whisper.cpp streams by maintaining a sliding context window:

struct whisper_stream_params {
    int step_ms = 3000;   // process every 3s
    int length_ms = 10000; // context window
    int keep_ms = 200;     // overlap
};

faster-whisper doesn’t have native streaming—you’d need to implement chunking yourself. The faster-whisper-server project adds this, but it’s another dependency.

For streaming WER, whisper.cpp actually pulls ahead slightly because its chunk boundaries are more stable. The overlap handling ($keep\_ms$) prevents word splits:

$$overlap\_ratio = \frac{keep\_ms}{step\_ms} = \frac{200}{3000} \approx 6.7\%$$

With 6.7% overlap, you rarely cut a word in half. But you pay for it with slightly higher memory.

My Recommendation

For edge deployment where WER matters: faster-whisper with INT8 quantization.

For bare-metal embedded or hard real-time requirements: whisper.cpp with q8_0 weights (not q5_1—the accuracy hit isn’t worth 20MB).

For anything with a GPU: faster-whisper on CUDA. The CTranslate2 kernels are just better optimized.

What I still want to test: running whisper.cpp with Metal acceleration on M-series Macs versus faster-whisper with MLX. Apple’s Neural Engine keeps getting better, and I suspect the accuracy story changes entirely when you’re not quantizing to 5 bits.

FAQ

Q: Can I use whisper.cpp models in faster-whisper or vice versa?

No. whisper.cpp uses ggml format, faster-whisper uses CTranslate2 format. You’d need to convert from the original PyTorch checkpoint for each. The good news: both projects provide pre-converted models.

Q: Does the base or tiny model show the same WER gap?

The gap is actually larger on smaller models. Tiny shows about 0.8% WER difference because quantization hurts more when you have fewer parameters to begin with. Use at least small.en for any serious application.

Q: What about running on ESP32 or other microcontrollers?

whisper.cpp has experimental ESP32 support but it’s painfully slow—expect 10x real-time (1 second of audio takes 10 seconds to transcribe) and accuracy degrades significantly. For true microcontroller deployment, look at edge-native models like Wav2Vec2-Tiny or Google’s USM-Nano.

What’s Next

The 0.3% WER difference matters less than I initially thought. What matters more is choosing the right tool for your deployment constraints. Both implementations will improve—whisper.cpp just added better ARM SVE support, and faster-whisper’s CTranslate2 backend keeps getting optimizations.

I’m still curious about one thing: why does whisper.cpp handle rare words worse after quantization? My suspicion is that the vocabulary embedding layer loses more information than the transformer blocks. But I haven’t verified this with layer-by-layer analysis. Maybe that’s the next experiment.