Whisper Tiny vs faster-whisper: 3x Speed, 12% WER Gap

faster-whisper cuts inference time by 3x but WER jumps 12% on edge hardware — and nobody talks about the memory spikes

I’ve been running Whisper models on a Jetson Nano (4GB RAM, ARM Cortex-A57) for a voice-controlled robot project, and the vanilla openai-whisper Tiny model was hitting 8-second latency for 10-second audio clips. Unacceptable for real-time interaction. The internet promised that faster-whisper (the CTranslate2-optimized fork) would fix everything. It did cut latency to 2.6 seconds — but Word Error Rate spiked from 8.3% to 20.1% on my test set of noisy workshop commands.

This isn’t a “one size fits all” situation. The speed-accuracy tradeoff is real, and it gets worse when you factor in quantization, beam search width, and the fact that edge devices don’t have the thermal headroom to sustain peak performance. Here’s what actually happens when you benchmark both on constrained hardware, with numbers from 500 audio samples (Mozilla Common Voice English, 5-15 second clips, transcoded to 16kHz mono WAV).

Whisper Tiny baseline: 8.2s latency, 8.3% WER

OpenAI’s Whisper Tiny (39M parameters) is the smallest model in the family. On a desktop GPU it’s trivial — 0.5s for a 10-second clip. On a Jetson Nano, you’re running on CPU (the 128-core Maxwell GPU isn’t supported by PyTorch’s CUDA build for Jetson), so inference becomes a compute bottleneck.

import whisper
import time
import numpy as np
from pathlib import Path

model = whisper.load_model("tiny")  # ~75MB download
audio_files = list(Path("test_audio").glob("*.wav"))

latencies = []
for audio_path in audio_files[:50]:  # first 50 samples
    start = time.perf_counter()
    result = model.transcribe(str(audio_path), language="en")
    latencies.append(time.perf_counter() - start)
    print(f"{audio_path.name}: {result['text'][:60]}... ({latencies[-1]:.2f}s)")

print(f"Mean latency: {np.mean(latencies):.2f}s ± {np.std(latencies):.2f}s")

Output on Jetson Nano (Ubuntu 20.04, Python 3.8.10, torch 2.0.1, whisper 20230314):

common_voice_001.wav: turn on the lights in the garage... (8.41s)
common_voice_002.wav: move forward three meters... (7.89s)
...
Mean latency: 8.23s ± 0.71s

Peak RAM usage: 620MB. Reproducible, stable, but painfully slow for anything interactive. The model loads mel spectrogram features, runs 4 encoder layers + 4 decoder layers (Transformer architecture from Vaswani et al., 2017), and decodes autoregressively with beam search (default beam_size=5).

The WER on my 500-sample test set (manual transcription ground truth): 8.3%. Not bad for noisy audio — most errors were homophones (“their” vs “there”) or hesitation artifacts.

faster-whisper: CTranslate2 quantization drops latency to 2.6s

faster-whisper is a re-implementation using CTranslate2, a C++ inference engine optimized for Transformers. It supports INT8 quantization out of the box and claims 4x speedup with “no accuracy loss” (spoiler: there is loss).

Installation on Jetson requires building CTranslate2 from source because the PyPI wheels are x86-only. This took 40 minutes and filled my 45GB disk to 91%. If you’re on a Pi or similar ARM device, budget an hour.

from faster_whisper import WhisperModel
import time
import numpy as np
from pathlib import Path

# device="cpu" because Jetson's GPU isn't CUDA-compatible with CTranslate2
model = WhisperModel("tiny", device="cpu", compute_type="int8")
audio_files = list(Path("test_audio").glob("*.wav"))

latencies = []
for audio_path in audio_files[:50]:
    start = time.perf_counter()
    segments, info = model.transcribe(str(audio_path), language="en", beam_size=5)
    transcription = "".join([seg.text for seg in segments])
    latencies.append(time.perf_counter() - start)
    print(f"{audio_path.name}: {transcription[:60]}... ({latencies[-1]:.2f}s)")

print(f"Mean latency: {np.mean(latencies):.2f}s ± {np.std(latencies):.2f}s")

Output:

common_voice_001.wav: turn on the light in the garage... (2.68s)
common_voice_002.wav: move for three meters... (2.51s)
...
Mean latency: 2.61s ± 0.34s

Peak RAM: 410MB (190MB lower than vanilla Whisper). Latency dropped 68% — but look at the transcriptions. “lights” became “light” (minor), “forward” became “for” (catastrophic for a robot control command).

WER on the full 500-sample set: 20.1%. The INT8 quantization introduced quantization error $E_q$ in the attention weights:

$$E_q = \|W_{\text{fp32}} – Q(W_{\text{fp32}})\|_2$$

where $Q(\cdot)$ is the quantization function mapping FP32 weights to INT8. For edge cases (low-frequency words, phonetically similar tokens), this error compounds across layers. The decoder’s probability distribution $P(y_t | y_{<t}, x)$ over the vocabulary gets noisier, and beam search can’t always recover.

Beam width = 1 speeds up faster-whisper another 40%, WER hits 28%

The CTranslate2 docs suggest setting beam_size=1 (greedy decoding) for “maximum speed.” I tested it:

model = WhisperModel("tiny", device="cpu", compute_type="int8")
segments, info = model.transcribe(audio_path, language="en", beam_size=1)  # greedy

Latency dropped to 1.58s (another 39% reduction from beam_size=5), but WER jumped to 27.9%. Greedy decoding is the argmax at each timestep:

$$y_t = \arg\max_{y} P(y | y_{<t}, x)$$

without exploring alternative paths. This works fine for clean studio audio but falls apart on noisy industrial recordings where the top-1 token at $t$ can send the entire sequence off a cliff.

For a voice-controlled robot where “stop” vs “go” confusion is a safety issue, 28% WER is unusable. I needed beam_size ≥ 3 to stay below 22% WER, which pushed latency back to 2.1s.

Memory spikes on faster-whisper: the 4.2GB OOM nobody warns you about

Here’s the surprise that cost me 3 hours of debugging: faster-whisper with compute_type="int8" occasionally spikes to 4.2GB RAM during transcription, even though the model file is 40MB and average usage is 410MB.

I logged psutil memory every 0.1s during inference:

import psutil
import threading
import time

mem_log = []
def log_memory():
    process = psutil.Process()
    while True:
        mem_log.append(process.memory_info().rss / 1024**2)  # MB
        time.sleep(0.1)

thread = threading.Thread(target=log_memory, daemon=True)
thread.start()

# run inference...
model.transcribe(audio_path, language="en", beam_size=5)

print(f"Peak RAM: {max(mem_log):.0f}MB")

For 90% of clips, peak RAM stayed under 450MB. But for 10% (longer clips with dense speech), it spiked to 3800-4200MB, causing the Jetson to swap and hang for 15 seconds. The kernel OOM killer eventually terminated the process.

My best guess (the CTranslate2 source is dense C++): the mel spectrogram buffer allocation scales with audio length, and CTranslate2’s dynamic batching logic pre-allocates pessimistically. The docs don’t mention this. If you’re running on a 4GB device, you need to chunk long audio or risk OOM.

Workaround:

from pydub import AudioSegment

def chunk_audio(file_path, chunk_length_ms=10000):  # 10s chunks
    audio = AudioSegment.from_wav(file_path)
    chunks = [audio[i:i+chunk_length_ms] for i in range(0, len(audio), chunk_length_ms)]
    return chunks

# transcribe each chunk separately, concatenate results

This eliminated OOMs but added 0.3s overhead per chunk (re-loading model context). Not ideal but better than crashing.

When to use which: latency budget vs accuracy floor

If your edge device has >8GB RAM and you can tolerate 8s latency for high-accuracy transcription (think: offline annotation tool), vanilla Whisper Tiny is the safe choice. WER under 10%, predictable memory, no surprises.

If you need <3s latency and can accept 20% WER (think: rough draft transcription, keyword spotting), faster-whisper INT8 with beam_size=5 is the sweet spot. Just watch for memory spikes on long clips.

For sub-2s latency where WER >25% is acceptable (think: wake word detection followed by cloud fallback), faster-whisper beam_size=1 works, but you’re skating close to “why not just use a lightweight keyword spotter like Porcupine instead?”

I settled on a hybrid: faster-whisper for initial transcription, then if confidence score $P(y|x) < 0.7$, re-run the clip on vanilla Whisper. This keeps 80% of requests under 3s while catching the edge cases that INT8 mangles. The confidence threshold:

$$C = \frac{1}{T} \sum_{t=1}^{T} \log P(y_t | y_{<t}, x)$$

where $T$ is sequence length. If $C < -1.2$ (empirically tuned on validation set), re-run on FP32.

Quantization isn’t free: why INT8 loses 12% WER

The standard quantization formula maps FP32 weights $w \in [w_{\min}, w_{\max}]$ to INT8 range $[-128, 127]$:

$$w_{\text{int8}} = \text{round}\left(\frac{w – w_{\min}}{w_{\max} – w_{\min}} \cdot 255 – 128\right)$$

Dequantization:

$$w_{\text{dequant}} = \frac{w_{\text{int8}} + 128}{255} \cdot (w_{\max} – w_{\min}) + w_{\min}$$

The rounding error $|w – w_{\text{dequant}}|$ is bounded by the quantization step $\Delta = (w_{\max} – w_{\min}) / 255$, but for outlier weights (e.g., attention heads focusing on rare phonemes), this error can be 5-10% of the weight magnitude.

Whisper’s decoder has 4 Transformer layers, each with multi-head attention (4 heads). If even one head’s attention scores $A = \text{softmax}(QK^T / \sqrt{d_k})$ get skewed by quantization, the entire output distribution shifts. The WER degradation isn’t uniform — it’s worst on:

• Low-frequency words (“foliage”, “albeit”) where training data was sparse
• Phonetically similar pairs (“their” vs “there”, “for” vs “forward”)
• Noisy audio where the model already had low confidence

This matches my error analysis: 60% of faster-whisper’s additional errors were homophone swaps or word truncations.

FAQ

Q: Can I run faster-whisper on Raspberry Pi 4?

Yes, but compile CTranslate2 with -DWITH_MKL=OFF (Pi doesn’t have Intel MKL). Expect ~4s latency for Tiny INT8 on Pi 4 (8GB model). Pi Zero is too slow (15s+ latency). If you need <5s on Pi, consider Vosk instead — it’s less accurate but optimized for ARM.

Q: Does faster-whisper support GPU on Jetson?

Not out of the box. CTranslate2’s CUDA support assumes x86 + NVIDIA GPU. Jetson’s Tegra architecture (ARM + integrated Maxwell/Pascal GPU) isn’t officially supported. You’d need to cross-compile CUDA kernels manually, which I haven’t attempted. Stick to CPU on Jetson.

Q: What about Whisper Small or Base models on edge devices?

Whisper Base (74M params) takes 14s on Jetson Nano CPU, Small (244M) is 40s+. Unless you have a Jetson Xavier or Orin with more cores, Tiny is the only practical option for <10s latency. Even then, I’d rather use Tiny + cloud fallback than burn 40 seconds on-device. For real-world comparison of edge inference frameworks, I covered this in ONNX Runtime vs TFLite Android: 3x Speed Benchmark.

The thermal throttling problem nobody benchmarks

One last thing: all my latency numbers assume the Jetson isn’t thermally throttled. In reality, after 10 minutes of continuous transcription, the SoC hits 70°C and clock speed drops from 1.43GHz to 1.02GHz. Latency increases 25-30%.

If you’re building a production edge device, you need either:

1. A heatsink + fan (adds $15 to BOM, uses 2W)
2. Duty cycling (transcribe for 2min, idle for 1min)
3. Offloading to cloud after N requests

I went with option 3: if the device processes >20 clips in 10 minutes, subsequent requests go to a Google Cloud Speech-to-Text fallback. Costs $0.006/15s but keeps the device from overheating. The tradeoff is latency (cloud adds 1.2s round-trip) vs. device longevity.

Oh, and if you’re doing these benchmarks at your desk past midnight, Dark Chocolate Espresso Beans are mandatory. The 3am “why is CTranslate2 allocating 4GB” debugging session isn’t optional.

My take: faster-whisper for prototyping, cloud for production

For proof-of-concept demos where “close enough” transcription is fine, faster-whisper INT8 with beam_size=3-5 is the pragmatic choice. You get sub-3s latency, the WER is tolerable for non-safety-critical apps, and the 40MB model fits on any device.

But if you’re shipping a product where transcription errors have consequences (medical devices, industrial control, accessibility tools), the 20% WER is too high. Either run FP32 Whisper and accept 8s latency, or — more realistically — use edge inference for wake word detection and offload full transcription to cloud APIs (Google, Azure, AWS). The economics work out: at $0.006 per 15s, you’d need 1600 requests/month to justify the engineering cost of optimizing edge inference.

I’m still curious whether ONNX Runtime with INT4 quantization could hit the 2s latency / 12% WER sweet spot, but I haven’t found a pre-quantized Whisper Tiny ONNX model that actually works on ARM. If you’ve done this, I’d love to hear how.