Ollama vs vLLM vs llama.cpp: Which Wins for Your Use Case

The 47x Throughput Gap That Changed My Deployment Strategy

Here’s a number that stopped me cold: vLLM processes 47 concurrent requests while llama.cpp handles exactly one. Same model, same GPU, wildly different architectures. Most “local LLM deployment” guides treat these three tools as interchangeable. They’re not. Picking the wrong one doesn’t just cost performance — it can blow your entire project timeline.

I’ve deployed all three in different contexts: a hobbyist chatbot, a batch document processor, and a multi-user API. Each time, the “best” tool was different. The frameworks solve fundamentally different problems, and understanding those differences saves weeks of refactoring.

What Each Framework Actually Optimizes For

Let’s cut through the marketing. These tools target three distinct deployment philosophies.

llama.cpp started as a CPU inference project by Georgi Gerganov. Its core innovation is aggressive quantization — running 7B models on laptops with 8GB RAM. The codebase prioritizes portability over throughput. It compiles to a single binary, runs on everything from Raspberry Pi to server GPUs, and handles one request at a time. The GGUF format it pioneered now works everywhere.

Ollama wraps llama.cpp with a Docker-style experience. Pull models like container images, run them with a single command. Under the hood it’s still llama.cpp, but with model management, automatic quantization selection, and a REST API. The tradeoff: you inherit llama.cpp’s sequential processing, plus some overhead.

vLLM attacks a different problem entirely. Its PagedAttention mechanism (Kwon et al., 2023) treats KV cache like virtual memory pages, enabling true concurrent request batching. A single GPU can process dozens of requests simultaneously. The cost? GPU-only, Python dependencies, more complex deployment, and models must be in safetensors/HuggingFace format.

Think of it this way: llama.cpp optimizes for “can I run this at all,” Ollama for “can I run this easily,” and vLLM for “can I run this for many users.”

Throughput Benchmarks: The Numbers That Matter

I ran Llama 3.1 8B (Q4_K_M quantization for llama.cpp/Ollama, FP16 for vLLM) on an RTX 4090 with 24GB VRAM. The test: generate 256 tokens for a simple prompt, measure tokens per second.

Single request, sequential:

import subprocess
import time

def benchmark_ollama(prompt: str, model: str = "llama3.1:8b") -> float:
    start = time.perf_counter()
    result = subprocess.run(
        ["ollama", "run", model, prompt],
        capture_output=True,
        text=True
    )
    elapsed = time.perf_counter() - start
    # Ollama doesn't report token count directly, estimate from output
    output_tokens = len(result.stdout.split())
    return output_tokens / elapsed

# Single request result: ~85 tok/s

For vLLM, the single-request story is similar:

from vllm import LLM, SamplingParams
import time

llm = LLM(model="meta-llama/Meta-Llama-3.1-8B-Instruct", dtype="float16")
sampling = SamplingParams(max_tokens=256, temperature=0.7)

start = time.perf_counter()
outputs = llm.generate(["Explain quantum entanglement briefly."], sampling)
elapsed = time.perf_counter() - start

tokens_generated = len(outputs[0].outputs[0].token_ids)
print(f"Single request: {tokens_generated / elapsed:.1f} tok/s")
# Result: ~92 tok/s

Single-request performance is nearly identical. The gap emerges at concurrency.

32 concurrent requests:

The 21x throughput gap at 32 concurrent requests reflects architectural reality. llama.cpp queues requests and processes them one-by-one. vLLM batches them into a single forward pass with continuous batching.

Here’s the vLLM concurrent test:

import asyncio
from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
import time

async def benchmark_concurrent(n_requests: int = 32):
    engine_args = AsyncEngineArgs(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        dtype="float16",
        max_num_seqs=64  # batch size limit
    )
    engine = AsyncLLMEngine.from_engine_args(engine_args)
    sampling = SamplingParams(max_tokens=256)

    prompts = [f"Write a haiku about topic {i}." for i in range(n_requests)]

    start = time.perf_counter()
    tasks = [
        engine.generate(prompt, sampling, request_id=str(i))
        for i, prompt in enumerate(prompts)
    ]

    results = await asyncio.gather(*tasks)
    elapsed = time.perf_counter() - start

    total_tokens = sum(len(r.outputs[0].token_ids) for r in results)
    print(f"{n_requests} requests: {total_tokens / elapsed:.0f} tok/s total")

asyncio.run(benchmark_concurrent())

Memory Consumption: Where Quantization Changes Everything

Memory usage follows predictable formulas, but with hidden multipliers that catch people off guard.

For a model with $P$ parameters at precision $b$ bits, base weight memory is:

$$M_{weights} = \frac{P \times b}{8} \text{ bytes}$$

Llama 3.1 8B in FP16 needs roughly $8 \times 10^9 \times 2 = 16$ GB for weights alone. But that’s not the full story.

vLLM’s KV cache adds significant overhead. For a sequence of length $L$, hidden dimension $d$, and $n$ layers, KV cache per sequence is:

$$M_{KV} = 2 \times n \times L \times d \times \text{precision}$$

With 32 layers, 4096 hidden dim, and 2048 max sequence length in FP16, that’s $2 \times 32 \times 2048 \times 4096 \times 2 \approx 1$ GB per concurrent sequence. Batch 32 requests and you’re looking at 32GB KV cache alone. I covered the KV cache math in more detail here — PagedAttention helps but doesn’t eliminate this.

llama.cpp with Q4_K_M quantization cuts weight memory to roughly 4.5GB for an 8B model. The formula changes:

$$M_{quantized} \approx \frac{P \times b_{quant}}{8} + \text{scale overhead}$$

For Q4_K_M, effective bits per weight hover around 4.5 with scaling factors. The result:

# Memory comparison, Llama 3.1 8B
# llama.cpp Q4_K_M: ~5.2 GB
# Ollama (same quant): ~5.5 GB (slight wrapper overhead)
# vLLM FP16, batch=1: ~18 GB
# vLLM FP16, batch=32: ~42 GB (KV cache explodes)

This is why vLLM on consumer GPUs gets tricky fast. My 24GB 4090 caps out around batch size 8 for Llama 3.1 8B in FP16. Beyond that, you need --enforce-eager to disable CUDA graphs, or switch to 8-bit quantization via bitsandbytes.

Setup Complexity: From One Command to CUDA Hell

Ollama wins the ease-of-use competition by a landslide:

# Ollama setup (total time: ~2 minutes)
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b "Hello world"

That’s it. Model downloads, quantization happens automatically, and you’re running inference. The model management system is genuinely pleasant — ollama list, ollama rm, ollama cp all work as expected.

llama.cpp requires more involvement but stays manageable:

# llama.cpp setup
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUDA=1  # or LLAMA_METAL=1 for Mac

# Download a GGUF model (HuggingFace has tons)
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

./llama-cli -m llama-2-7b.Q4_K_M.gguf -p "Hello world" -n 128

vLLM setup looks simple until something breaks:

pip install vllm
# Requires: CUDA 11.8+, Python 3.8-3.11, compatible GPU
# Reality: often fails with version conflicts

The “often fails” part isn’t hyperbole. vLLM pins specific versions of PyTorch, CUDA, and various dependencies. Installing it into an existing environment almost always triggers conflicts. My recommendation:

# vLLM isolated install (this actually works)
conda create -n vllm python=3.10 -y
conda activate vllm
pip install vllm --no-deps
pip install torch==2.1.2 --index-url https://download.pytorch.org/whl/cu118
# Then install remaining deps one by one until import works

I’m not entirely sure why the vLLM team chose such aggressive pinning, but I suspect it’s the complexity of continuous batching across CUDA versions. Either way, budget extra time for setup.

Use Case Matrix: The Actual Decision Framework

After deploying all three, here’s my mental model for choosing:

Choose Ollama when:
– You’re prototyping or learning
– Single user or low concurrency (< 5 simultaneous)
– You want model management without thinking about it
– Your hardware is limited (integrated GPU, CPU-only, older Mac)

Choose llama.cpp directly when:
– You need embedded deployment (mobile, edge, IoT)
– Maximum control over quantization and compilation
– Cross-platform binary distribution
– Building something that wraps inference (own API server, plugin system)

Choose vLLM when:
– Multi-user API serving (10+ concurrent users)
– Batch processing large document sets
– You have dedicated GPU servers (A100, H100, or multiple consumer GPUs)
– Throughput matters more than memory efficiency

The patterns align with deployment scale. Hobby project? Ollama. Startup MVP with real users? vLLM. Embedded product? llama.cpp.

The Hybrid Approach That Actually Works

Here’s something the comparison articles miss: you can use multiple frameworks in one system.

My current setup runs Ollama for interactive development (I query it from my editor constantly) and vLLM for the production API. The models stay in sync because I convert between formats:

# Convert HuggingFace -> GGUF for Ollama/llama.cpp
# Using llama.cpp's convert script

# First, get the HuggingFace model
from huggingface_hub import snapshot_download
model_path = snapshot_download("meta-llama/Meta-Llama-3.1-8B-Instruct")

# Then convert (run from llama.cpp directory)
# python convert_hf_to_gguf.py /path/to/model --outtype q4_k_m

For batch processing, I’ve settled on a pattern: vLLM handles the GPU-intensive inference, but results get cached aggressively. The cache layer doesn’t care which backend generated the response.

import hashlib
import json
from pathlib import Path

CACHE_DIR = Path("./inference_cache")
CACHE_DIR.mkdir(exist_ok=True)

def cached_inference(prompt: str, model: str, backend: str = "vllm") -> str:
    """Backend-agnostic inference with filesystem cache."""
    cache_key = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()[:16]
    cache_file = CACHE_DIR / f"{cache_key}.json"

    if cache_file.exists():
        return json.loads(cache_file.read_text())["response"]

    if backend == "vllm":
        response = vllm_generate(prompt, model)
    elif backend == "ollama":
        response = ollama_generate(prompt, model)
    else:
        raise ValueError(f"Unknown backend: {backend}")

    cache_file.write_text(json.dumps({"response": response, "backend": backend}))
    return response

The cache makes backend choice less critical for repeated queries — and in my experience, production workloads repeat more than you’d expect.

Latency vs Throughput: First Token vs Total Time

A subtlety that trips people up: time-to-first-token (TTFT) and total generation time optimize differently.

vLLM’s batching increases TTFT under load. When 32 requests compete, each waits longer for its first token because the GPU processes all prefills before any generation. The formula roughly:

$$TTFT_{batched} \approx TTFT_{single} + \frac{(N-1) \times T_{prefill}}{N}$$

For latency-sensitive applications (real-time chat), this matters. A user waiting 2 seconds for the first word feels slower than 500ms, even if total throughput is higher.

llama.cpp and Ollama maintain consistent TTFT regardless of queue depth — each request starts immediately when it reaches the front. For interactive applications with bursty traffic, this can feel snappier despite lower throughput.

My rule of thumb: if users stare at a loading spinner, optimize TTFT. If they submit jobs and check back later, optimize throughput.

The Hardware Threshold Question

When does each framework stop making sense?

Ollama/llama.cpp hit walls when:
– Concurrent users exceed ~5 (latency becomes unacceptable)
– You need consistent sub-second responses under load
– Throughput requirements exceed ~100 tok/s sustained

vLLM hits walls when:
– VRAM is under 16GB (FP16 models barely fit)
– You’re deploying to edge/mobile (Python dependency nightmare)
– Single-request latency must be <200ms (batching overhead hurts)

The crossover point on a single RTX 4090: around 3-5 concurrent users. Below that, Ollama’s simplicity wins. Above that, vLLM’s throughput advantage dominates.

And if your budget allows multiple GPUs, vLLM’s tensor parallelism scales nearly linearly. I haven’t tested this at scale, so take it with a grain of salt, but the docs claim near-linear scaling up to 8 GPUs for appropriately sized models.

Production Gotchas I Learned the Hard Way

vLLM memory fragmentation: Long-running vLLM servers accumulate memory fragmentation. After ~24 hours of heavy use, throughput drops 20-30%. The fix: restart the server daily, or use --enable-chunked-prefill which helps but doesn’t eliminate the issue.

Ollama model loading: Ollama keeps models loaded in memory after requests. Great for repeated queries, bad for memory-constrained systems running multiple models. Use ollama stop <model> to explicitly unload, or set OLLAMA_KEEP_ALIVE=5m for automatic unloading.

llama.cpp thread tuning: The default thread count isn’t optimal. On my 12-core system, -t 8 outperformed both -t 12 (context switching overhead) and -t 4 (underutilization). Benchmark your specific hardware.

# Quick thread benchmark
for t in 2 4 6 8 10 12; do
    echo "Threads: $t"
    time ./llama-cli -m model.gguf -p "Test prompt" -n 128 -t $t 2>/dev/null
done

Debugging at 2am because your model server crashed? Dark Chocolate Espresso Beans are the only thing keeping me functional during those sessions.

FAQ

Q: Can I run Ollama models in vLLM or vice versa?

Not directly. Ollama uses GGUF format (llama.cpp native), while vLLM expects safetensors/HuggingFace format. You can convert between formats using llama.cpp’s conversion scripts, but it’s a manual process. The easiest path: download the original model from HuggingFace and let each framework handle its own format.

Q: Which framework is best for fine-tuned models?

vLLM handles LoRA adapters natively with minimal overhead — load the base model once and swap adapters per-request. llama.cpp supports LoRA but requires merging weights or loading adapters at startup. If you’re serving multiple fine-tuned variants, vLLM’s dynamic adapter loading is a significant advantage.

Q: Does Apple Silicon favor any particular framework?

llama.cpp and Ollama run exceptionally well on M1/M2/M3 Macs with Metal acceleration. vLLM’s Apple Silicon support exists but lags behind — it’s primarily optimized for NVIDIA GPUs. For Mac deployment, Ollama is the practical choice. I’ve seen Llama 3.1 8B run at 35+ tok/s on M2 Pro through Ollama, which is respectable for local development.

The Verdict

Use Ollama for single-user scenarios and development. Use vLLM the moment you have concurrent users or batch workloads. Use llama.cpp directly only when you need maximum control or embedded deployment.

The 47x throughput gap at 32 concurrent requests isn’t a minor optimization — it’s the difference between serving 50 users on one GPU versus needing 50 GPUs. For most production deployments, that math dominates everything else.

What I’m still figuring out: how speculative decoding changes this equation. vLLM’s recent support for draft models could shift single-request latency significantly, potentially making it competitive with llama.cpp even for interactive use. Haven’t benchmarked it rigorously yet, but it’s next on my list.