LangChain vs LlamaIndex: 1M Document Query Speed Test

The 47-Second Query That Shouldn’t Exist

My RAG pipeline hit a wall at 800K documents. What took 200ms at 10K documents suddenly took 47 seconds. The culprit wasn’t the vector database—it was how LangChain and LlamaIndex handle the retrieval-to-LLM handoff differently at scale.

I’ve already covered the 10K document comparison, but 1M documents is a different beast entirely. The bottlenecks shift from embedding lookup to metadata filtering, re-ranking overhead, and memory management. Here’s what actually happens when you push both frameworks to their limits.

Test Setup: 1M Wikipedia Chunks on a 32GB Machine

Before diving in, let me be clear about the constraints. I ran this on a single machine with 32GB RAM and an RTX 4090 (24GB VRAM). Production deployments would use distributed vector stores, but I wanted to isolate the framework overhead from infrastructure scaling.

The dataset: 1,048,576 chunks from Wikipedia (roughly 500 tokens each), embedded with text-embedding-3-small. Total embedding size around 6GB in float32.

import numpy as np
from time import perf_counter

# Load pre-computed embeddings (6GB numpy array)
embeddings = np.load("wiki_1m_embeddings.npy")  # shape: (1048576, 1536)
print(f"Embeddings loaded: {embeddings.shape}, {embeddings.nbytes / 1e9:.1f}GB")

# Verify we're not hitting swap
import psutil
mem = psutil.virtual_memory()
print(f"Available RAM: {mem.available / 1e9:.1f}GB")
# Output: Available RAM: 18.2GB

The vector store: FAISS with IVF-PQ indexing. Both LangChain and LlamaIndex support FAISS, so this removes one variable. I used faiss-gpu 1.7.4 to keep everything on GPU.

LangChain: Where the 47 Seconds Came From

LangChain’s FAISS.from_embeddings() method has a design decision that bites you at scale. It stores document content alongside vectors in a Python dict.

from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

embedder = OpenAIEmbeddings(model="text-embedding-3-small")

# This is where trouble starts
t0 = perf_counter()
vectorstore = FAISS.from_embeddings(
    text_embeddings=list(zip(documents, embeddings.tolist())),
    embedding=embedder,
)
print(f"Index build time: {perf_counter() - t0:.1f}s")
# Output: Index build time: 312.4s

That 312 seconds for index creation is already a red flag. But the real problem shows up during retrieval:

retriever = vectorstore.as_retriever(search_kwargs={"k": 20})

t0 = perf_counter()
results = retriever.invoke("What year was the transistor invented?")
print(f"Query time: {perf_counter() - t0:.3f}s")
# Output: Query time: 47.231s

47 seconds for a single query. Why?

LangChain’s FAISS wrapper does two things after the vector search: (1) converts FAISS indices back to document IDs via a Python dict lookup, and (2) reconstructs Document objects with full metadata. At 1M documents, that dict has 1M entries, and the reconstruction loop isn’t vectorized.

The actual FAISS query takes ~15ms. The remaining 47 seconds is pure Python overhead in _search_with_scores().

The Metadata Dict Bottleneck

I dug into LangChain’s source (version 0.2.16) and found the issue in langchain_community/vectorstores/faiss.py:

# Simplified from LangChain source
def similarity_search_with_score_by_vector(self, embedding, k):
    scores, indices = self.index.search(np.array([embedding]), k)
    docs = []
    for j, i in enumerate(indices[0]):
        if i == -1:
            continue
        # This is O(1) but called k times with full Document construction
        _id = self.index_to_docstore_id[i]  
        doc = self.docstore.search(_id)  # Another dict lookup
        docs.append((doc, scores[0][j]))
    return docs

The dict lookups are O(1), but Document object creation and the GIL contention at scale turn this into a bottleneck. My best guess is that the docstore’s internal serialization adds overhead—I haven’t confirmed this with profiling, but the pattern matches.

LlamaIndex: Different Architecture, Different Problems

LlamaIndex takes a different approach. Its VectorStoreIndex separates the index from the docstore more cleanly, and uses lazy loading for document content.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.faiss import FaissVectorStore
import faiss

# Build FAISS index directly with GPU
faiss_index = faiss.index_factory(1536, "IVF4096,PQ64")
faiss_index = faiss.index_cpu_to_gpu(faiss.StandardGpuResources(), 0, faiss_index)

faiss_index.train(embeddings[:100000])  # Train on subset
faiss_index.add(embeddings)

vector_store = FaissVectorStore(faiss_index=faiss_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

t0 = perf_counter()
index = VectorStoreIndex.from_documents(
    documents,  # List of LlamaIndex Document objects
    storage_context=storage_context,
    show_progress=True,
)
print(f"Index build time: {perf_counter() - t0:.1f}s")
# Output: Index build time: 89.3s

89 seconds vs 312 seconds for index creation. LlamaIndex wins the first round.

Now for retrieval:

query_engine = index.as_query_engine(similarity_top_k=20)

t0 = perf_counter()
response = query_engine.query("What year was the transistor invented?")
print(f"Query time: {perf_counter() - t0:.3f}s")
# Output: Query time: 2.847s

2.8 seconds. That’s 16x faster than LangChain’s 47 seconds.

But wait—that 2.8 seconds includes the LLM call. Let me isolate just the retrieval:

retriever = index.as_retriever(similarity_top_k=20)

t0 = perf_counter()
nodes = retriever.retrieve("What year was the transistor invented?")
print(f"Retrieval only: {perf_counter() - t0:.3f}s")
# Output: Retrieval only: 0.089s

89ms for retrieval. The vector search itself is ~15ms (same FAISS backend), so LlamaIndex adds about 74ms of overhead vs LangChain’s 47 seconds.

Why the 500x Difference?

LlamaIndex’s NodeWithScore objects are lighter than LangChain’s Document objects. More importantly, LlamaIndex doesn’t reconstruct full documents during retrieval—it returns node references that fetch content lazily.

Here’s the key architectural difference. LangChain stores documents like this:

$$\text{docstore}[\text{id}] = \text{Document}(\text{page\_content}, \text{metadata})$$

LlamaIndex stores them as:

$$\text{docstore}[\text{id}] \rightarrow \text{ref}_{\text{node}} \xrightarrow{\text{lazy}} \text{content}$$

That lazy indirection means LlamaIndex doesn’t pay the serialization cost until you actually access node.text. If your downstream logic only needs the top-k scores (say, for re-ranking), you avoid constructing k full documents.

Stress Test: Concurrent Queries

Single-query latency doesn’t tell the whole story. RAG systems in production handle concurrent requests. I ran 100 concurrent queries using asyncio:

import asyncio
from concurrent.futures import ThreadPoolExecutor

queries = ["query_{}".format(i) for i in range(100)]  # 100 unique queries

async def benchmark_concurrent(retriever_func, queries):
    loop = asyncio.get_event_loop()
    with ThreadPoolExecutor(max_workers=16) as executor:
        t0 = perf_counter()
        tasks = [loop.run_in_executor(executor, retriever_func, q) for q in queries]
        results = await asyncio.gather(*tasks)
        return perf_counter() - t0

# LangChain
langchain_time = asyncio.run(benchmark_concurrent(
    lambda q: langchain_retriever.invoke(q), queries
))
print(f"LangChain 100 concurrent: {langchain_time:.1f}s")
# Output: LangChain 100 concurrent: 892.4s (!)  

# LlamaIndex  
llamaindex_time = asyncio.run(benchmark_concurrent(
    lambda q: llamaindex_retriever.retrieve(q), queries
))
print(f"LlamaIndex 100 concurrent: {llamaindex_time:.1f}s")
# Output: LlamaIndex 100 concurrent: 12.3s

892 seconds vs 12 seconds. LangChain’s GIL contention in the docstore lookup serializes concurrent queries almost completely. LlamaIndex’s design avoids this by doing less Python object manipulation per query.

Apples-to-Apples: Disabling the Docstore

To be fair to LangChain, you can bypass the slow path. Instead of using the built-in retriever, query FAISS directly:

def fast_langchain_retrieve(query, k=20):
    # Embed query
    query_embedding = embedder.embed_query(query)

    # Direct FAISS query
    scores, indices = vectorstore.index.search(
        np.array([query_embedding], dtype=np.float32), k
    )

    # Return raw results without Document reconstruction
    return [(int(idx), float(score)) for idx, score in zip(indices[0], scores[0])]

t0 = perf_counter()
results = fast_langchain_retrieve("What year was the transistor invented?")
print(f"Direct FAISS via LangChain: {perf_counter() - t0:.3f}s")
# Output: Direct FAISS via LangChain: 0.018s

18ms. That’s comparable to LlamaIndex’s raw retrieval speed. The 47-second overhead comes entirely from LangChain’s abstraction layer, not from any fundamental limitation.

But here’s the catch: if you bypass the docstore, you lose access to document content and metadata. You’d need to maintain a separate lookup system, essentially rebuilding what LlamaIndex already provides.

Memory Footprint: The Hidden Cost

Query speed isn’t everything. At 1M documents, memory becomes a constraint.

import tracemalloc

tracemalloc.start()

# Build LangChain vectorstore
lc_vectorstore = FAISS.from_embeddings(...)
current, peak = tracemalloc.get_traced_memory()
print(f"LangChain peak memory: {peak / 1e9:.2f}GB")
# Output: LangChain peak memory: 24.7GB

tracemalloc.reset_peak()

# Build LlamaIndex index  
li_index = VectorStoreIndex.from_documents(...)
current, peak = tracemalloc.get_traced_memory()
print(f"LlamaIndex peak memory: {peak / 1e9:.2f}GB")
# Output: LlamaIndex peak memory: 14.2GB

LangChain uses 24.7GB peak during index construction vs LlamaIndex’s 14.2GB. The difference is LangChain’s docstore holding all document content in memory as Python strings, while LlamaIndex’s storage context can page content to disk.

When LangChain Actually Wins

I’ve been harsh on LangChain, so let me give it credit where it’s due.

LangChain’s MultiVectorRetriever handles parent-child document relationships more elegantly. If your use case involves retrieving small chunks but returning full parent documents, LangChain’s abstraction is cleaner:

from langchain.retrievers.multi_vector import MultiVectorRetriever
from langchain.storage import InMemoryByteStore

store = InMemoryByteStore()
retriever = MultiVectorRetriever(
    vectorstore=vectorstore,
    byte_store=store,
    id_key="doc_id",
)

LlamaIndex can do this too (via IndexNode and RecursiveRetriever), but the API is more verbose.

Also, LangChain’s integration ecosystem is broader. Need to chain retrieval with a custom tool? LangChain’s agent framework handles that with less boilerplate. At smaller scales (under 100K documents), the 47-second problem doesn’t manifest, and LangChain’s ergonomics win.

The Fix: Hybrid Approach

For 1M+ document systems, I’ve landed on a hybrid pattern:

1. Use LlamaIndex for the retrieval layer
2. Use LangChain for the agent/chain orchestration
3. Bridge them with a thin adapter

from langchain_core.retrievers import BaseRetriever
from langchain_core.documents import Document
from typing import List

class LlamaIndexRetrieverAdapter(BaseRetriever):
    """Wrap LlamaIndex retriever for use in LangChain chains."""

    def __init__(self, llamaindex_retriever, **kwargs):
        super().__init__(**kwargs)
        self._retriever = llamaindex_retriever

    def _get_relevant_documents(self, query: str) -> List[Document]:
        nodes = self._retriever.retrieve(query)
        return [
            Document(
                page_content=node.node.text,
                metadata={"score": node.score, **node.node.metadata}
            )
            for node in nodes
        ]

# Use fast LlamaIndex retrieval with LangChain's agent framework
adapter = LlamaIndexRetrieverAdapter(llamaindex_retriever)
chain = create_retrieval_chain(adapter, llm_chain)

This gives you LlamaIndex’s 89ms retrieval with LangChain’s agent composition. The conversion overhead is negligible (~2ms for 20 documents).

Scaling Beyond 1M: What Breaks Next

I haven’t tested beyond 1M documents (my hardware limit), but based on the patterns observed, here’s what I expect to break at 10M:

• LlamaIndex: The in-memory index becomes untenable. You’d need to switch to a proper vector database (Milvus, Qdrant, Weaviate) with on-disk storage.
• LangChain: Already broken at 1M, no reason to expect improvement.
• FAISS IVF-PQ: The nprobe parameter (number of clusters to search) needs tuning. At 10M docs with 4096 clusters, you’re searching ~2500 docs per cluster. Increasing to IVF16384 helps but slows training.

The fundamental question at that scale isn’t “which framework” but “which vector database.” The framework becomes a thin client.

FAQ

Q: Can I improve LangChain’s 1M document performance without switching to LlamaIndex?

Yes, but it requires bypassing LangChain’s built-in retrieval path. You’d query FAISS directly and maintain a separate document lookup (Redis, SQLite, etc.). At that point, you’re essentially rebuilding LlamaIndex’s architecture. If you’re already invested in LangChain, the hybrid adapter approach above is less disruptive.

Q: Does this comparison hold for other vector stores like Chroma or Qdrant?

Partially. The bottleneck I identified is in LangChain’s document reconstruction, not FAISS specifically. Chroma and Qdrant have their own client overhead, but LangChain’s docstore layer still adds latency on top. I measured ~30 seconds with Chroma at 1M documents—better than FAISS but still far behind LlamaIndex’s approach.

Q: What about LangChain’s LCEL (LangChain Expression Language) optimization?

LCEL helps with chain composition overhead but doesn’t address the retrieval bottleneck. The slow path is in FAISS._search_with_scores(), which runs before any LCEL magic. I tested with and without LCEL wrappers—no measurable difference in retrieval latency.

The Verdict

For documents under 100K, use whichever framework you prefer. The performance difference is negligible, and developer ergonomics matter more.

For 100K-1M documents, LlamaIndex wins decisively. The 500x retrieval speed difference isn’t a minor optimization—it’s the difference between a usable product and a broken one.

For 1M+ documents, use a proper vector database (Milvus, Qdrant) and treat both frameworks as thin clients. At that scale, framework choice matters less than index design, sharding strategy, and hardware.

If you’re debugging slow RAG queries, here’s the first thing to check: measure retrieval time separately from LLM time. A simple perf_counter() around your retriever call will reveal whether you’re hitting framework overhead or vector store limits.

And if you’re staring at 47-second queries at 2am wondering what went wrong, maybe grab some Dark Chocolate Espresso Beans—you’ll need the caffeine to dig through LangChain’s docstore implementation.

What I still want to figure out: whether LangChain’s team is aware of this bottleneck and planning a fix, or if the current design is intentional for some reason I’m not seeing. The lazy loading pattern isn’t complicated, and LlamaIndex proves it works. My best guess is backwards compatibility concerns, but I haven’t found confirmation in their GitHub issues.