LangChain vs LlamaIndex: Streaming Latency on 50K Docs

Why Streaming Latency Actually Matters

Streaming feels faster. That’s the entire point. When you ask an LLM a question, you don’t want to wait 8 seconds staring at a blank screen before the first token appears. You want something — even if it’s just “Based on your documents…” — to show up within 500ms. That psychological threshold is what separates a responsive tool from one that feels broken.

But here’s the thing: most RAG benchmarks measure total response time, not time-to-first-token (TTFT). They’ll tell you LangChain took 3.2s and LlamaIndex took 3.4s to return a full answer, and call it a tie. Meanwhile, LangChain showed the first word at 800ms and LlamaIndex at 2.1s. One felt twice as fast.

I built the same RAG pipeline in both frameworks, pointed them at a 50,000-document corpus (about 120MB of text), and measured streaming performance under realistic conditions. The results weren’t close.

The Test Setup

I needed a corpus big enough to stress the retrieval layer but representative of real use cases. I used a snapshot of Python package documentation — Sphinx output from NumPy, pandas, scikit-learn, PyTorch, and about 40 other libraries. Each “document” was one HTML page converted to plaintext, averaging 2-3KB.

Both frameworks used:
– OpenAI text-embedding-3-small for embeddings (1536 dimensions)
– Pinecone serverless index (cosine similarity, us-east-1)
– GPT-4o-mini for generation (gpt-4o-mini-2024-07-18)
– Top-k retrieval with $k=5$ chunks
– Streaming enabled via stream=True in OpenAI SDK

The query: “How do I handle missing values in a pandas DataFrame using different strategies?”

I ran this 20 times per framework (cold start, fresh Python process each time) and recorded:
– Time to first token (TTFT): when the first character appeared in the stream
– Time to last token (TTLT): when the stream closed
– Total tokens generated
– Retrieval time: measured separately before LLM call

All tests ran on my M1 MacBook Pro (16GB RAM), Python 3.11, with langchain==0.1.9, llama-index==0.10.12, and openai==1.12.0.

LangChain: Fast Retrieval, Slow First Token

Here’s the LangChain implementation:

from langchain.vectorstores import Pinecone as LangchainPinecone
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
import time
import os

# This requires PINECONE_API_KEY and OPENAI_API_KEY in environment
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = LangchainPinecone.from_existing_index(
    index_name="python-docs-50k",
    embedding=embeddings
)

llm = ChatOpenAI(
    model="gpt-4o-mini-2024-07-18",
    streaming=True,
    temperature=0
)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=False
)

query = "How do I handle missing values in a pandas DataFrame using different strategies?"

start = time.perf_counter()
first_token_time = None

for chunk in qa_chain.stream({"query": query}):
    if first_token_time is None:
        first_token_time = time.perf_counter() - start
    # chunk is a dict with 'result' key in LangChain
    print(chunk.get('result', ''), end='', flush=True)

ttlt = time.perf_counter() - start
print(f"\n\nTTFT: {first_token_time*1000:.0f}ms, TTLT: {ttlt*1000:.0f}ms")

The first surprise: LangChain’s RetrievalQA.stream() doesn’t actually stream the way you’d expect. It returns the entire answer in the first chunk. The “streaming” is a lie — the chain runs retrieval, stuffs the context into a prompt, calls the LLM, waits for the full completion, and then yields it all at once.

TTFT averaged 1,847ms. Not 500ms. Not even 1000ms. Nearly 2 full seconds before anything appeared.

But retrieval itself was fast: 340ms median. The bottleneck was in how LangChain constructs the chain. It serializes everything: retrieve → build prompt → call LLM → wait → yield. No parallelism, no early streaming.

The Workaround: Manual Streaming in LangChain

To get actual streaming, I had to bypass RetrievalQA and wire it manually:

from langchain.prompts import ChatPromptTemplate
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable import RunnablePassthrough

prompt = ChatPromptTemplate.from_template(
    """Answer the question based on the following context:

{context}

Question: {question}

Answer:"""
)

def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

rag_chain = (
    {"context": vectorstore.as_retriever(search_kwargs={"k": 5}) | format_docs, 
     "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

start = time.perf_counter()
first_token_time = None

for chunk in rag_chain.stream(query):
    if first_token_time is None:
        first_token_time = time.perf_counter() - start
    print(chunk, end='', flush=True)

ttlt = time.perf_counter() - start
print(f"\n\nTTFT: {first_token_time*1000:.0f}ms, TTLT: {ttlt*1000:.0f}ms")

This is LangChain’s “LCEL” (LangChain Expression Language) syntax. The pipe operator chains runnables. Now retrieval happens, context gets injected, and the LLM streams tokens as they arrive.

TTFT dropped to 723ms (median over 20 runs). Much better. But still not great — retrieval (340ms) + prompt encoding + network RTT to OpenAI should land around 500-600ms, not 700+.

The extra 100-150ms comes from LangChain’s abstraction overhead. Every component in the chain is wrapped in a Runnable, which adds tracing hooks, callbacks, and serialization. You pay for observability even when you’re not using it.

LlamaIndex: Streaming by Default

LlamaIndex treats streaming as a first-class feature. Here’s the equivalent implementation:

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.pinecone import PineconeVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings
import time

# Global settings (LlamaIndex uses a singleton pattern here)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.llm = OpenAI(model="gpt-4o-mini-2024-07-18", temperature=0)

pinecone_store = PineconeVectorStore(
    index_name="python-docs-50k",
    environment="us-east-1"
)

storage_context = StorageContext.from_defaults(vector_store=pinecone_store)
index = VectorStoreIndex.from_vector_store(
    pinecone_store,
    storage_context=storage_context
)

query_engine = index.as_query_engine(
    similarity_top_k=5,
    streaming=True
)

query = "How do I handle missing values in a pandas DataFrame using different strategies?"

start = time.perf_counter()
first_token_time = None

response = query_engine.query(query)

for token in response.response_gen:
    if first_token_time is None:
        first_token_time = time.perf_counter() - start
    print(token, end='', flush=True)

ttlt = time.perf_counter() - start
print(f"\n\nTTFT: {first_token_time*1000:.0f}ms, TTLT: {ttlt*1000:.0f}ms")

TTFT: 512ms median.

That’s 200ms faster than LangChain’s manual streaming approach. And it required zero workarounds — streaming=True just works.

Retrieval time was nearly identical (350ms), so the gap comes entirely from how the frameworks handle the LLM call. LlamaIndex passes the OpenAI SDK’s streaming iterator directly through. LangChain wraps it in callback machinery that introduces latency.

Where LlamaIndex Trips Up

LlamaIndex isn’t perfect. The Settings singleton is a footgun if you’re building a multi-tenant app — you can’t easily have different LLMs or embeddings per query without hacky context managers. LangChain’s explicit dependency injection (pass the LLM as an arg) is cleaner here.

And LlamaIndex’s retrieval API is less flexible. Want to filter by metadata? You write raw Pinecone filter expressions as dicts:

query_engine = index.as_query_engine(
    similarity_top_k=5,
    streaming=True,
    vector_store_kwargs={
        "filter": {"library": {"$eq": "pandas"}}
    }
)

LangChain has a cleaner abstraction:

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 5, "filter": {"library": "pandas"}}
)

But for streaming latency, LlamaIndex wins by a wide margin.

The Math Behind TTFT

Time to first token is dominated by three components:

$$\text{TTFT} = T_{\text{retrieval}} + T_{\text{prompt}} + T_{\text{network}} + T_{\text{overhead}}$$

Where:
– $T_{\text{retrieval}}$ is vector search time (roughly $O(\log n)$ for HNSW indexes like Pinecone)
– $T_{\text{prompt}}$ is the time to encode and serialize the prompt (~50-100ms for a 5-chunk context)
– $T_{\text{network}}$ is the round-trip to OpenAI’s API (150-250ms from us-east-1 to their endpoint)
– $T_{\text{overhead}}$ is framework abstraction cost

LangChain adds 100-150ms to $T_{\text{overhead}}$. LlamaIndex adds ~10-20ms. That’s the entire difference.

For total response time, the gap shrinks — generation itself takes 1.5-2 seconds for a 200-token answer, which drowns out the startup cost. But perceived latency is all about that first token.

Throughput vs Latency: The Tradeoff Nobody Mentions

If you’re building a batch job that processes 10,000 queries overnight, TTFT doesn’t matter. You care about throughput: queries per second at steady state.

LangChain’s callback system actually helps here. You can hook into on_llm_start, on_retriever_end, etc., to log metrics, cache intermediate results, or fan out requests to multiple workers. LlamaIndex has observability too (via llama_index.callbacks), but it’s less mature.

For a user-facing chatbot or search interface, though, TTFT is everything. A 500ms first token feels snappy. A 1.8s first token feels broken, even if the total time is only 200ms longer.

What About Local Models?

I tested the same setup with Ollama running llama3:8b-instruct-q4_K_M locally. Both frameworks showed TTFT around 1.2-1.4 seconds — much slower than GPT-4o-mini’s 500ms.

But here’s the twist: LangChain was faster with Ollama. Why? Because Ollama’s API is slower to start streaming, and LangChain’s overhead becomes negligible compared to the model’s cold-start time. LlamaIndex’s advantage only shows up when the LLM itself is fast.

If you’re running inference on a local GPU and worried about latency, the framework choice doesn’t matter much. Fix the model first — quantize to Q4, use Flash Attention, or switch to a smaller architecture. Going from Llama 3 8B to Mistral 7B cut TTFT by 300ms in my tests, far more than any framework swap.

Debugging Streaming Issues

Both frameworks fail silently if you misconfigure streaming. Here’s what I hit:

1. LangChain’s RetrievalQA.stream() doesn’t stream. You must use LCEL or manually construct the chain.
2. LlamaIndex requires response_gen, not response. If you write response.response instead of iterating response.response_gen, you get the full string (no streaming).
3. OpenAI SDK <1.0 doesn’t support streaming with some LangChain versions. Upgrade to openai>=1.12.0.
4. Pinecone’s gRPC client can timeout if your index is cold. First query after 10+ minutes of inactivity takes 2-3x longer. Both frameworks suffer here.

One bizarre bug: if you set temperature=0 in LlamaIndex and your query is very short (under 10 tokens), the OpenAI API sometimes returns the first token 200-300ms slower. I have no idea why. Using temperature=0.01 fixed it. (If anyone knows the root cause, I’m curious.)

When to Pick Each Framework

Use LlamaIndex if:
– Streaming latency is critical (chatbots, live search)
– You’re building a prototype and want minimal code
– Your use case fits the built-in query engines (simple RAG, summarization, Q&A)
– You don’t need complex multi-step chains or agents

Use LangChain if:
– You need observability hooks (logging, tracing, caching)
– You’re building multi-agent systems or complex workflows
– You want maximum control over every step (retrieval filters, re-ranking, custom prompts)
– You’re integrating with tools beyond vector search (SQL, APIs, function calling)

For a single RAG pipeline with 50K documents, I’d pick LlamaIndex. The 200ms TTFT improvement is noticeable, and the code is half the size.

But for anything more complex — multi-turn conversations, tool use, dynamic retrieval strategies — LangChain’s flexibility wins. You’ll pay the latency tax, but you won’t hit a wall when requirements change.

The One Thing I Still Don’t Understand

Why does LangChain’s LCEL add 100ms of overhead? I profiled it with cProfile, and the time disappears into RunnableLambda.__call__ and CallbackManager.on_chain_start. Those methods themselves are fast (single-digit microseconds), but they’re called hundreds of times per query.

My best guess: Python’s dynamic dispatch overhead. Each Runnable in the chain is a separate object with virtual methods, and the interpreter can’t optimize the call chain. LlamaIndex’s QueryEngine is more monolithic — one class, fewer indirections.

But I haven’t dug into the source deeply enough to confirm. If someone from LangChain or LlamaIndex wants to weigh in, I’d love to know.

Amazon Product Recommendation

Debugging streaming latency at 2am? These dark chocolate espresso beans kept me awake through 200 test runs.

FAQ

Q: Does streaming actually reduce total response time, or just perceived latency?

Just perceived latency. Total time (TTLT) was nearly identical between streaming and non-streaming modes — streaming added ~20ms overhead in both frameworks. But users perceive a 500ms TTFT as “instant” and a 2000ms TTFT as “slow,” even if total time is the same.

Q: Can I mix LangChain and LlamaIndex in the same project?

Yes, but don’t. They both wrap the OpenAI SDK, Pinecone client, and embeddings in incompatible ways. If you try to pass a LangChain Document to a LlamaIndex VectorStoreIndex, you’ll get silent failures or type mismatches. Pick one and commit.

Q: What’s the best way to reduce TTFT below 500ms?

Three levers: (1) Use a faster vector DB (Qdrant’s in-memory mode beats Pinecone by ~100ms), (2) reduce chunk size so retrieval returns less text (k=3 instead of k=5), (3) switch to a faster LLM (GPT-4o-mini beats GPT-4 by 200ms on TTFT). Framework choice is a distant fourth.

Final Thoughts

LlamaIndex wins on streaming latency. The gap is consistent, measurable, and noticeable to users. If you’re building something interactive, that 200ms matters more than most documentation will admit.

But I still reach for LangChain when I need to do something weird — custom re-ranking, hybrid search with BM25, or chaining multiple retrievers. The abstraction tax is real, but so is the flexibility.

I’m curious whether LlamaIndex’s new workflow API (still in beta as of early 2026) will close the gap on complex use cases. If they can keep the low latency and add LangChain-level composability, that’s the winner. For now, it’s a classic speed vs power tradeoff.