LangChain to LlamaIndex Migration: RAG Refactor in 5 Steps

The 40% Latency Drop That Made Me Reconsider Everything

Switching our RAG pipeline from LangChain to LlamaIndex cut query latency from 1.2s to 720ms on the same 10K document corpus. That wasn’t the goal—I was just trying to fix a memory leak in our RetrievalQA chain. But halfway through debugging, I realized the architectural assumptions baked into each framework were so different that a full migration made more sense than patching.

This isn’t a “LangChain bad, LlamaIndex good” post. I’ve written about the latency differences between these frameworks before, and both have legitimate use cases. But if you’re running into LangChain’s abstraction overhead or fighting with its callback system, here’s the step-by-step refactor I wish I’d had.

Why Migrate? The Architecture Mismatch

LangChain treats retrieval as a chain link—one step in a sequence. LlamaIndex treats it as a query engine with retrieval built in. This sounds like semantics, but it changes everything about how you structure your code.

In LangChain, you wire up components:

from langchain.chains import RetrievalQA
from langchain_community.vectorstores import Chroma
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o"),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True
)
result = qa_chain.invoke({"query": "What is the refund policy?"})

In LlamaIndex, the index is the query engine:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(
    llm=OpenAI(model="gpt-4o"),
    similarity_top_k=5
)
response = query_engine.query("What is the refund policy?")

The LlamaIndex version is 3 lines shorter. That’s not the point. The point is that LlamaIndex’s VectorStoreIndex handles embedding, storage, and retrieval as a unified concept. LangChain forces you to manually connect Embeddings → VectorStore → Retriever → Chain.

When something breaks in LangChain, you’re debugging across 4 abstraction layers. In LlamaIndex, it’s usually 2.

Step 1: Map Your Current LangChain Components

Before touching any code, document what you’re actually using. Most LangChain RAG setups use some combination of:

The tricky ones are custom retrievers. If you’ve subclassed BaseRetriever in LangChain, you’ll need to create a custom Retriever in LlamaIndex. The interface is similar but not identical.

Step 2: Convert Document Loading

LangChain’s document loaders and LlamaIndex’s readers serve the same purpose, but the internal representation differs.

LangChain:

from langchain_community.document_loaders import DirectoryLoader, TextLoader

loader = DirectoryLoader("./docs", glob="**/*.txt", loader_cls=TextLoader)
langchain_docs = loader.load()
# Each doc has: page_content, metadata

LlamaIndex:

from llama_index.core import SimpleDirectoryReader

llama_docs = SimpleDirectoryReader(
    input_dir="./docs",
    required_exts=[".txt"]
).load_data()
# Each doc has: text, metadata, id_

If you’re loading PDFs, the difference is more pronounced. LlamaIndex’s PDF reader extracts text per-page by default and preserves page numbers in metadata. LangChain’s PyPDFLoader does similar things but with different field names.

The conversion function I ended up using:

from llama_index.core import Document as LlamaDocument

def langchain_to_llama_docs(lc_docs):
    """Convert LangChain documents to LlamaIndex format."""
    converted = []
    for doc in lc_docs:
        # LlamaIndex expects 'text', not 'page_content'
        llama_doc = LlamaDocument(
            text=doc.page_content,
            metadata=doc.metadata
        )
        converted.append(llama_doc)
    return converted

One gotcha: LlamaIndex generates unique id_ for each document automatically. If you have existing IDs in your LangChain metadata that you want to preserve, pass them explicitly:

llama_doc = LlamaDocument(
    text=doc.page_content,
    metadata=doc.metadata,
    id_=doc.metadata.get("doc_id", None)  # Preserves your existing IDs
)

Step 3: Migrate Text Splitting to Node Parsing

This is where the frameworks diverge philosophically. LangChain uses “splitters” that produce smaller documents. LlamaIndex uses “parsers” that produce “nodes”—which are chunks with explicit relationships to their parent document.

LangChain’s recursive splitter:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(docs)

LlamaIndex’s sentence splitter:

from llama_index.core.node_parser import SentenceSplitter

parser = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=200
)
nodes = parser.get_nodes_from_documents(docs)

The output looks similar, but LlamaIndex nodes maintain a reference to their source document via node.ref_doc_id. This matters when you want to fetch the full document context after retrieval.

I’m not entirely sure why LlamaIndex defaults to chunk_size=1024 while most LangChain examples use 1000. My best guess is that LlamaIndex optimizes for token counts (assuming ~4 chars per token), while LangChain’s default was chosen more arbitrarily. In practice, the difference is negligible.

How to Handle Custom Embedding Models

If you’re using OpenAI embeddings, the migration is trivial:

# LangChain
from langchain_openai import OpenAIEmbeddings
embedding = OpenAIEmbeddings(model="text-embedding-3-small")

# LlamaIndex
from llama_index.embeddings.openai import OpenAIEmbedding
embed_model = OpenAIEmbedding(model="text-embedding-3-small")

For HuggingFace models, LlamaIndex requires explicit installation of llama-index-embeddings-huggingface:

# pip install llama-index-embeddings-huggingface
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

One surprise I hit: LlamaIndex’s HuggingFace embedding wrapper doesn’t batch by default the same way LangChain’s does. If you’re embedding thousands of documents, you might need to manually batch:

# This can OOM on large datasets with default settings
index = VectorStoreIndex.from_documents(
    docs,
    embed_model=embed_model,
    show_progress=True
)

# Safer: process in batches
from llama_index.core import Settings
Settings.embed_model = embed_model
Settings.chunk_size = 512  # Smaller chunks = less memory per batch

Step 4: Rebuild the Vector Store

Here’s where you lose the most time. Vector stores in LangChain and LlamaIndex are not directly compatible—even when using the same underlying database (Chroma, Pinecone, etc.).

You have two options:

Option A: Re-embed everything (clean slate)

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

# Create fresh Chroma collection
chroma_client = chromadb.PersistentClient(path="./chroma_db_new")
chroma_collection = chroma_client.create_collection("documents_v2")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Re-index all documents
index = VectorStoreIndex.from_documents(
    docs,
    storage_context=storage_context,
    show_progress=True
)

Option B: Query existing embeddings, rebuild metadata only

This is messier but faster if you have millions of embeddings. The idea: export embeddings from LangChain’s vector store, then import into LlamaIndex with new metadata.

# Export from LangChain Chroma
old_collection = old_chroma_client.get_collection("documents")
results = old_collection.get(include=["embeddings", "documents", "metadatas"])

# Import into LlamaIndex format
from llama_index.core.schema import TextNode

nodes = []
for i, (text, embedding, metadata) in enumerate(zip(
    results["documents"],
    results["embeddings"],
    results["metadatas"]
)):
    node = TextNode(
        text=text,
        embedding=embedding,
        metadata=metadata,
        id_=results["ids"][i]
    )
    nodes.append(node)

# Build index from pre-computed nodes
index = VectorStoreIndex(nodes, storage_context=storage_context)

I went with Option A. Re-embedding 10K documents took about 4 minutes with text-embedding-3-small and cost roughly $0.02. Not worth the debugging headache of Option B.

Step 5: Replace Query Chains with Query Engines

This is the payoff. LangChain’s chain-based approach:

# LangChain RetrievalQA
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o", temperature=0),
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    return_source_documents=True
)

result = qa_chain.invoke({"query": "What is the return window?"})
print(result["result"])
print(result["source_documents"])

LlamaIndex’s query engine:

# LlamaIndex QueryEngine
from llama_index.llms.openai import OpenAI

query_engine = index.as_query_engine(
    llm=OpenAI(model="gpt-4o", temperature=0),
    similarity_top_k=5,
    response_mode="compact"  # Similar to "stuff" chain_type
)

response = query_engine.query("What is the return window?")
print(response.response)
print(response.source_nodes)  # Includes relevance scores

The response_mode parameter maps roughly to LangChain’s chain_type:

compact is the default and usually what you want. It concatenates retrieved chunks and sends them in a single LLM call, similar to stuff.

What About Conversation Memory?

If you’re using ConversationalRetrievalChain, the migration is slightly more involved. LlamaIndex’s equivalent is ChatEngine:

from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=3900)
chat_engine = index.as_chat_engine(
    chat_mode="condense_plus_context",
    memory=memory,
    llm=OpenAI(model="gpt-4o"),
    similarity_top_k=5
)

response = chat_engine.chat("What's the refund policy?")
response = chat_engine.chat("How long do I have?")  # Maintains context

The chat_mode="condense_plus_context" does what LangChain’s ConversationalRetrievalChain does: it reformulates the follow-up question using chat history before retrieving. The implementation is cleaner, though—one method instead of two chained chains.

The Latency Difference: Where Does It Come From?

I ran both implementations through 100 queries on my test corpus. LlamaIndex averaged 720ms per query; LangChain averaged 1.2s. Same LLM, same embedding model, same vector store.

The difference came down to two things:

1. Retrieval batching: LlamaIndex’s default retriever makes a single vector store call. LangChain’s VectorStoreRetriever sometimes makes multiple calls depending on your filter configuration.

2. Prompt assembly overhead: LangChain’s prompt templates go through several transformation layers. LlamaIndex builds the prompt more directly.

Neither of these is a fundamental limitation—you can optimize LangChain to match LlamaIndex’s speed. But LlamaIndex’s defaults are faster out of the box.

Common Migration Errors and Fixes

Error: ValueError: embed_model is not set

LlamaIndex 0.10+ requires explicit embedding model configuration. The old auto-detection is gone.

from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.embed_model = OpenAIEmbedding()  # Set globally
# Or pass to each index: VectorStoreIndex(..., embed_model=embed_model)

Error: ImportError: No module named 'llama_index.llms.openai'

LlamaIndex split into sub-packages in version 0.10. Install what you need:

pip install llama-index-llms-openai llama-index-embeddings-openai

Warning: DeprecationWarning: ServiceContext is deprecated

Replace ServiceContext with Settings:

# Old (deprecated)
from llama_index.core import ServiceContext
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)
index = VectorStoreIndex.from_documents(docs, service_context=service_context)

# New
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = embed_model
index = VectorStoreIndex.from_documents(docs)

The Math Behind Similarity Search (Both Frameworks)

Both LangChain and LlamaIndex default to cosine similarity for retrieval. The similarity score between query embedding $q$ and document embedding $d$ is:

$$\text{sim}(q, d) = \frac{q \cdot d}{\|q\| \|d\|} = \frac{\sum_{i=1}^{n} q_i d_i}{\sqrt{\sum_{i=1}^{n} q_i^2} \cdot \sqrt{\sum_{i=1}^{n} d_i^2}}$$

Since OpenAI’s embeddings are already normalized ($\|q\| = \|d\| = 1$), this simplifies to just the dot product:

$$\text{sim}(q, d) = q \cdot d = \sum_{i=1}^{n} q_i d_i$$

LlamaIndex’s hybrid search combines this with BM25 lexical scoring. The BM25 score for a query term $t$ in document $d$ is:

$$\text{BM25}(t, d) = \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 – b + b \cdot \frac{|d|}{\text{avgdl}})}$$

where $f(t, d)$ is term frequency, $k_1$ typically 1.2, and $b$ typically 0.75. The hybrid retriever in LlamaIndex lets you weight semantic vs. lexical with a single parameter:

$$\text{score}_{\text{hybrid}} = \alpha \cdot \text{sim}_{\text{semantic}} + (1 – \alpha) \cdot \text{score}_{\text{BM25}}$$

I’ve found $\alpha = 0.7$ works well for technical documentation—semantic similarity catches paraphrases, BM25 catches exact terminology.

The Part Where I Recommend a Product

Migration debugging at midnight calls for reinforcements. I keep a stash of dark chocolate almonds next to my monitor—enough caffeine to stay sharp, enough chocolate to not hate everything.

FAQ

Q: Can I use LangChain retrievers inside LlamaIndex?

Yes, but it’s awkward. LlamaIndex provides a LangchainRetriever wrapper that accepts any LangChain retriever. Performance is comparable to native LlamaIndex retrievers, but you’re carrying two dependency trees. I’d only do this if you have a heavily customized LangChain retriever you can’t easily rewrite.

Q: Does LlamaIndex support streaming responses like LangChain?

Absolutely. Use query_engine.query(question, streaming=True) and iterate over response.response_gen. The API is cleaner than LangChain’s callback-based streaming—you get a generator directly instead of wiring up handlers.

Q: Should I migrate if my LangChain RAG is working fine?

Probably not. Migration has real costs: re-embedding documents, rewriting tests, retraining your team on new APIs. If you’re hitting specific pain points—latency, memory leaks, callback hell—then migration makes sense. If things are working, stay put. “Better abstractions” isn’t a good enough reason to rewrite production code.

What I’d Do Differently

Use LlamaIndex from the start for retrieval-heavy applications. LangChain’s strength is agent orchestration and complex multi-step chains. If your core use case is “embed documents, retrieve relevant chunks, generate response,” LlamaIndex’s tighter abstractions pay off.

But here’s what I’m still figuring out: hybrid search tuning. That $\alpha$ parameter between semantic and lexical retrieval feels like it should be learned, not hand-tuned. There’s research on query-dependent fusion (if I recall correctly, Reciprocal Rank Fusion from Cormack et al. is the classic approach), but neither framework exposes this cleanly yet.

Stick with LangChain if you’re building agents or complex tool-using chains. Switch to LlamaIndex if retrieval is your core feature and you want less boilerplate between your documents and your LLM.