LlamaIndex Playbook: RAG, Agents, and Production Gotchas

LlamaIndex is an open-source Python and TypeScript framework for connecting large language models to your own data. Jerry Liu and Simon Suo, former Uber AI research scientists, started it in November 2022.

LLMs are trained on public data and have no access to your PDFs, databases, or internal wikis. LlamaIndex adds the loading, indexing, retrieval, and query layer that grounds answers in real documents.

As of June 2026, the GitHub repository has over 50,000 stars. The framework powers applications at KPMG. Monthly package downloads exceed 25 million.

Below: architecture, the five-line quickstart, agents and workflows, LlamaIndex vs LangChain, and the production gotchas beginner tutorials skip.

Key Takeaways

• LlamaIndex is a three-layer framework (data loading, indexing, querying) designed for retrieval-augmented generation (RAG) and document-centric agentic workflows.
• A working RAG pipeline takes 5 lines of Python, but production systems need explicit LLM overrides to avoid the silent OpenAI fallback.
• The framework pivoted in 2025 from "RAG framework" to "AI agents for document OCR and workflows." Most ranking content has not caught up.
• For complex agent orchestration, LangChain or LangGraph is a stronger choice. For deep RAG and data-ingestion pipelines, LlamaIndex leads.
• LiteParse v2.1 (June 2026) is a new model-free, open-source PDF parser that outperforms all existing open-source alternatives on three benchmarks.

What Is LlamaIndex?

LlamaIndex (originally "GPT Index") is the data framework between your private documents and your LLM. Treat it as the onboarding system for an AI assistant: it reads files, chunks them, and stores them in a vector index.

The retriever pulls the right passage before the LLM answers. That pattern is retrieval-augmented generation (RAG).

The framework supports Python (primary) and TypeScript. License: MIT. It integrates with 160+ data sources including PDFs, Notion, Slack, SQL databases, Google Docs, S3, GitHub, and APIs.

Why LlamaIndex Matters in 2026

The official site no longer leads with "RAG framework." As of 2026, LlamaIndex positions itself around "AI agents for document OCR and workflows," with LlamaParse as the flagship commercial product. Virtually every ranking guide still describes it as a data indexing tool for Q&A.

The framework raised a $19 million Series A in March 2025 alongside the launch of LlamaCloud. Google Trends data shows breakout growth for "llamaindex workflows" and "llamaindex agents" year-over-year, which tracks the framework's own product direction.

How LlamaIndex Works: Three-Layer Architecture

Every LlamaIndex app splits into three layers. Get these wrong and you debug the wrong object for hours.

Layer 1: Data Loading and Ingestion

Your documents enter LlamaIndex as Document objects, which are Python objects containing a text field and a metadata dict. A common beginner mistake is treating a Document as a file; it is not. It is a text string with metadata.

SimpleDirectoryReader handles a local folder and auto-detects PDFs, Markdown, Word, PowerPoint, images, and audio. For nested tables, multi-column layouts, or embedded charts, LlamaParse recovers structure the default parser garbles. LlamaHub provides 160+ connectors for external sources.

Once loaded, a NodeParser (default: SentenceSplitter) splits Documents into Nodes. Each Node is an atomic text chunk that carries a node_id, text, metadata, and parent-child relationships to neighboring chunks.

Layer 2: Indexing and Storage

Three index types address different retrieval strategies:

By default, index data lives in memory. For persistent storage, LlamaIndex supports 40+ vector stores including Chroma, Pinecone, Weaviate, Qdrant, pgvector, Milvus, and MongoDB.

Layer 3: Querying and Retrieval

The QueryEngine embeds your query, retrieves relevant Nodes, and passes them to the LLM for synthesis. The ChatEngine is the stateful version: it maintains conversation history across multiple turns.

LlamaIndex offers four retrieval strategies. Pure vector (semantic) search works well for broad, conceptual questions. Keyword (BM25) search works better for exact-term matching.

Hybrid search combines both; ensemble search fuses multiple retrieval strategies. As of 2026, the @llama_index account states the production answer directly:

Pure semantic search dead-ends when an answer spans multiple chunks. Hybrid retrieval is the architecture you want in production.

Key Abstractions

These are the nine objects you will touch in every LlamaIndex project:

The Settings object is where most production headaches start. See the production gotchas section before relying on global defaults.

Build Your First RAG Pipeline

The minimum working example is five lines:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does the document say about pricing?")
print(response)

Prerequisites: pip install llama-index and a valid OPENAI_API_KEY environment variable.

Running with a Local LLM (No OpenAI)

To run locally via Ollama, install the sub-packages and set overrides explicitly:

pip install llama-index-core llama-index-readers-file llama-index-llms-ollama llama-index-embeddings-huggingface

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.llm = Ollama(model="llama3", request_timeout=60.0)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the key points.")
print(response)

Read the production gotchas section on the silent OpenAI fallback before running either snippet in a shared environment.

Persisting to a Vector Store

An in-memory index rebuilds on every restart. For persistence, wire in Chroma:

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext, VectorStoreIndex, SimpleDirectoryReader

chroma_client = chromadb.PersistentClient()
chroma_collection = chroma_client.create_collection("my_docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

On the next run, reload the index from the Chroma collection instead of re-embedding all documents.

Agents and Workflows

Agents: FunctionAgent and ReActAgent

Agents extend the QueryEngine idea to tool-using workers. A FunctionAgent wraps any Python function as an LLM-callable tool:

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can multiply two numbers.",
)

async def main():
    response = await agent.run("What is 1234 * 4567?")
    print(str(response))

asyncio.run(main())

Tim Ruscica (Tech With Tim) demonstrated this pattern in a tutorial with 172,000 views:

"You can wrap any Python function as a tool that you can pass to the LLM. The possibilities of agents [become] quite unlimited."

AgentWorkflow coordinates multiple agents in parallel when a job splits into independent subtasks.

Workflows: Event-Driven Orchestration

Workflows (introduced in v0.10) are the lower-level primitive for fine-grained control. You define steps with @step decorators and connect them with typed Event objects:

from llama_index.core.workflow import (
    Workflow, step, Event, Context, StartEvent, StopEvent
)

class QueryEvent(Event):
    query: str

class SimpleRAGWorkflow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent) -> QueryEvent:
        await ctx.set("query", ev.get("query"))
        return QueryEvent(query=ev.get("query"))

    @step
    async def generate(self, ctx: Context, ev: QueryEvent) -> StopEvent:
        # LLM synthesis call here
        return StopEvent(result="answer")

Workflows support branching, loops, parallel execution, streaming, and state management. They are observable via Arize Phoenix and OpenTelemetry. Use FunctionAgent or ReActAgent for common tasks; reach for Workflows when you need explicit branching, custom error correction, or strict execution order.

LlamaIndex also launched Agentic Document Workflows (ADW) in January 2025: LlamaParse plus agentic reasoning to extract, classify, route, and trigger downstream actions.

LlamaIndex vs LangChain

Combined monthly search volume for "llamaindex vs langchain" and "langchain vs llamaindex" exceeds 800 queries. Short answer: LlamaIndex wins on deep RAG and data ingestion; LangChain wins on general LLM orchestration and complex chains.

IBM's comparison summarizes the positioning: "LlamaIndex: streamlined search-and-retrieval. LangChain: versatile, modular platform." ZenML is more direct: "LlamaIndex is the go-to for data-intensive agentic workflows. LangChain is the comprehensive LLM application framework."

LlamaIndex can own the retrieval layer while LangChain or LangGraph owns orchestration and conditional logic. That split shows up in Python automation pipelines that pull from internal documents and feed a broader workflow.

Simon Suo (@disiok), LlamaIndex's co-founder, framed the longer-term architecture in 2023:

That modular vision is why the two frameworks compose. For Python data analysis workloads, LlamaIndex's retrieval layer plugs into whatever orchestration the team already runs.

LlamaParse, LlamaCloud, and LiteParse v2.1

LlamaParse: Commercial Document Parser

LlamaParse is LlamaIndex's flagship commercial product. It uses a vision-language model (VLM) to handle the document parsing cases where SimpleDirectoryReader breaks down: nested tables, multi-column PDF layouts, embedded charts, and scanned documents with images.

It supports 90+ file formats, 100+ languages, and has processed 1 billion+ documents for 300,000+ users. A verified Applied AI Data Scientist at a major Private Equity fund called it "the premier solution for parsing complex documents in Enterprise RAG pipelines."

Pricing (per LlamaParse): Free tier, Starter, Pro, and Enterprise plans are available; current rates and page limits are listed on the pricing page.

You can use LlamaParse as a drop-in extractor inside SimpleDirectoryReader:

from llama_index.core import SimpleDirectoryReader
from llama_parse import LlamaParse

parser = LlamaParse(result_type="markdown")
file_extractor = {".pdf": parser}
documents = SimpleDirectoryReader("data", file_extractor=file_extractor).load_data()

LlamaCloud: Commercial Platform

LlamaCloud is the managed enterprise platform combining Parse, Extract, and Index in one service. It launched in general availability on March 4, 2025, the same day as the Series A announcement. The waitlist at launch had 10,000+ organizations, including 90 Fortune 500 companies.

Pricing is not publicly listed; it is contact-sales for enterprise.

LiteParse v2.1: Open-Source PDF Parser (June 2026)

In June 2026, LlamaIndex released LiteParse v2.1, a model-free, Rust-based PDF-to-markdown converter. It requires no API key and carries an Apache 2.0 license.

It outperforms pymupdf4llm, opendataloader, pdf-inspector, and markitdown on three benchmarks: olmOCR0-bench, opendataloader-bench, and ParseBench at CVPR 2026.

Jerry Liu announced it on X:

LiteParse is available as CLI, Python, Rust, Node, and WASM interfaces. The intended split with LlamaParse: use LiteParse for fast first-pass parsing in open-source or privacy-sensitive pipelines; use LlamaParse when you need VLM-level accuracy for compliance or accuracy-critical workflows.

Production Gotchas You Need to Know

The Silent OpenAI Fallback

When OPENAI_API_KEY is set anywhere in your environment, LlamaIndex defaults to OpenAI for all LLM inference, even after you configured Ollama or another local model. It does this silently.

On r/LocalLLaMA, the recurring production complaint is leftover tutorial keys quietly sending supposedly local pipelines to OpenAI. A core maintainer confirmed the behavior in March 2026:

"This is a well documented aspect of the library. There is a global enum for setting global defaults, or you can override at the object level. We could always change this behaviour of course, but imo too disruptive/breaking."

u/grilledCheeseFish in r/LocalLLaMA (March 2026)

"the number of 'local-only' setups quietly phoning home because OPENAI_API_KEY was set from some tutorial six months ago is… a lot."

u/theagentledger in r/LocalLLaMA (March 2026)

The fix: override the LLM at the object level in every component, not just globally via Settings:

from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

# Set globals first
Settings.llm = Ollama(model="llama3")
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Then pass explicitly at instantiation to be safe
query_engine = index.as_query_engine(llm=Ollama(model="llama3"))

Scope and Abstraction Trade-offs

On r/LocalLLaMA, practitioners often treat LlamaIndex as a RAG prototyping layer and move agent orchestration to LangGraph. The same maintainer posted this self-diagnosis in December 2025:

"Maintainer of LlamaIndex here 🫡 … Imo the breadth and scope of a lot of projects, including LlamaIndex, is too wide. Really hoping to bring more focus in the new year. All these frameworks are centralizing around the same thing."

u/grilledCheeseFish in r/LocalLLaMA (December 2025)

LlamaIndex has 160+ connectors, and not every connector is equally maintained. The Settings object interacts with components in non-obvious ways.

For complex stateful agent orchestration or workflows that need precise, auditable execution (financial, legal, compliance), LangGraph or direct API clients may be more predictable.

Where LlamaIndex earns its abstraction cost: RAG prototyping, rapid iteration from new data sources, and multi-format document ingestion pipelines. The InfoWorld review frames this cleanly: "Fairly easy to use for LLM applications." The recommendation is to evaluate it alongside LangChain, Semantic Kernel, and Haystack before committing to a production stack.

LlamaIndex in Practice: KPMG

KPMG uses LlamaIndex to ground professional-services analysis in the right source documents.

The pattern is representative of enterprise adoption. Large volumes of unstructured documents (regulatory filings, client reports, audit records) need LLMs that reason over source text, not training-time memory.

LlamaIndex's provenance tracking (which Nodes were retrieved, from which Documents) supports the auditability professional services firms need.

Common LlamaIndex Mistakes to Avoid

Relying on Global Settings Without Explicit Overrides

The Settings object sets defaults, but components can silently fall back to other values if the override chain breaks. Pass llm= and embed_model= explicitly at every component instantiation in privacy-sensitive or air-gapped deployments.

Using SimpleDirectoryReader on Complex PDFs

SimpleDirectoryReader uses basic PDF text extraction and will produce garbled output on PDFs with tables, multi-column layouts, or embedded images. Swap in LlamaParse or LiteParse v2.1 (for model-free parsing) before indexing complex documents.

Garbage-in means garbage-out in RAG systems. Tim Ruscica made the point directly in his LlamaIndex tutorial: RAG quality is bounded by data quality. A bad vector index produces bad results, regardless of the LLM behind it.

Using VectorStoreIndex for Everything

Pure vector search returns the top-k most semantically similar chunks. When the correct answer spans multiple chunks or requires exact keyword matches, vector search alone produces incomplete answers. For production RAG, configure a hybrid retriever that combines vector and keyword search:

from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever

vector_retriever = index.as_retriever(similarity_top_k=5)
bm25_retriever = BM25Retriever.from_defaults(index=index, similarity_top_k=5)

retriever = QueryFusionRetriever(
    [vector_retriever, bm25_retriever],
    similarity_top_k=5,
    num_queries=1,
    mode="reciprocal_rerank",
)

Building Production Agent Workflows in LlamaIndex

On r/LocalLLaMA, the consensus (matching the maintainer's own admission) is that LlamaIndex is primarily a data-ingestion and retrieval layer. For production agent workflows that need fine-grained state, complex conditionals, or strict reproducibility, LangGraph or direct API clients are easier to debug.

Treating LlamaIndex as One Dependency

pip install llama-index pulls the metapackage, including sub-packages you may never import. For lean production deploys, install only the components you use:

pip install llama-index-core llama-index-llms-openai llama-index-vector-stores-chroma