Large language models are confidently wrong more often than most teams admit. They hallucinate citations, invent dates, and fabricate statistics with the same fluency they use for things they actually know. The core issue is structural: an LLM only has access to what it memorized during training. It cannot look something up.
Retrieval-Augmented Generation (RAG) fixes this by adding a search step before generation. Retrieve relevant documents, feed them into the prompt alongside the question, and the model's answer becomes grounded in evidence rather than static training memory.
This is not a new idea. Search engines have always paired retrieval with ranking. RAG applies the same principle to generation: the model writes, but the retriever decides what it writes from.
Why this matters now
Three things changed.
First, context windows grew large enough to hold meaningful retrieved content. A 128k-token window can absorb dozens of document chunks without truncation.
Second, embedding models got good. Semantic search over dense vectors now reliably surfaces the right passage for a natural-language question, not just keyword overlap.
Third, the cost of retraining dropped from "prohibitive" to merely "expensive," but it is still too slow for knowledge that changes weekly. RAG sidesteps the retraining problem entirely. You update the documents, and the next query picks up the change.
The real payoff: enterprise assistants that read internal docs, support bots that cite the current FAQ instead of last quarter's, research tools that ground claims in papers. Anywhere the model needs to be right, not just fluent.
The architecture
The pipeline is five steps: Query, Embed, Retrieve, (optionally) Rerank, Generate.
Think of it as giving the model a librarian. The user asks a question. The query encoder translates it into a vector. The vector store acts as the library index. The retriever fetches the most relevant passages. The LLM writes an answer while quoting what the librarian handed it.
Query encoding turns the user's question into a dense vector. Retrieval runs k-NN search over your vector store (FAISS, Chroma, Pinecone, whichever) to get the top-k chunks. Reranking, if you use it, applies a cross-encoder or lexical signal to reorder those hits by actual relevance, not just embedding similarity. Generation calls the LLM with a prompt containing the question plus retrieved contexts. The response should cite the passages it used.
Each step has its own failure mode. Bad embeddings surface wrong passages. Missing reranking lets noisy results through. A prompt that does not ask for citations lets the model drift back to hallucinating. The pipeline is only as strong as its weakest stage.
A minimal example
Here is a working RAG pipeline in LangChain. It loads a text file, chunks it, indexes the chunks, and answers questions with citations.
from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
loader = TextLoader("knowledge.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(docs)
embeddings = OpenAIEmbeddings()
vector_store = Chroma.from_documents(chunks, embeddings)
retriever = vector_store.as_retriever(
search_type="similarity", search_kwargs={"k": 5}
)
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = PromptTemplate.from_template(
"Use the documents below to answer the question. "
"Cite the source chunk ID or short title for every factual claim.\n\n"
"Documents:\n{context}\n\nQuestion: {query}\nAnswer:"
)
qa = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=retriever,
return_source_documents=True,
chain_type_kwargs={"prompt": prompt},
)
res = qa({"query": "What are the early signs of glaucoma?"})
print(res["result"])
for d in res["source_documents"]:
print("SOURCE:", d.metadata.get("source", "unknown"))
The k=5 is a starting point. Tune it per domain. Use map_reduce or refine chain types when retrieved text exceeds context limits. The prompt explicitly asks for citations because without that instruction, the model will summarize without attribution and you lose the entire trust benefit of RAG.
Where RAG systems actually break
Building the pipeline is the easy part. Keeping it reliable is the engineering problem.
Bad retrieval produces bad answers. If the retriever misses the right passage, the model will still generate something confident. It fills gaps with plausible fiction. The fix is better embeddings, better chunk boundaries, and overlapping chunks with metadata (titles, dates, section headers). Retrieval recall at k is the single most important metric: what fraction of queries surface the correct document in the top-k results. If that number is below 90%, improve retrieval before touching anything else.
Hallucinations do not disappear. RAG reduces them, but contradictory or sparse context still invites the model to improvise. Requiring citations helps. Running a second verification pass (another LLM checking claims against retrieved docs) helps more. Neither eliminates the problem entirely.
Latency compounds. Every stage adds time: embedding the query, searching the index, reranking, generating. For frequent queries, cache the retrieval results. Keep hot indexes in memory. Precompute embeddings on ingest rather than at query time.
Context windows are not infinite. Pasting too many chunks into the prompt wastes tokens and dilutes signal. Summarize long documents, use extractive steps to pick the most relevant sentences, or apply map-reduce chains that process chunks independently before merging.
Security is easy to overlook. Your vector database can leak internal content if access controls are missing. Prompt injection through user inputs is a real attack surface. In regulated domains, vet retrieved documents before sending them to the model.
Indexes go stale. If you do not re-embed documents when they change, your "fresh" answers are still stale. Automate indexing, batch or streaming, and keep timestamps in metadata so you can prefer recent documents over old ones.
Engineering advice from building these systems
Chunk at paragraph level, 500 to 1,200 characters with small overlaps. Sentence-level is too granular, document-level is too coarse.
Use the best embedding model you can afford. The quality gap between a good embedder and a mediocre one shows up directly in retrieval recall, and no amount of prompt engineering compensates for retrieving the wrong passage.
Add a cross-encoder reranker when precision matters. The first pass (approximate nearest neighbor) is fast but noisy. The second pass (cross-encoder scoring the top N results) is slow but accurate. This two-stage approach gets you both speed and precision.
Force the LLM to cite source IDs and quote short passages. This is not a nice-to-have. Without citations, you cannot debug whether a wrong answer came from bad retrieval or bad generation.
Log everything: retrieved docs, query embeddings, model outputs. Evaluate retrieval recall and answer accuracy as separate metrics. A system that retrieves well but generates poorly needs prompt work. A system that generates well from bad retrievals needs index work. You cannot fix what you cannot distinguish.
Build a test set of 50 queries with expected source documents. Unit test retrieval and generation independently. This catches regressions before users do.
When RAG is the wrong tool
Not every system needs retrieval.
Small applications with no specialized or changing knowledge work fine with the model's internal knowledge. The retrieval step adds complexity and latency for no benefit.
Ultra-low-latency requirements where the additional search time is unacceptable, and caching is not an option, are better served by fine-tuning or distillation.
Systems where you cannot guarantee the quality of source documents will produce garbage answers from garbage sources. RAG does not filter for quality. It retrieves what you indexed.
The relationship between search and generation
RAG does not make a model smarter. It makes a model honest.
Search and generation are complementary skills. Search is precise but cannot synthesize. Generation is fluent but cannot verify. The combination, retrieval grounding generation, produces answers that are both articulate and factual.
The valuable question is no longer "how do we make models know more." It is "how do we make models admit what they do not know, and look it up." RAG is the current best answer to that question. It will not be the last.