Retrieval-Augmented Generation (RAG) is the pattern almost every serious LLM app uses in 2026. Instead of hoping the model remembers your data, you feed it relevant chunks at query time. This is the honest walkthrough of what RAG actually is, how to build one in Python, and where teams still get it wrong two years after the pattern went mainstream.
Quick answer for 2026
RAG is retrieval + prompt injection. You embed your documents into a vector database, then at query time you find the top matches and paste them into the LLM prompt as context. Use RAG when facts change often (product docs, policies, prices). Use fine-tuning when style or format must stay consistent. Use both together for the best production results.
What RAG actually is in 2026
RAG stands for Retrieval-Augmented Generation. It is a two-step pattern:
- Retrieval: Given a user question, find the most relevant chunks from your knowledge base.
- Generation: Send those chunks plus the question to an LLM. The model answers using the retrieved context.
That is the entire concept. RAG became the default LLM pattern because it solves three real problems at once:
- The model does not need to memorize your data (which it cannot do well anyway)
- You can update the knowledge base at any time without retraining anything
- You can cite sources back to the user for trust and verifiability
In 2026, RAG powers most customer support chatbots, most internal enterprise search tools, most legal and medical Q&A apps, and a growing share of BSIT capstone projects that involve LLMs.
Why RAG beats plain LLM calls for factual apps
A plain LLM call has three failure modes RAG solves:
- Hallucination: The model makes up plausible-sounding facts. RAG grounds answers in retrieved source text.
- Stale knowledge: The model was trained on data with a cutoff date. RAG gives it fresh info at query time.
- No source citations: Users cannot verify plain LLM answers. RAG lets you show which chunks the answer came from.
Prompt engineering alone cannot fix these. You can beg the model to say “I do not know” and it will still confidently invent an answer for tricky questions. RAG changes the physics: if the retrieved chunks do not contain the answer, the model is far more likely to admit it.
The three pieces of a RAG pipeline
Every RAG system in 2026, no matter how fancy, boils down to three pieces:
1. Embedding model. Turns text (documents and queries) into numeric vectors. Popular 2026 choices: OpenAI text-embedding-3-large, Cohere embed-v4, Voyage AI voyage-3, or open-source alternatives like BGE-M3 and Nomic-Embed. Same model must embed both your documents and your queries.
2. Vector store. Stores your document embeddings and finds the nearest matches to a query embedding. Popular 2026 choices: pgvector (Postgres extension), Chroma, Qdrant, Pinecone, Weaviate, and Milvus.
3. LLM. Answers the question using retrieved context. Any 2026 LLM works: GPT-5, Claude Sonnet 4.7, Gemini 2.5, Llama 3, Mistral, or a fine-tuned local model.
Real code: minimum viable RAG in Python
This is the smallest working RAG. It uses Chroma as the vector store and the OpenAI API for both embeddings and generation. Roughly 40 lines of Python.
Install the dependencies:
pip install openai chromadb tiktoken
The full script:
import os
import chromadb
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="company_docs")
# 1. Load and index your documents
docs = [
"Our return policy allows returns within 30 days of purchase.",
"Standard shipping to Metro Manila takes 2 to 4 business days.",
"We accept GCash, credit cards, and bank transfer.",
"Warranty on all electronics is 1 year from purchase date.",
]
# Embed each doc
embeddings = client.embeddings.create(
model="text-embedding-3-small",
input=docs,
).data
collection.add(
ids=[f"doc_{i}" for i in range(len(docs))],
embeddings=[e.embedding for e in embeddings],
documents=docs,
)
# 2. Query
question = "How long do I have to return an item?"
query_embedding = client.embeddings.create(
model="text-embedding-3-small",
input=[question],
).data[0].embedding
results = collection.query(
query_embeddings=[query_embedding],
n_results=2,
)
retrieved = "\n".join(results["documents"][0])
# 3. Generate answer with retrieved context
prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say so.
Context:
{retrieved}
Question: {question}
Answer:"""
response = client.chat.completions.create(
model="gpt-5",
messages=[{"role": "user", "content": prompt}],
temperature=0.1,
)
print(response.choices[0].message.content)
That is a complete RAG system. In production you replace the in-memory Chroma with a persistent vector store, add a document loader for PDFs and Word files, and wrap the whole thing in a FastAPI or Django endpoint. But the pattern above is the entire idea.
Vector stores worth using in 2026
You have five solid options in 2026. Pick based on what you already run in production:
- pgvector (Postgres extension): best pick if you already run Postgres. No new database to operate. Handles up to a few million vectors comfortably.
- Chroma: best pick for prototypes and small apps. Easy Python API, runs embedded in your process, or as a small server. Not for millions of vectors.
- Qdrant: best pick for self-hosted production workloads. Rust-based, fast, generous free tier on Qdrant Cloud.
- Pinecone: best pick for zero-ops managed vector search. More expensive than self-hosted but you never touch infrastructure.
- Weaviate: best pick when you want hybrid search (keyword + vector) built in.
For a BSIT capstone or personal project, start with pgvector or Chroma. Both are free and cover 95% of learning use cases.
Chunking strategy that actually works
Chunking is where most RAG systems silently fail. If your chunks are wrong, retrieval will surface useless context and the LLM will answer badly no matter how good the model is.
The rules that hold up in 2026:
- Chunk by semantic unit, not raw character count. Split on paragraphs, section headings, or Markdown headers first. Fall back to character splitting only inside long paragraphs.
- Target 300 to 800 tokens per chunk. Too small and each chunk lacks context. Too large and the LLM drowns in irrelevant text.
- Add 10 to 20 percent overlap between chunks. Prevents important context from getting split across chunk boundaries.
- Preserve document structure metadata. Store the source URL, section title, and page number with each chunk so you can cite them in the answer.
- For PDFs with tables and figures, use a parser that understands layout (LlamaParse, Unstructured.io, or Docling). Naive PDF text extraction destroys tables.
Common RAG mistakes and when RAG is not the answer
These are the mistakes that show up in real 2026 RAG projects:
- Retrieving too few chunks. Top-1 retrieval loses when the answer spans multiple chunks. Default to top-3 to top-8 and let the LLM sort through them.
- Retrieving too many chunks. Above top-15, the LLM starts missing the actual answer buried in noise. Rerank aggressively.
- Using different embedding models for indexing and querying. Silent quality killer. Always use the same model on both sides.
- Not evaluating retrieval quality. Track precision and recall against a labeled test set. A great LLM cannot fix bad retrieval.
- Skipping metadata filtering. If you have per-user, per-tenant, or per-department data, use metadata filters at query time instead of hoping semantic search sorts it out.
When RAG is not the right answer: if your task needs a specific format (JSON schema, XML, custom DSL), consistent tone or brand voice, or a compressed prompt for cost reasons, fine-tune instead. If your task is math, code execution, or multi-step planning, use tool calling and agents. RAG is for factual Q&A over documents. Do not force it onto tasks it was not designed for.
Official documentation
Where to run your RAG pipeline in production
The links below are affiliate links. We may earn a commission at no extra cost to you if you purchase through them. See our affiliate disclosure for details.
- OnSpace AI for hosting your RAG API and LLM inference endpoint
- Kamatera cloud for running Postgres with pgvector or self-hosted Qdrant
- Rheinwerk Publishing for LLM engineering and applied AI books
Frequently asked questions
Do I need LangChain or LlamaIndex to build a RAG system?
No. The 40-line example above uses neither. Frameworks help when your pipeline grows past 10 files (document loaders for many formats, retrieval strategies, evaluation harnesses). For learning, build one from scratch first. You will understand exactly what LangChain and LlamaIndex are doing when you use them later.
How much does a small RAG app cost to run in 2026?
A small RAG app (10,000 documents, 1,000 queries per day) costs roughly $5 to $30 per month in 2026. Embeddings are cheap (roughly $0.02 per million tokens with OpenAI text-embedding-3-small). LLM inference is the largest cost: about $0.001 to $0.005 per RAG query depending on model. Self-host with Ollama plus Llama 3 8B and it drops to just server costs.
Is RAG a good BSIT capstone project in 2026?
Yes. RAG over a specific corpus (school handbook, LGU services, DTI regulations, Tagalog customer support scripts) is defendable in front of panels, uses skills employers actually value, and produces a working app your school can keep. Bonus points if you evaluate retrieval quality with a labeled test set and cite sources in the answers.
What is the difference between RAG and fine-tuning?
RAG injects fresh facts at query time. Fine-tuning bakes patterns into the model’s weights. Use RAG for facts that change (product docs, prices, policies). Use fine-tuning for consistent style, format, or specialized behavior. Most production LLM apps use both: fine-tuned model for tone plus RAG for current facts.
Can RAG work with local open-source LLMs?
Yes. Swap the OpenAI client for Ollama or vLLM serving Llama 3, Mistral, or Qwen. Use a local embedding model like BGE-M3 or Nomic-Embed. Cost drops to zero after hardware, and your data never leaves your server. Trade-off: smaller local models produce lower-quality answers than GPT-5 or Claude on complex questions.
How do I evaluate whether my RAG system is any good?
Build a labeled test set of 50 to 200 question-answer pairs from your actual users or realistic scenarios. Track two metrics separately: retrieval precision (did the right chunks come back?) and answer quality (was the final response correct?). Tools like Ragas, DeepEval, and LangSmith automate this scoring in 2026.
Teams often store their RAG document corpus, prompt templates, and evaluation notes in Notion or an equivalent team wiki so multiple engineers can iterate on the same knowledge base. For the theory behind embeddings and vector search, courses on Coursera and DataCamp cover the foundations with hands-on notebooks.
Bottom line for 2026 AI developers
RAG is not magic. It is embedding plus vector search plus prompt injection. The 40-line Python example above is the whole concept. Everything else you will read about (advanced retrievers, rerankers, HyDE, multi-query, agentic RAG) is optimization on top of the same three pieces.
Start simple. Build the 40-line version on your own documents. Measure whether the answers are good enough for your users. Add complexity only when you have a specific problem the simple version cannot solve. Feel free to comment below if you get stuck on chunking or retrieval quality on your own project.
Adones Evangelista
Programmer & Technical Writer at PIES IT Solution
Adones Evangelista is a programmer and writer at PIES IT Solution, author of over 900 tutorials and error-fix guides at itsourcecode.com. Specializes in JavaScript, Django, Laravel, and Python error debugging covering ValueError, TypeError, AttributeError, ModuleNotFoundError, and RuntimeError, plus C/C++ and PHP capstone projects for BSIT students.
Expertise: JavaScript · Python · Django · Laravel · Error Debugging · C/C++ · View all posts by Adones Evangelista →