RAG Guide

Building and evaluating RAG systems. Retrieval helps grounding. It does not eliminate unsupported answers.

TypeScript / Node stacks: If you implement RAG behind a Node or Next.js service, pair this guide’s architecture sections with LangChain’s JavaScript / TypeScript documentation. Code snippets in this file are mostly Python.

Table of Contents

• Introduction
• What is RAG?
• RAG Architecture
• Components of RAG
• Building RAG Systems
• Advanced RAG Techniques
• Vector Databases
• Evaluation and Optimization
• Production Deployment
• Best Practices
• Common Pitfalls
• Resources

Introduction

What is RAG?

Retrieval Augmented Generation (RAG) is an architecture that enhances Large Language Model (LLM) responses by:

1. Retrieving relevant information from external knowledge bases
2. Augmenting the prompt with retrieved context
3. Generating responses using the LLM with the augmented context

Why RAG?

Problems RAG Helps With:

• Outdated Information: LLMs have training cutoffs; RAG can inject fresher documents
• Domain-Specific Knowledge: Use private/custom knowledge bases without retraining weights
• Grounding (not a truth layer): Retrieved text can reduce unsupported answers; the model may still ignore context, stitch sources badly, or invent between chunks. See AI Myths Busted
• Transparency: You can return retrieved passages as citations (still verify them)
• Cost Efficiency: Often cheaper than fine-tuning when knowledge changes often

Use Cases:

• Question-answering systems
• Chatbots with domain knowledge
• Document Q&A systems
• Customer support automation
• Research assistants
• Code documentation assistants

What is RAG?

Core Concept

RAG combines two key components:

1. Retrieval System:

• Vector database storing document embeddings
• Semantic search to find relevant documents
• Returns top-k most relevant chunks

2. Generation System:

• LLM (GPT, Llama, Claude, etc.)
• Takes user query + retrieved context
• Generates response based on augmented prompt

RAG vs Fine-Tuning

When to Use RAG:

• Frequently changing knowledge
• Need source citations
• Multiple knowledge domains
• Limited training data

When to Use Fine-Tuning:

• Stable domain knowledge
• Need specific output format
• Latency-critical applications
• Sufficient training data

RAG Architecture

Basic RAG Flow

Detailed Architecture

Same path with the usual pieces named:

Components of RAG

1. Document Processing

Text Splitting:

• Chunk documents into smaller pieces
• Overlap chunks for context preservation
• Consider semantic boundaries

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Size of each chunk
    chunk_overlap=200,    # Overlap between chunks
    length_function=len,
    separators=["\n\n", "\n", " ", ""]  # Split on these
)

documents = text_splitter.split_documents(your_documents)

Metadata Extraction:

• Extract document metadata (title, author, date)
• Store with chunks for filtering
• Use for source citation

2. Embedding Models

Choosing Embedding Model:

• OpenAI: text-embedding-3-small (current default path; check docs for latest)
• Hugging Face: sentence-transformers/all-MiniLM-L6-v2 (free, good quality)
• Cohere: current Cohere embed models (high quality)
• Instructor: Task-specific embeddings

from langchain_openai import OpenAIEmbeddings
from langchain_community.embeddings import HuggingFaceEmbeddings

# Option 1: OpenAI (paid)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Option 2: Hugging Face (free, good quality)
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

3. Vector Database

Popular Vector Databases:

• FAISS: Facebook AI Similarity Search (local, fast)
• Pinecone: Managed cloud service
• Weaviate: Open-source, self-hosted
• Chroma: Simple, embedded
• Qdrant: High performance
• Milvus: Scalable vector DB used in many production stacks

# FAISS (Local)
from langchain_community.vectorstores import FAISS

vectorstore = FAISS.from_documents(documents, embeddings)
vectorstore.save_local("faiss_index")

# Pinecone (Cloud). API shape changes often. Check current Pinecone + LangChain docs
from langchain_community.vectorstores import Pinecone

vectorstore = Pinecone.from_documents(
    documents, 
    embeddings, 
    index_name="rag-index"
)

4. Retrieval

Retrieval Strategies:

• Similarity Search: Cosine similarity
• MMR (Maximal Marginal Relevance): Diversity in results
• Hybrid Search: Combine semantic + keyword search
• Re-ranking: Re-rank retrieved results

# Basic similarity search
retriever = vectorstore.as_retriever(
    search_kwargs={"k": 5}  # Top 5 results
)

# MMR for diversity
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 20}
)

# With metadata filtering
retriever = vectorstore.as_retriever(
    search_kwargs={
        "k": 5,
        "filter": {"category": "technical"}
    }
)

5. Prompt Engineering

RAG Prompt Template:

• Include context in prompt
• Clear instructions for LLM
• Format for citations

from langchain.prompts import PromptTemplate

template = """Use the following pieces of context to answer the question.
If you don't know the answer, just say that you don't know, don't try to make up an answer.

Context: {context}

Question: {question}

Answer:"""

prompt = PromptTemplate(
    template=template,
    input_variables=["context", "question"]
)

Building RAG Systems

Basic RAG with Langchain

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

# 1. Load and split documents
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
documents = text_splitter.split_documents(your_documents)

# 2. Create embeddings
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

# 3. Create vector store
vectorstore = FAISS.from_documents(documents, embeddings)

# 4. Create retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# 5. Create LLM (temperature=0 lowers randomness; not a correctness guarantee)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# 6. Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    return_source_documents=True
)

# 7. Query
query = "What is machine learning?"
result = qa_chain({"query": query})
print(result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print(f"- {doc.page_content[:100]}...")

Conversational RAG

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

# Add conversation memory
memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True
)

# Conversational RAG
c>
    llm=llm,
    retriever=retriever,
    memory=memory,
    return_source_documents=True
)

# Query with conversation history
resp>"question": "What is RAG?"})
print(response["answer"])

# Follow-up question (uses conversation history)
resp>"question": "How does it work?"})
print(response2["answer"])

RAG with LlamaIndex

from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.llms import OpenAI

# Load documents
documents = SimpleDirectoryReader('data').load_data()

# Create index
index = VectorStoreIndex.from_documents(documents)

# Create query engine
query_engine = index.as_query_engine()

# Query
resp>"What is machine learning?")
print(response)
print("\nSources:")
for node in response.source_nodes:
    print(f"- {node.text[:100]}...")

Advanced RAG Techniques

1. Query Expansion

Expand query to improve retrieval:

from langchain_openai import ChatOpenAI

def expand_query(original_query):
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
    prompt = f"""Expand this query to improve search results:
    
    Original: {original_query}
    
    Expanded:"""
    expanded = llm.invoke(prompt)
    return expanded.content

# Use expanded query for retrieval
expanded_query = expand_query("ML")
results = retriever.get_relevant_documents(expanded_query)

2. Re-ranking

Re-rank retrieved results for better relevance:

from sentence_transformers import CrossEncoder

# Load re-ranker
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

# Re-rank results
pairs = [[query, doc.page_content] for doc in retrieved_docs]
scores = reranker.predict(pairs)
ranked = sorted(zip(retrieved_docs, scores), key=lambda x: x[1], reverse=True)

3. Hybrid Search

Combine semantic and keyword search:

from langchain.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever

# Semantic retriever
semantic_retriever = vectorstore.as_retriever()

# Keyword retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(documents)
bm25_retriever.k = 5

# Ensemble retriever
ensemble_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, bm25_retriever],
    weights=[0.7, 0.3]  # Weight semantic more
)

4. Parent Document Retriever

Retrieve parent documents after finding relevant chunks:

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore

# Store for parent documents
store = InMemoryStore()

# Parent document retriever
parent_retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=text_splitter,
    parent_splitter=parent_splitter
)

# Add documents
parent_retriever.add_documents(documents)

5. Self-Query Retriever

Extract metadata filters from query:

from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Metadata fields
metadata_field_info = [
    {"name": "category", "description": "Document category"},
    {"name": "date", "description": "Publication date"}
]

# Self-query retriever
self_query_retriever = SelfQueryRetriever.from_llm(
    llm=llm,
    vectorstore=vectorstore,
    document_contents="Documents about machine learning",
    metadata_field_info=metadata_field_info
)

# Query with metadata
results = self_query_retriever.get_relevant_documents(
    "What are recent papers about neural networks?"
)

Vector Databases

Comparison

FAISS Example

import faiss
import numpy as np

# Create index
dimension = 384  # Embedding dimension
index = faiss.IndexFlatL2(dimension)

# Add embeddings
embeddings = np.array([...])  # Your embeddings
index.add(embeddings)

# Search
query_embedding = np.array([...])
k = 5
distances, indices = index.search(query_embedding.reshape(1, -1), k)

Pinecone Example

from pinecone import Pinecone

# Initialize (Pinecone v3+ client; check current docs if this drifts)
pc = Pinecone(api_key="your-key")

# Create index if needed (serverless/pod options vary by account)
# pc.create_index(name="rag-index", dimension=384, metric="cosine", ...)

index = pc.Index("rag-index")

# Upsert vectors
index.upsert(vectors=[
    ("id1", [0.1, 0.2, ...]),
    ("id2", [0.3, 0.4, ...])
])

# Query
results = index.query(
    vector=[0.1, 0.2, ...],
    top_k=5,
    include_metadata=True
)

Evaluation and Optimization

Evaluation Metrics

1. Retrieval Metrics:

• Recall@K: Fraction of relevant docs in top K
• Precision@K: Fraction of retrieved docs that are relevant
• MRR (Mean Reciprocal Rank): Average reciprocal rank of first relevant doc

2. Generation Metrics:

• BLEU: N-gram overlap with reference
• ROUGE: Recall-oriented metrics
• BERTScore: Semantic similarity
• Answer Accuracy: Human evaluation

Evaluation Framework

from langchain.evaluation import QAEvalChain

# Create evaluation chain
eval_chain = QAEvalChain.from_llm(llm)

# Evaluate
predicti>"query": q} for q in questions])
eval_results = eval_chain.evaluate(
    examples=examples,
    predictions=predictions
)

Optimization Strategies

1. Chunk Size:

• Too small: Lose context
• Too large: Irrelevant content
• Optimal: 500-1000 tokens

2. Chunk Overlap:

• Prevents context loss at boundaries
• Optimal: 10-20% of chunk size

3. Top-K Retrieval:

• More docs: More context, higher cost
• Optimal: 3-5 for most cases

4. Embedding Model:

• Better embeddings = better retrieval
• Consider domain-specific models

5. Prompt Engineering:

• Clear instructions improve generation
• Include examples in prompt

Production Deployment

Architecture Considerations

1. Scalability:

• Use managed vector databases (Pinecone, Weaviate)
• Implement caching for frequent queries
• Load balance across multiple instances

2. Latency:

• Optimize retrieval (limit top-k)
• Use faster embedding models
• Implement response caching

3. Cost:

• Cache embeddings
• Use smaller models when possible
• Batch processing for non-real-time

Deployment Example

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

app = FastAPI()

class QueryRequest(BaseModel):
    question: str
    top_k: int = 3

@app.post("/rag/query")
async def query_rag(request: QueryRequest):
    # Retrieve
    docs = retriever.get_relevant_documents(
        request.question,
        k=request.top_k
    )
    
    # Generate
    result = qa_chain({
        "query": request.question,
        "context": "\n\n".join([doc.page_content for doc in docs])
    })
    
    return {
        "answer": result["result"],
        "sources": [doc.metadata for doc in docs]
    }

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Monitoring

Key Metrics:

• Query latency
• Retrieval quality
• Generation quality
• Error rates
• Cost per query

Best Practices

1. Document Processing

• Clean data before chunking
• Preserve structure (headers, lists)
• Extract metadata for filtering
• Handle different formats (PDF, HTML, Markdown)

2. Embedding Selection

• Use domain-specific embeddings when available
• Test multiple models for your use case
• Consider multilingual if needed
• Balance quality and cost

3. Retrieval Optimization

• Tune chunk size for your documents
• Use appropriate overlap
• Implement re-ranking for better results
• Add metadata filtering when possible

4. Prompt Engineering

• Be explicit about using context
• Include examples in prompt
• Specify format for citations
• Handle "I don't know" cases

5. Evaluation

• Evaluate retrieval separately from generation
• Use human evaluation for quality
• Monitor in production
• Iterate based on feedback

Common Pitfalls

1. Poor Chunking

Problem: Chunks too small or too large
Solution: Experiment with chunk sizes, preserve semantic boundaries

2. Irrelevant Retrieval

Problem: Retrieved docs not relevant
Solution: Improve embeddings, use re-ranking, expand queries

3. Context Overflow

Problem: Too much context, exceeds token limit
Solution: Limit top-k, use summarization, filter retrieved docs

4. Hallucination / unsupported answers

Problem: LLM invents facts or ignores retrieved context
Solution: Better prompts, stronger retrieval/re-ranking, citation checks, and abstain-when-unsure behavior. RAG helps; it does not eliminate this failure mode (see AI Myths Busted).

5. Slow Performance

Problem: High latency
Solution: Cache embeddings, use faster models, optimize retrieval

Resources

Libraries

• Langchain: RAG framework
• LlamaIndex: RAG and data indexing
• Haystack: End-to-end NLP framework
• Chroma: Vector database
• Pinecone: Managed vector database

Papers

• "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Lewis et al., 2020)
• "In-Context Retrieval-Augmented Language Models" (Ram et al., 2023)

Tutorials

• Langchain RAG tutorials
• LlamaIndex documentation
• Pinecone RAG guide

Tools

• FAISS: Vector similarity search
• Pinecone: Managed vector DB
• Weaviate: Vector database
• Qdrant: Vector database

Try next: Pick 20 real user questions. Measure retrieval hit rate before you touch the generator.