RAG in Production: What Actually Works

Retrieval Augmented Generation sounds simple: chunk your documents, embed them, retrieve the relevant ones, feed them to an LLM. In practice, every step hides decisions that determine whether your system gives useful answers or confidently wrong ones.

This is what we learned building RAG systems that process millions of documents in production.

The retrieval problem nobody talks about

Most teams spend their time tuning the LLM: adjusting temperature, rewriting system prompts, testing different models. This is a mistake.

The first thing we do on any RAG project is measure retrieval quality independently. Before we even look at the LLM output, we check: did the retrieval step return the right documents? If the answer is no, no amount of prompt engineering will fix it.

Chunking matters more than you think

There is a study from Vectara (presented at NAACL 2025) that surprised us: chunking configuration affects retrieval quality as much as embedding model choice. Up to 9% recall difference between best and worst chunking strategy, with the same embedding model.

Most teams use whatever default their framework provides and never revisit it. Here is what we found works:

# What most tutorials show (and what breaks in production)
chunks = text_splitter(chunk_size=1000, overlap=200)
# What actually works: smaller chunks, semantic boundaries
chunks = recursive_splitter(
chunk_size=512,
chunk_overlap=64, # 10-15% overlap is enough
separators=["\\n\\n", "\\n", ". ", " "], # respect document structure
)
# The real unlock: enrich chunks with metadata
for chunk in chunks:
chunk.metadata = {
"source": doc.filename,
"section": extract_heading(chunk),
"doc_type": classify_document(doc),
"date": doc.created_at,
}

Hybrid search is not optional

Pure vector search misses exact keywords. Pure keyword search (BM25) misses semantic similarity. You need both.

A customer asked us to build a search system over their compliance documents. With vector search alone, the query "FINMA circular 2017/1" returned documents about general regulatory frameworks. It understood the meaning but missed the specific reference. Adding BM25 keyword search alongside vector search fixed this immediately.

1. User query comes in
2. Run BM25 keyword search → top 20 results
3. Run vector similarity search → top 20 results
4. Merge and deduplicate
5. Rerank with cross-encoder → top 5 results
6. Feed to LLM with context

The reranking step (step 5) deserves special attention. A cross-encoder reranker evaluates each query-document pair together, which is much more accurate than the initial retrieval. This single step consistently delivers the highest quality improvement per engineering hour invested.

When RAG is the wrong answer

Not everything needs RAG. We have talked clients out of building RAG systems when a simpler solution was better.

Your corpus is small and static? Consider Cache Augmented Generation (CAG). You load the entire corpus into the LLM context window. No chunking, no embeddings, no vector database. In benchmarks, CAG runs 40x faster than RAG for cacheable corpora and produces more consistent answers because there is no retrieval step to fail.

Your problem is behavior, not knowledge? Fine-tuning is better when you need the model to write in a specific format, follow a domain-specific style, or perform a narrow skill. RAG adds knowledge. Fine-tuning changes behavior. Confusing the two wastes months.

Your users need exact answers from structured data? A SQL query or API call is faster, cheaper, and more reliable than any RAG pipeline. We have seen teams build elaborate RAG systems over structured databases when a simple query interface would have been better.

Vector databases: what we actually use

The vector database market is crowded and confusing. Here is the practical reality:

Under 10 million vectors, almost everything works. pgvector in your existing Postgres database is fine. No need for a separate service, no additional ops burden, no vendor lock-in. We default to this unless there is a specific reason not to.

Between 10M and 100M vectors, Qdrant offers the best combination of latency (around 4ms at p50), filtering capabilities, and cost. It is roughly 3-5x cheaper than Pinecone at scale.

Above 100M vectors, you need to think carefully about architecture. Costs diverge dramatically at this scale, from $800 to $7000 per month depending on your choice. Milvus or Zilliz are designed for this range.

The architecture that survives production

After building several RAG systems, we converged on a pattern that handles most use cases:

class ProductionRAG:
    def answer(self, query: str) -> str:
        # Step 1: Classify the query
        query_type = self.router.classify(query)
        
        # Not all queries need retrieval
        if query_type == "general_knowledge":
            return self.llm.generate(query)
        
        if query_type == "structured_data":
            return self.sql_agent.query(query)
        
        # Step 2: Hybrid retrieval
        keyword_results = self.bm25.search(query, top_k=20)
        vector_results = self.vector_db.search(
            self.embedder.encode(query), top_k=20
        )
        
        # Step 3: Merge + Rerank
        candidates = deduplicate(keyword_results + vector_results)
        ranked = self.reranker.rank(query, candidates, top_k=5)
        
        # Step 4: Generate with context
        context = format_context(ranked)
        return self.llm.generate(query, context=context)

The query router in step 1 is underappreciated. A significant portion of user queries do not actually need retrieval at all. Questions like "summarize this document" or "what does this term mean" can be answered directly by the LLM. Routing these away from the retrieval pipeline saves cost and reduces latency.

What we would do differently

If we started a new RAG project today, we would spend the first week entirely on data quality and chunking experiments. No LLM integration, no UI, no demo. Just: load the documents, try different chunking strategies, measure retrieval recall against a manually curated test set of 50 questions.

The teams that skip this step end up spending months debugging "LLM issues" that are actually retrieval issues. The teams that invest here ship faster and with higher quality.

RAG is not a solved problem. Context windows are getting larger, and the boundary between "retrieve and generate" and "just generate" is shifting. But for now, if you have more than a few hundred pages of domain-specific content that changes over time, a well-built RAG pipeline remains the most reliable approach.

If you are building a RAG system and want to avoid the mistakes we made, get in touch. We have strong opinions about chunking strategies.