RAG architecture for production systems

The components of a production RAG system

1. Connectors pull content from sources – docs, wikis, tickets, PDFs, databases – along with metadata: source, author, updated time, and access permissions.
2. Parsing and cleaning turns each format into clean text with structure preserved: headings, tables, lists, code blocks. Bad parsing is the most underrated cause of bad answers.
3. Chunking splits documents into retrievable units with enough context to stand alone.
4. Embedding and indexing stores each chunk with its vector, its text for keyword search, and its metadata.
5. Query processing rewrites the user’s question into one or more search queries, using conversation history where needed.
6. Retrieval runs hybrid search with permission and metadata filters.
7. Reranking reorders candidates with a more accurate model and keeps the best few.
8. Generation produces an answer from the selected chunks, with citations and an explicit path for “not found”.
9. Evaluation and monitoring measure retrieval and answer quality offline and in production.

Ingestion: chunking and metadata

Chunking is where most RAG implementation decisions hide. Fixed-size chunks with overlap are a reasonable baseline; structure-aware chunking – split on headings, keep tables and code blocks whole – is usually better. Prepend each chunk with its document title and heading path (“Billing › Refunds › Annual plans”) so a chunk that says “this applies after 30 days” still means something in isolation.

Store metadata you’ll filter on: tenant, access groups, product, version, language, document type and updated time. Make ingestion incremental and idempotent – detect changed and deleted documents and update only those – or your index will drift from reality and nobody will notice until a customer quotes an old policy back at you.

Keep the original documents and your chunking code versioned. When you change chunk size or embedding model, you’ll need to re-index everything and compare the results.

Retrieval: hybrid search and reranking

Pure vector search is good at meaning and bad at exact tokens: error codes, SKUs, function names, people’s names. Keyword search (BM25) is the opposite. Run both and merge the results – reciprocal rank fusion is a simple, effective way to combine the two ranked lists.

Then rerank. Retrieve a wide candidate set (say 50), score each against the query with a cross-encoder or reranking model, and keep the top handful. Reranking is often the single biggest quality improvement available, and it lets you send less context to the model, which cuts cost.

On storage: if you already run PostgreSQL, pgvector plus Postgres full-text search covers a lot of ground and keeps permissions, metadata and vectors in one transactional store. A dedicated vector database earns its place at large scale or when you need features Postgres doesn’t give you. Don’t start with three new pieces of infrastructure for a few thousand documents.

Permissions and multi-tenancy

In a SaaS product, RAG must never retrieve content the user isn’t allowed to see. Enforce access at retrieval time with filters on tenant and access-group metadata – not by asking the model to ignore things, and not by filtering after generation when the content has already influenced the answer.

Sync permission changes as carefully as content changes. A document whose sharing was revoked yesterday must stop appearing today. If your sources have complex permissions (per-document ACLs in a wiki, private channels), budget real engineering time for this; it’s usually the hardest part of a multi-tenant RAG system.

Generation: grounding and citations

• Tell the model to answer only from the provided passages and to say clearly when they don’t cover the question.
• Give each passage an ID and require citations as structured references. Verify in code that every cited ID was in the context.
• Order passages deliberately – most relevant first – and keep the count small. More context is not more accuracy; models miss details in long, noisy contexts.
• Pull live values (balances, prices, statuses) from tools or APIs rather than from indexed documents, which may be stale.
• For multi-step questions, let the model issue follow-up searches (agentic retrieval) – but cap iterations and measure whether it actually helps on your eval set.

The controls around generation are covered in more depth in LLM guardrails.

RAG failure modes

• The answer exists but wasn’t retrieved. The most common failure. Causes: poor parsing, chunks split mid-thought, vocabulary mismatch between user and docs, no keyword search, too small a candidate set.
• Retrieved but ignored. The right chunk was in context, buried among irrelevant ones. Rerank and send fewer passages.
• Conflicting sources. Old and new versions of a policy both retrieved. Filter by version and recency, and delete or archive superseded content.
• Stale index. Ingestion broke silently a week ago. Monitor freshness per source.
• Permission leaks. Missing tenant filter on one code path.
• Confident answers with nothing retrieved. The model fills the gap from general knowledge. Detect low retrieval scores and abstain.
• Question needs aggregation. “How many customers churned last quarter?” is a database query, not a retrieval task. Route it to a tool.

How to evaluate a RAG system

Evaluate retrieval and generation separately, or you won’t know which one to fix.

• Build a test set of 100–300 real questions with the document(s) that answer them and a reference answer. Pull them from search logs, support tickets and subject-matter experts. Include questions that should return “not covered”.
• Retrieval metrics: recall@k (is a correct chunk in the top k?) and mean reciprocal rank. If recall@k is poor, no prompt will save you.
• Answer metrics: correctness against the reference, faithfulness (every claim supported by cited passages), citation accuracy, and correct abstention. LLM-as-judge works here if you calibrate it against human labels first.
• Operational metrics: latency per stage, tokens per answer, index freshness.
• Run the suite on every change to chunking, embedding model, retrieval parameters, prompt or generation model.

I’ve written a full guide to LLM evaluation, including how to keep a judge model honest.

RAG implementation: a sensible order

1. Collect the evaluation questions first.
2. Ingest one source well, with clean parsing and metadata.
3. Ship hybrid retrieval with permission filters and measure recall.
4. Add reranking; measure again.
5. Add generation with citations and abstention.
6. Add sources one at a time, re-running the eval each time.
7. Monitor freshness, retrieval scores and user feedback in production.

If retrieval isn’t your bottleneck – if the model needs to learn a style or format rather than facts – read RAG vs fine-tuning first. If you want help building one, that’s my RAG development work.

Related

• RAG development services, measured on retrieval quality
• An AI knowledge base your team can actually trust
• RAG vs fine-tuning: how to choose
• LLM evaluation for product features
• LLM guardrails for real product features
• AI for customer support that actually resolves tickets