Building Reliable RAG Systems: Architecture and Evaluation

Why RAG instead of a bigger prompt

Language models answer from what is in their context window. Retrieval-augmented generation (RAG) is the practice of finding the right documents at query time and putting them in that context window, so the model answers from your data rather than from memory.

Compared with training a custom model, RAG has three practical advantages: content can be updated the moment a document changes, answers can cite the source they came from, and access control stays in your own systems instead of being baked into model weights.

The pipeline, stage by stage

1. Ingestion and normalisation

Everything downstream depends on clean text. Convert each source format (PDF, HTML, Markdown, tickets, wiki pages) into plain text plus metadata, and keep the metadata you will later want to filter on: source system, document id, title, section heading, author, last-updated timestamp, and access scope.

Store the extracted text separately from the original file. Re-extraction is common — parser improvements, new document types, corrected encodings — and you do not want to re-download the corpus each time.

2. Chunking

Chunking decides what a "unit of retrieval" is. Two rules of thumb serve most corpora:

• Split on structure first (headings, sections, list boundaries), then on length. Structural splits keep a chunk about one thing.
• Overlap adjacent chunks slightly so a sentence spanning a boundary is still retrievable from both sides.

Very small chunks retrieve precisely but lose context; very large chunks carry context but dilute the embedding and burn tokens. Treat chunk size as a tunable parameter and measure it rather than guessing once.

Always keep a pointer from a chunk back to its document and position. That pointer is what lets the UI show "from Onboarding Guide, section 3".

3. Indexing

Embed each chunk and store the vector alongside its metadata. Two properties matter more than the choice of database:

• Filterable metadata. Most real queries are scoped: this workspace, this product version, documents the user may read. Filtering during search is far cheaper and more correct than filtering afterwards.
• Reindex ergonomics. You will change embedding models. Version your index and support building a new one in the background, then switching atomically.

4. Retrieval

Pure vector search is a good default and a poor complete answer. Dense embeddings capture meaning but often miss exact tokens: error codes, SKUs, function names, unusual acronyms. Keyword (BM25-style) search catches exactly those. Running both and merging the result lists — hybrid retrieval — is usually a bigger quality win than swapping embedding models.

After merging, a reranker scores each candidate against the query with a model that reads the query and passage together. Retrieval can then be generous (say, dozens of candidates) while generation stays selective (a handful of passages), which is the combination that tends to help recall without polluting the prompt.

5. Generation

Build the prompt so the model's job is unambiguous: answer only from the supplied passages, cite the passage ids used, and say when the passages do not contain the answer. Number the passages and ask for those numbers back — it makes citations checkable and makes hallucinated sources obvious.

Two habits keep this stage honest:

• Include the metadata (title, date) with each passage so the model can prefer newer material and so users can judge the source.
• Keep an explicit "insufficient context" path. A system that admits ignorance is far more useful than one that guesses fluently.

Evaluating the thing you built

RAG failures are usually retrieval failures wearing a generation costume. Evaluate the stages separately.

Retrieval. Build a set of real questions with the passages that should be retrieved for each. Track recall@k (did the needed passage appear at all) and the rank at which it appeared. Recall bounds everything: if the passage never arrives, no prompt fixes the answer.

Generation. Given fixed passages, check three things: is the answer supported by them (groundedness), does it address the question (relevance), and does it abstain when the passages are insufficient.

End to end. Measure what users feel: answer correctness on a held-out question set, latency at the tail rather than the mean, and cost per answered question.

Keep the evaluation set in version control next to the code, and grow it from production traffic: every reported bad answer becomes a test case. A modest, curated set that runs on every change beats a large one that runs once.

Operational concerns that decide success

• Freshness. Ingest incrementally on document change events rather than rebuilding nightly, and surface the "last updated" date in answers.
• Permissions. Apply the user's access scope as a retrieval filter. Post-filtering leaks through citations and token counts.
• Observability. Log the query, the retrieved chunk ids and scores, the final prompt size, and the answer. Debugging RAG without those logs is guesswork.
• Cost control. Cache embeddings of unchanged chunks, cache answers for repeated queries, and cap the number of passages per request.

A pragmatic build order

1. Ship the simplest pipeline that works end to end: structural chunking, one embedding model, vector search, a strict prompt with citations.
2. Assemble twenty to fifty real questions with known answers and measure it.
3. Add hybrid retrieval, then a reranker, and re-measure after each change.
4. Only then tune chunk sizes and prompts, using the same question set.

The order matters because each step's value is only visible once you can measure the step before it. A RAG system is less a model problem than an information-retrieval problem with a language model at the end — and information retrieval rewards measurement.