RAG chatbots that don't hallucinate: a pragmatic build guide

Most businesses don't need a custom-trained model. They need their existing documents to answer questions accurately.

Retrieval-augmented generation (RAG) is the pattern that makes this work: instead of teaching facts to a model, you fetch the right facts at query time and hand them to the model to phrase.

Why RAG beats fine-tuning for knowledge tasks

Fine-tuning teaches a model a style or a behavior. It's terrible at teaching new facts - the model will happily hallucinate, and updating knowledge means retraining.

RAG keeps knowledge in a database you control. Update a policy doc, re-index, and the bot is current. No retraining, no GPU bill.

The 5 decisions that keep answers grounded

1. Chunk by meaning, not character count. Split documents on headings and paragraphs, not every 500 tokens. A chunk that contains one complete idea retrieves far better.
2. Store metadata, filter before retrieve. Tag chunks with product, version, and date. Filter the vector search by "version >= 3.2" before you ever rank - you'll cut irrelevant hits dramatically.
3. Retrieve more, rerank, then cite. Pull 20 candidates, rerank to 5 with a cross-encoder, and force the model to cite the chunk ID for every claim. No citation = the answer is thrown out.
4. Route low-confidence to a human. If the top chunk's score is below your threshold, don't guess - hand off. Your customers trust a honest "let me get a human" over a confident lie.
5. Log everything. Every query, retrieved chunks, model output, and user feedback. This is your flywheel - the logs tell you which docs to improve.

What this looks like in practice

For a recent client, this pattern took a support bot from ~60% accuracy (fine-tuned model) to 95% (RAG with citations) in three weeks - and the team can update its knowledge by editing a markdown file.

That's the promise of RAG done right: better answers, lower cost, and knowledge that stays current.