Enterprise RAG: where to start

A RAG always impresses in the demo: you ask a question, it answers by citing your documents, the room is convinced. Three weeks later in production, it cites the wrong paragraph, mixes up two versions of the same procedure, or misses on a question everyone asks. The model has not changed in the meantime. What changed is that it now faces real questions on real documents, and the invisible part of the system, retrieval, was not given the same care as the part that writes. Most RAGs that go off the rails got the order of priorities wrong. Here is how I build one that holds, step by step, in the order I work through them.

Where do you start a RAG project without getting it wrong?

With the data, not the model. The principle of a RAG comes in two steps: I retrieve the relevant passages from your documents, then ask the model to answer from those passages, with the sources. In 70 to 90% of cases, the quality of the answer is decided by that first step, retrieval, not by the model that writes. It is counterintuitive, because the model is the visible, spectacular part. But a brilliant model fed bad passages produces a wrong answer that reads well, which is worse than a visible error. So you start with the corpus: how ready your data is already weighs heavily on the result, before you have even chosen a model.

Why clean the documents before plugging in the model?

Because a dirty corpus produces wrong answers stated with confidence. The model does not sort things out: it answers from what it is served, without asking whether the source is current or reliable. If your documents hold three versions of a procedure with no dates, it will draw from the wrong one without flagging it, and that is exactly the ground that makes it answer confidently and wide of the mark. Before anything else, I sort, deduplicate and date. It is the most thankless work in the project, the part everyone wants to skip, and it is the one that decides the final reliability.

• Remove duplicates and outdated versions, or date them so the system knows which one prevails.
• Extract the text cleanly from PDFs, scans and tables, then check what comes out.
• Attach metadata to each document: source, date, department, confidentiality level.
• Decide who is allowed to see what, before the system surfaces a passage to a user who should not read it.

How do you chunk the documents well?

By respecting logical units, not cutting every 500 characters. A RAG does not retrieve whole documents, it retrieves pieces, chunks, and it is a chunk it hands to the model. If you cut at random, an idea starts in one chunk and ends in the next: each passage becomes half-understandable, and the model receives mangled fragments it completes in its own way. I chunk by units that keep their meaning on their own: a section, a paragraph, a procedure, a clause. A table stays with its title, a step stays with its context. It depends on your documents, it is not automatic by default, and it is often where you gain the most quality for the least effort.

How do you retrieve the right passages?

This is the heart of the system, and a single technique rarely does the job. Semantic search (embeddings) finds a passage even when the question does not use the same words as the document. It has a blind spot: exact references, a contract number, a product code, an internal acronym, which keyword search picks up effortlessly. Combining them gives what is called hybrid search, and it is often the right default. Then you rerank the retrieved passages to bring the most relevant to the top, you filter by metadata when the question is about a specific period or department, and you require every answer to cite its sources. A RAG you cannot verify is not worth much more than the model on its own.

• Semantic search: finds passages close to the intent, even phrased differently.
• Keyword search: catches the exact references that meaning alone lets slip.
• Reranking: pushes the genuinely relevant passages to the top of what was retrieved.
• Metadata filters and source citations: narrow the search, and make every answer verifiable.

How do you know if the RAG answers correctly?

By testing it on real business questions, not the three examples from the demo. I build a set of questions your teams actually ask, each with the right answer and the right source document, then I measure two things separately: did the system retrieve the right passage, and did the model answer well from it. That distinction tells you where to fix and saves days spent rewriting a prompt when the problem is elsewhere. Most of the time it is upstream, so I iterate on the indexing, not on the prompt. And because a corpus lives and changes, measuring the RAG continuously is what keeps it from drifting silently after a document update. When an answer goes wrong, I work back up the chain in this order:

• Was the right passage retrieved at all? If not, the problem is in the chunking or the search, never in the prompt.
• Was the passage understandable on its own, or cut in the middle? If it is mangled, it is the chunking.
• Was the source document up to date? If not, it is the corpus that needs cleaning.
• Did the model have the right passage and still drift? There, and only there, do you touch the prompt or the model.

Do you always need a RAG?

No, and the question should be settled before you start, not after six weeks of development. RAG is the right tool when the knowledge lives in documents that change, that need to be cited, and that you want to update without retraining anything. To teach a style, an output format or a recurring line of reasoning, it is not the tool, and the choice between RAG and fine-tuning deserves to be laid out plainly rather than settled by habit. When RAG is the right choice, I start narrow: a single corpus, the one your teams consult most, clean and well chunked. A tight scope that answers correctly beats a large volume that answers halfway.

An LLM does not fix bad retrieval. It dresses it up and makes it credible.

A RAG is not a model plugged into documents. It is a chain: prepare the corpus, chunk it right, retrieve the right passages, cite, measure. The model is the last piece, and almost always the simplest. If yours impresses in the demo and disappoints in production, the sticking point is almost certainly upstream of the model, in one of the steps someone rushed. That is exactly the kind of project I take over: let's talk.