Before You Build RAG, Trace One Business Question to Its Source

A service employee is waiting for an answer while a customer is on the line. Three internal documents mention the policy. One is current, one was copied into a team folder, and one contains an exception that may no longer apply.

The natural request is: “Can we put a chatbot over our documents?”

Retrieval-augmented generation, or RAG, may help. It can retrieve material from an external knowledge collection and provide that material as context for a model-generated response. But retrieval does not decide which document is authoritative, grant permission to read it, repair contradictions, or make a citation true. Those are properties of the whole knowledge journey.

Before choosing RAG, trace one recurring question from the person who asks it to the source that deserves authority, the answer they may safely use, and the route when certainty runs out. The map often reveals whether the next need is retrieval, ordinary search, workflow redesign, or basic knowledge cleanup.

<a id="begin-with-the-decision-not-the-chatbot"></a>

Begin with the decision, not the chatbot

The customer in the opening does not need “access to company knowledge.” The employee needs to decide what policy applies to this case, explain it, and know when to escalate.

That narrower statement changes the design. It identifies a user, a time-sensitive question, a consequence, and an acceptable kind of answer. It also makes clear that a fluent response is not enough.

The original RAG research combined a language model’s internal parameters with retrieved external material for knowledge-intensive tasks. The authors presented retrieval as a way to bring explicit non-parametric memory into generation while noting that provenance and updating model knowledge were open problems. See the original NeurIPS 2020 RAG paper.

Business RAG systems now take many forms, but the durable idea is similar: find relevant material, put it into the response context, and generate an answer. What matters operationally is everything hidden inside relevant material and answer.

<a id="follow-the-question-to-source-path"></a>

Follow the question-to-source path

Draw one line across seven points:

User → decision → authoritative source → permission → freshness owner → answer check → escalation

For the service question, the map might reveal that the policy owner maintains an approved document in one repository; employees may read it but not the draft annex; the owner reviews it after regulatory or product changes; an answer must show the supporting passage; and unusual cases go to a named specialist.

Each point carries a different failure.

• If the decision is vague, the system cannot know how much context or caution is enough.
• If no source owns authority, retrieval may select among contradictions without resolving them.
• If permissions are weak, a useful answer can become an information leak.
• If no one owns freshness, the system can retrieve stale text perfectly.
• If the answer cannot be checked, citations may look reassuring without supporting the claim.
• If escalation is absent, uncertainty lands on the employee or customer.

RAG can participate in this path. It does not create the path by itself.

The site’s data-readiness guide is a useful continuation when the map exposes missing ownership, quality, or preparation across the source estate. This article stays with the single knowledge journey.

<a id="locate-the-failure-before-choosing-the-intervention"></a>

Locate the failure before choosing the intervention

Different failures need different responses.

Use ordinary search when the main problem is finding a known, well-structured source. If users can search by a distinctive policy name and need the document rather than a composed answer, better indexing, metadata, filters, or navigation may be simpler and easier to verify.

Redesign the workflow when the knowledge arrives at the wrong moment or in the wrong place. If employees already know the answer but must copy it across systems, retrieval may add another interface instead of removing the handoff.

Clean up knowledge when authority is fragmented. Three contradictory policy documents do not become one truth because a model can retrieve all three. Assign ownership, archive superseded versions, mark effective dates, and resolve conflicts first.

Consider RAG when users ask varied natural-language questions, the authoritative material is identifiable, and composing a contextual answer saves meaningful effort. The system still needs access controls, retrieval and answer evaluation, visible support, monitoring, and escalation.

Consider model adaptation only for a different problem. Fine-tuning can change model behaviour or task performance; it is not the obvious way to keep a changing policy collection current or to show which source supports an answer.

This is not a build-versus-buy choice. It is a choice about where the failure actually lives.

<a id="make-trust-observable-in-evaluation"></a>

Make trust observable in evaluation

Start with real questions from the bounded workflow, stripped of personal or sensitive information as needed. Include ordinary cases, ambiguous wording, conflicting evidence, missing answers, permission boundaries, and questions that should escalate.

Evaluate the stages separately. Did retrieval bring back the authoritative passage? Did it miss a necessary exception? Did the generated answer remain faithful to the retrieved material? Did the citation support the claim? Did the system refuse or escalate when the source was absent or access was not allowed?

Then evaluate the work. Can the employee verify the answer in time? Does correction take less effort than the current process? Are uncertainty and ownership visible? Can a reported error reach the person who can repair the source or system?

Do not collapse these observations into one reassuring accuracy percentage. A set of frequent easy questions can hide rare, costly failures. A citation can point to a document that is stale or irrelevant. A correct answer can still expose information to the wrong user.

NIST’s AI RMF calls for evaluation tied to the intended context, documented test sets and metrics, human feedback routes, and performance monitoring in operation. Its AI RMF Core provides a vendor-neutral basis for deciding what evidence matters.

<a id="choose-the-smallest-useful-next-move"></a>

Choose the smallest useful next move

Return to the employee with three conflicting documents. The first improvement may not be a model. It may be naming the policy owner, retiring two copies, adding an effective date, and giving staff one reliable route.

If questions remain difficult to phrase as searches and answers still require assembling several authorised passages, a bounded RAG evaluation becomes more sensible. The scope is one user group, one consequential question family, one approved source collection, defined permissions, and a named escalation owner—not “all company knowledge.”

That scope does not guarantee success. It creates a fair test.

When wider roles, oversight, and risk decisions become the next problem, continue to the site’s AI governance guide for business leaders. Do that after the knowledge journey is visible, not instead of making it visible.

The strongest RAG proposal is not the one with the largest document count or the most fluent demo. It is the one in which a person can ask a real question, see why the source deserves trust, recognise the limit, and know where the question goes next.