What is RAG: how AI answers from your company's data

Ask a general AI assistant about your refund policy or last year's contract, and it will either refuse or confidently invent something. A large language model knows what it saw in training. It has never seen your documents.

RAG, or retrieval-augmented generation, solves this problem. Before answering, the system finds the relevant passages in your own sources and gives them to the model. The answer is then based on your data, and it can show where each statement came from.

How RAG works

The process has two parts: preparing the knowledge in advance and answering a question.

Preparation:

1. Collect the sources: documents, wiki pages, tickets, policies.
2. Split them into small passages, often called chunks.
3. Turn each passage into a numeric representation of its meaning, an embedding, and store it in a search index.

Answering:

1. The user asks a question.
2. The system finds the most relevant passages. Good systems combine meaning-based search with ordinary keyword search, because product codes and exact names are often lost in meaning-based search alone.
3. The model receives the question and the passages, and writes an answer based on them, with links to the sources.

The model itself does not change. You can update the knowledge simply by updating the index, without any retraining.

When a company needs RAG

RAG fits well when the answer lives in text and the text changes:

• Internal knowledge. Policies, procedures and instructions that employees keep asking about.
• Customer support. Draft answers based on documentation and past tickets.
• Contracts and regulations. Finding the right clause across thousands of pages, with a link to it.
• Technical documentation. Manuals, specifications and engineering notes.

When RAG is the wrong tool

• Questions about numbers. "What were last quarter's sales?" needs a query against a database or a BI model, not a search through text. Text passages will give you an old number from an old report.
• A few stable documents. If the knowledge fits on a couple of pages, you can include it directly in the instructions to the model.
• Decisions that require a full picture. RAG finds fragments. A question such as "which of all our contracts expire this year?" requires processing every document, not retrieving a few.

What usually goes wrong

The sources are poor

This is the most common cause. If three versions of a policy exist and nobody knows which is current, the model will quote any of them. RAG does not fix data problems, it makes them visible faster. A document owner and a simple cleanup do more than a better model. The same principles apply as in any data quality audit.

The passages are cut badly

Split a document without regard for its structure, and a condition will land in one passage and its exception in another. Tables and scanned documents need separate handling, otherwise their content is lost.

Access rights are ignored

The assistant must show each person only what that person is allowed to see. If the index holds salaries and board minutes alongside everything else, a casual question can expose them. Rights have to be applied at search time, using the same permissions as the source systems.

Nobody measures quality

Without measurement, a team judges by a few impressive demos, and every change is a guess. The first complaint from users then arrives as a surprise.

The index goes stale

A document changes in the source, but the index still holds the old text. Updates need to be automatic, including removal of deleted documents.

How to check the quality

Build a test set of real questions, around 50 to 100, with known correct answers and sources. Run it after every change and look at four things:

• Retrieval. Did the system find the right passage?
• Accuracy. Does the answer say what the source says, without additions?
• Sources. Do the links lead to passages that really support the answer?
• Refusal. When the answer is not in the data, does the system say so rather than make something up?

Retrieval comes first: if the right passage was not found, the model has nothing to work with.

Where to start

1. Choose one use case with a clear owner and a real question flow.
2. Select the sources for it and clean them up.
3. Write the test set before building anything.
4. Build a pilot, including access rights from day one.
5. Give it to a small group with a way to mark bad answers, and add those answers to the test set.
6. Expand only after the quality figures hold.

You also need to decide where the system runs. If documents cannot leave your infrastructure, compare the options in on-premise or cloud for enterprise AI.

Summary

RAG lets a model answer from your data rather than from memory. The technology is well understood. Results depend mostly on source quality, access rights and regular measurement.

If you are considering this for your company, see our RAG service. And if the assistant also has to act in your systems, read about AI agents.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., 2020, the paper that introduced the term