What is RAG and when is it worth using?

RAG (Retrieval-Augmented Generation) is a technique in which a system looks up relevant material before formulating an answer and provides it to the model as context. This lets an answer use current information outside the model’s parameters and point to sources. Lewis and co-authors described the name and core mechanism in the original 2020 RAG paper.

That is the whole idea. The rest of this article covers its consequences, costs and pitfalls.

How RAG works step by step

There are five stages. Source and index preparation repeats when the material changes; retrieval and generation repeat for each question:

1. Collecting sources. Documents, agreements, procedures, email threads and tables. This happens once, then periodically as updates arrive.
2. Segmenting and describing the material. A document is divided according to its structure and the fragments receive metadata. The effect of fragment size must be tested in evaluation; there is no one good length for every source.
3. Indexing. In a common variant, fragments receive vector representations; the index can also support word search, filters and metadata. The right mechanism depends on the questions.
4. Retrieval for a question. The system obtains candidates through vector, text or hybrid search, or a query to a structured source, and can reorder them.
5. Generating and checking the answer. The model receives the question and retrieved material. A production implementation should identify its sources and have a refusal or escalation rule when the evidence is insufficient.

An instruction alone does not guarantee that a model will comply, so you need verifiable citations, a criterion for sufficient evidence and a correct “no data” response. RAG can reduce confabulation, but it does not remove it.

RAG or fine-tuning? A decision table

RAG and fine-tuning can complement each other, but their main uses differ. RAG provides material at answer time, while fine-tuning changes model behaviour from examples; fine-tuning itself is not a reliable store of current, citable knowledge.

	RAG	Fine-tuning
Answers the question	“what is in our data”	“in what style and format to answer”
Knowledge update	refresh the index after a source changes	usually requires a new tuning cycle
Providing a source	the answer can be connected to a retrieved fragment	does not follow from tuning; requires a separate mechanism
Upfront cost	depends on sources, permissions and evaluation	depends on training data and the tuning process
Changing the base model	requires re-evaluating retrieval and answers	may require tuning and evaluation again

A practical rule is that current, citable knowledge is usually easier to provide through retrieval, while format and behaviour are shaped through instructions, examples or tuning. The techniques can be combined; the choice is confirmed by a test on the target tasks.

What determines whether RAG works well

Four areas should be tested together; their impact depends on the sources and type of question:

• State of the sources. Conflicting versions of a procedure can provide conflicting evidence. You need versioning, a document owner and a precedence rule for sources.
• How fragments are split and enriched. A fragment without context (“section 4.2 does not apply to group B customers”) is useless or harmful. Add metadata: where it came from, which version it is and whom it concerns.
• Retrieval quality. Vector similarity alone can handle proper names, numbers and symbols poorly. Hybrid search and reranking are hypotheses to test on a set of questions, not automatic remedies.
• Evaluation. A set of questions with expected answers, run after every change. Without it, “we improved it” means “we changed it and hope”.

When RAG is not the right fit

• When knowledge cannot be obtained reliably. Text can be indexed directly, and images or scans after extraction or in a multimodal pipeline. If an answer exists only in one person’s head, document it first.
• When questions require counting or aggregation. “How many customers left in June?” is a database query, not a semantic-search question. You need an analytics layer or a controlled mechanism to create and validate queries.
• When questions require calculations or traversing relationships. “Which contracts with this subcontractor end before the audit?” is better served by a query to structured data, sometimes assisted by a knowledge graph, than by fragment similarity alone.
• When a small, stable set safely fits in context. Providing it all may be simpler than building a pipeline, provided cost, permissions and answer quality pass a test.

What does a RAG implementation cost?

Our ranges as of August 2026: a pilot on a knowledge subset: 12,000–30,000 PLN net (roughly 3–5 weeks, including evaluation); a production system with integration: 30,000–120,000 PLN net; quality maintenance: 2,000–6,000 PLN net/month.

There is also the ongoing cost of models and infrastructure, depending on volume, context length, caching and retries, among other factors. A pilot lets you measure quality, user adoption and cost on your own data before making a production decision.

Frequently asked questions

Will our documents be used to train a model?

RAG does not decide this on its own. Documents can stay in your database, but retrieved fragments still go to the chosen model. Use for training, retention and processing location depend on the service, plan and agreement. Verify these terms before implementation.

Does RAG eliminate hallucinations?

It reduces them; it does not eliminate them. The model can still misinterpret a fragment or answer when it should say “I don’t know”. That is why we require source citations and measure how often the system answers without evidence.

How many sources are needed before it makes sense?

There is no minimum number. Consider the frequency and cost of retrieval, risk of a wrong answer, document quality and the cost of building and maintaining the system. A small collection may justify simpler search or providing the whole controlled context rather than a full pipeline.

---

RAG and knowledge graphs are technologies; what matters to you is the outcome we call a second brain for companies. We describe our own tools in the knowledge-graph laboratory — we built them for ourselves, so we clearly distinguish research experience from client results.