Retrieval-augmented generation, or RAG, is the technique behind most “chat with your documents” tools. When you ask a question, the system first searches your own material for the most relevant passages, then hands those passages to a language model with an instruction to answer from them. The model writes the reply; your documents supply the facts.
It is the most practical way for a small business to get AI answers grounded in its own policies, price lists, manuals and contracts, and it is cheap to run. It is not a cure for made-up answers. Independent testing of commercial RAG tools has found they still get a meaningful share of answers wrong, so a person needs to check anything that matters.
- RAG has two stages: retrieve the right passages, then generate an answer from them. Errors can creep in at either stage.
- A Stanford study of legal research tools built on RAG found they produced incorrect or misleading answers on more than 17% of test queries, against 58% to 82% for general chatbots in earlier work.
- Running costs are usually small: OpenAI lists its smaller embedding model at $0.02 per million tokens (prices checked 22 November 2026, US dollars, no UK prices published).
- Off-the-shelf tools suit most firms. Build your own when answers must come from live systems, respect user permissions or sit inside your product.
Why AI needs RAG at all
A language model knows what was in its training data, up to a cut-off date. It does not know your returns policy, last month’s price rise or the terms of a particular client contract. Ask anyway and it will often produce a plausible guess, which is how most AI hallucinations happen.
There are three ways to give a model your information:
- Paste it in. Fine for one document. Current models accept long inputs, but you pay for every word each time, and it doesn’t scale to a whole shared drive.
- Retrain the model on it. Known as fine-tuning. It is good at teaching a style or format, poor at teaching facts that change, and costly to repeat whenever a document is updated.
- Retrieve it when needed. That is RAG. Only the relevant passages are sent with each question, and updating your knowledge means updating a document, not a model.
The approach was named in a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, which combined a model with a searchable index of Wikipedia. The basic recipe has barely changed since; the tools around it have become much easier to use.
How RAG works, step by step
Step 1 of 6
Collect
Gather the documents the AI may use, such as policies, manuals, FAQs and contracts.
Step 2 of 6
Chunk
Split each document into short passages of a few paragraphs, keeping headings attached.
Step 3 of 6
Embed
Convert each passage into a list of numbers that captures its meaning, and store it in a vector index.
Step 4 of 6
Retrieve
Turn each question into numbers too and pull back the passages closest in meaning.
Step 5 of 6
Generate
Send the question and those passages to the model with an instruction to answer only from them.
Step 6 of 6
Cite
Show which passages were used, so a person can check the answer against the source.
The “embedding” step is what makes RAG better than keyword search. A question about “sending something back” can find a passage headed “Returns”, because the two mean similar things even though they share no words. Many systems combine this with ordinary keyword search, which helps with product codes, names and exact phrases that meaning-based search can miss.
The quality of the answer depends far more on the retrieval step than most people expect. If the right passage isn’t retrieved, the model can’t use it, and it may fill the gap from its general knowledge instead. If an out-of-date version of a policy is retrieved alongside the current one, the model may blend them.
Why RAG still gets things wrong
RAG is often sold as the fix for hallucination. The best independent evidence says it reduces the problem rather than removing it.
In 2024, researchers at Stanford’s RegLab and Institute for Human-Centered AI tested leading legal research tools from LexisNexis and Thomson Reuters, all of which use RAG over databases of case law, on more than 200 legal questions. Their preprint reports that the tools each hallucinated between 17% and 33% of the time. The best performer, Lexis+ AI, answered 65% of queries accurately. These tools did better than general-purpose chatbots, which an earlier Stanford study found hallucinated on 58% to 82% of legal queries.
The authors’ conclusion is a fair summary for any business: legal RAG “can reduce hallucinations compared to general-purpose AI systems”, but hallucinations “remain substantial, wide-ranging, and potentially insidious”. Law is a hard case, with long documents and fine distinctions. Your staff handbook is easier. The lesson still applies: the more an answer matters, the more a person should check it against the cited source.
The common causes of wrong answers in a business setting are:
- Messy source material. Three versions of the same price list, scanned PDFs with no readable text, or tables that break when split into chunks.
- The answer isn’t in the documents. A good system should say “I don’t know”. Many will guess unless told firmly not to.
- Missing context. A chunk that says “this applies to all orders” is useless if the heading saying which country it covers was cut off.
- Permission leaks. If HR files are in the same index as the sales FAQ, the assistant can quote one to anyone who asks the right question.
Your options, from ready-made to bespoke
You may not need to build anything. Most major AI assistants now let you upload files to a project or workspace and ask questions of them, and Microsoft and Google offer similar features across their office suites. The comparison below is based on published documentation and prices, not hands-on testing. Hahu has no affiliate or commercial relationship with any vendor named.
| Option | What it is | Published limits or prices | Best fit if |
|---|---|---|---|
| Gemini Notebook (Google, previously NotebookLM) | Upload documents to a notebook and ask questions with citations | 50 sources per notebook free; 100 on Plus, 300 on Pro, up to 600 on Ultra. Google says limits are subject to change. | A small team needs to research a defined set of documents |
| Projects in ChatGPT, Claude and similar | Files attached to a shared workspace inside the assistant | Varies by plan; check each vendor’s current help pages | Staff already use that assistant and the files are fairly static |
| Developer file search (for example OpenAI’s) | A hosted search index that a developer connects to an app | OpenAI: $2.50 per 1,000 searches and $0.10 per GB of storage a day, first GB free | You want RAG inside your own app without running a database |
| Custom build | Your own pipeline using an embedding model and a vector database | OpenAI embeddings from $0.02 per million tokens, plus hosting and development | Answers must respect user permissions, use live data or be part of your product |
Prices checked 22 November 2026. OpenAI publishes these prices in US dollars only, before any VAT. Gemini Notebook’s higher limits come with Google’s AI Plus, Pro and Ultra subscriptions, so check Google’s current UK prices before you choose.
What a RAG system costs to run
The retrieval part is cheap. At OpenAI’s listed $0.02 per million tokens for its text-embedding-3-small model, converting even a few million tokens of documents into an index costs cents. The larger text-embedding-3-large model is listed at $0.13 per million tokens. You re-embed only the documents that change.
The bigger running cost is the generation step, because each question sends the retrieved passages to the model as input. Five passages of a few hundred words each, on every question, add up across thousands of questions a month. Our guide to what AI costs to run shows how to estimate token bills. For a custom build, development and maintenance usually dwarf the model costs; see what it costs to keep an app running for the ongoing side.
When to build your own
An off-the-shelf tool is the right start for most small firms. A custom system makes sense when:
- Answers depend on live data, such as stock levels, order status or a client’s account in your CRM, not just static documents.
- Different people may see different things. Retrieval needs to respect the same permissions as your file system, so a junior member of staff can’t retrieve board papers.
- It is part of what you sell. If customers will use it inside your product, you need control over quality, cost and data handling. Our guide to building a SaaS product covers the wider journey.
- You need an audit trail showing which source supported each answer, for regulated work.
Whichever route you take, personal data in your documents is still personal data. Check where the supplier stores and processes it, whether it is used for training, and how long it is kept. Buyers of a SaaS product will ask exactly these questions, as covered in security and compliance for a UK SaaS.
Common questions
Is RAG the same as training AI on my data?
No. Training (fine-tuning) changes the model itself. RAG leaves the model unchanged and sends it relevant passages with each question. That makes RAG easier to update and easier to check, because answers can cite their sources.
Do long context windows make RAG unnecessary?
Not for most businesses. You can paste a long document straight in, but you pay for every word on every question and the model can still miss details buried in the middle. RAG sends only what is relevant, which is cheaper and usually more accurate across a large library.
How do I stop it making things up?
Clean up your source documents first, tell the model to answer only from the passages provided and to say when it doesn’t know, show citations, and test it with real questions whose answers you already know. Expect to keep checking important answers.
Which documents should we start with?
The ones staff and customers ask about most and that change least: product FAQs, policies, process guides and onboarding material. Remove duplicates and old versions before you upload anything.
Where to start
Pick one set of documents, clean it up, try an off-the-shelf notebook or project tool for a month and keep a list of questions it got wrong. If it needs live data, permissions or a place inside your own product, that is a build job, and the kind of thing our SaaS product development team does.
Sources
- Stanford HAI: AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries (23 May 2024, updated 30 May 2024)
- OpenAI: API pricing, embeddings and file search (checked 22 November 2026)
- Google: Gemini Notebook Help, upgrade and usage limits (checked 22 November 2026)
- Magesh, Surani, Dahl, Suzgun, Manning and Ho: Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv 2405.20362 (2024)
- Lewis and others: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv 2005.11401 (2020)