A plain walk-through of retrieval-augmented generation, a minimal pgvector query, the decisions that decide quality (chunking, permissions, freshness, evaluation) and when RAG is the wrong tool.
A large language model knows what it learned in training and nothing about your price list, your policies or last week’s support tickets. Ask it about them and it will either say it does not know or, worse, write a confident answer that sounds right and is not. Retrieval-augmented generation, usually shortened to RAG, is the standard fix: before the model answers, your system finds the relevant passages in your own documents and hands them to the model with the question.
This guide explains how RAG works in plain terms, shows a minimal retrieval query, and then covers the decisions that decide whether a RAG system is trusted or ignored: chunking, permissions, freshness and evaluation. It ends with the cases where RAG is the wrong tool. For the architecture around it (gateway, tools, evaluation and monitoring), see our guide to building an LLM application.
Key takeaways
- RAG lets a language model answer from your own, current documents by retrieving relevant passages and adding them to the prompt.
- Most of the quality comes from retrieval, not the model: how documents are split, searched and filtered decides what the model gets to read.
- Permissions must be enforced at retrieval time, before text reaches the model, or a user can read documents they should never see.
- Measure retrieval and answers separately on a fixed test set, so you know which half to fix.
- RAG is the wrong tool for questions about live numbers, calculations or style; use tools, databases or fine-tuning there.
What retrieval-augmented generation means
The term comes from a 2020 research paper by Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, which combined a text generator with a searchable index of documents. The idea has since become the default way to connect language models to private knowledge. Google describes the same technique, which it also calls grounding, as one of the ways its AI features in Search pick the web pages they show and cite, in its guide to generative AI features in Search.
In business terms, RAG gives you three things a bare model cannot: answers based on your current documents, a citation for each answer so people can check it, and control over who can see what. Updating the knowledge means updating the documents, not retraining a model.
How a RAG system works, step by step
A RAG system has two halves. The first runs whenever documents change: it prepares your content for search. The second runs on every question: it finds the right passages and writes the answer.
Step 1: Collect the sources
Pull documents from where they live: help centre, policy PDFs, product sheets, tickets, CRM notes. Record the owner, date and access rules of each one.
Step 2: Split into chunks
Break each document into passages small enough to be specific, such as a section under one heading, and keep the title and heading path with each chunk.
Step 3: Create embeddings
An embedding model turns each chunk into a list of numbers (a vector) that captures its meaning, so passages about the same thing sit close together.
Step 4: Store with metadata
Save the chunk text, its vector and its metadata (source, date, language, who may read it) in a vector index such as PostgreSQL with pgvector.
Step 5: Retrieve for a question
Turn the user's question into a vector, find the closest chunks the user is allowed to see, and optionally combine this with keyword search and a re-ranking step.
Step 6: Build the prompt
Put the best few chunks into the prompt with clear instructions: answer only from these passages, cite them, and say so when they do not contain the answer.
Step 7: Generate and cite
The language model writes the answer with references back to the source passages, which the interface shows as links.
Step 8: Log and evaluate
Store the question, the retrieved chunks and the answer, and score samples against a test set so you can see where quality drops.
A minimal retrieval query with pgvector
You do not need a separate database to start. pgvector is an open-source extension that adds vector search to PostgreSQL, so chunks, metadata and permissions live next to the rest of your data. It supports cosine distance (the <=> operator), L2 distance and inner product, and two index types, HNSW and IVFFlat. An HNSW index on the standard vector type covers up to 2,000 dimensions (halfvec goes to 4,000).
The sketch below creates a chunks table, indexes it for cosine distance and retrieves the five closest chunks that the current user’s groups may read. The vector size must match your embedding model; 1024 is only an example.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
document_id bigint NOT NULL,
heading text NOT NULL,
body text NOT NULL,
groups text[] NOT NULL, -- who may read this chunk
updated_at timestamptz NOT NULL DEFAULT now(),
embedding vector(1024) NOT NULL -- match your embedding model
);
-- Approximate nearest-neighbour index for cosine distance
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (groups);
-- $1 = question embedding, $2 = the user's groups, e.g. '{sales,uk}'
SELECT id, document_id, heading, body,
1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE groups && $2
ORDER BY embedding <=> $1
LIMIT 5;Two notes from the pgvector documentation matter in production. First, approximate indexes apply filters after the index scan, so a strict permission filter can return fewer than five rows; since version 0.8.0 you can enable iterative index scans (SET hnsw.iterative_scan = strict_order) so the index keeps scanning until enough rows match. Second, vector search can be combined with PostgreSQL full-text search for hybrid search, which helps with product codes, names and exact phrases that embeddings handle poorly.
When the collection grows to many millions of chunks, or needs features Postgres does not offer, a dedicated vector database may pay off. We compare those options in a separate article on vector databases.
Want a second opinion on your plans?
Tell us what you are working on. A specialist will look at it and reply with practical next steps.
Get in touchThe decisions that decide quality
Chunking: what the model gets to read
Chunks that are too large bury the answer in unrelated text and waste tokens. Chunks that are too small lose the context that makes a sentence meaningful (“this applies to annual plans” means little without the plan’s name). We start by splitting on the document’s own structure, one section under one heading, and add the document title and heading path to each chunk so it reads correctly on its own. Tables, price lists and FAQs often deserve special handling, such as one chunk per row or per question.
Permissions: enforced before the model sees anything
A model cannot keep a secret it has been shown. If an HR policy draft is retrieved into the prompt, the model may quote it to anyone. The only safe place to enforce access is the retrieval query itself, as in the groups filter above, using the same rules as the source system. The OWASP Top 10 for LLM applications lists “Vector and Embedding Weaknesses” (LLM08:2025) for this reason, alongside “Prompt Injection” (LLM01:2025): a document can contain instructions aimed at the model, so retrieved text must be treated as data, never as commands.
Freshness: keeping the index in step with the source
The most common complaint about a RAG assistant is an answer from an outdated document. Build the sync into the design: re-index a document when it changes, remove chunks when it is deleted or archived, store updated_at on every chunk, and prefer the newest version when two passages conflict. For fast-changing facts such as stock, prices or order status, do not index them at all; call the live system instead.
Retrieval quality: search, then re-rank
Pure vector search finds passages with similar meaning but can miss exact terms. Hybrid search (vectors plus keywords) and a re-ranking step that scores the top 20 or 50 candidates more carefully are the first changes we test when answers miss, before trying a bigger language model. Measure before and after each change, on the same test set.
Answer rules: cite or decline
Instruct the model to answer only from the retrieved passages, cite them, and say clearly when they do not contain the answer. In customer-facing assistants we also route “I could not find this” answers to a person or a ticket, so missing knowledge becomes a list of documents to write.
How to measure a RAG system
Evaluate retrieval and generation separately, on a fixed test set of real questions with known good answers and known source passages. The open-source Ragas library names the core metrics most teams use.
| Metric | What it checks | What a low score tells you |
|---|---|---|
| Context precision | Whether the retrieved chunks that matter are ranked near the top | Retrieval returns noise; improve search or re-ranking |
| Context recall | Whether the retrieved chunks contain everything needed for the answer | Content is missing or badly chunked |
| Faithfulness | Whether every claim in the answer is supported by the retrieved context | The model is adding facts; tighten instructions or the model choice |
| Response relevancy | Whether the answer addresses the question that was asked | Prompt or model issue, or the question needs clarifying |
Source: Ragas documentation, available metrics, fetched 13 Nov 2026
Automated scores are a guide, not a verdict. We pair them with a weekly sample reviewed by someone who knows the subject, and we add every wrong answer found in production to the test set.
When RAG is the wrong tool
RAG is the right default for “answer from our documents”. It is a poor fit in other situations:
- Live numbers and records: “What is the status of order 4471?” needs a tool call to the order system, not a search through documents.
- Questions over structured data: totals, averages and trends belong in SQL or a BI tool. A model can write or call the query; it should not estimate from retrieved text.
- A small, stable knowledge base: current models read up to a million tokens in one request (as of November 2026 for the flagship tiers of OpenAI, Anthropic and Google). If all your reference material fits comfortably and rarely changes, passing it in full with prompt caching can be simpler.
- Tone, format or a narrow skill: if the problem is how the model writes, not what it knows, better instructions or fine-tuning are the levers. Our comparison of RAG, fine-tuning and prompt engineering covers that choice.
RAG is a good fit when
- Answers must come from your own documents, with citations
- Knowledge changes weekly or daily and must stay current
- Different users may see different documents
- You want to update knowledge without retraining anything
Look elsewhere when
- The answer is a live value held in another system
- The question needs calculation over many records
- The whole knowledge base fits in one prompt and rarely changes
- The problem is the model's style or format, not its knowledge
Checklist for a production RAG system
Before a RAG assistant goes live
- Each source has an owner, an update process and access rules
- Chunks follow document structure and carry title, heading path, date and permissions
- Permission filters run inside the retrieval query, using the source system's rules
- Deleted and archived documents are removed from the index automatically
- Hybrid search or re-ranking is tested against plain vector search on your test set
- The prompt requires citations and a clear "not found" answer
- Retrieved text is treated as data, and prompt injection tests are part of the test set
- Retrieval and answer quality are scored before every release
- Questions, retrieved chunks, answers and costs are logged with personal data handled under your privacy policy
- A person reviews a weekly sample and unanswered questions become content tasks
How we build RAG at AcmaSoft
Retrieval over a knowledge base runs inside our own products. In Acmafone CRM, our contact centre software, a copilot panel suggests replies to agents and finds answers in the knowledge base, so agents check one panel instead of searching manuals. In AI Call Studio, each voice agent has its own knowledge base alongside its call script.
On client projects we start with a discovery week: list the sources, agree the access rules, and write a test set of real questions with the client’s team. We usually build the first version on PostgreSQL with pgvector in Python, measure it, and only add a dedicated vector store or re-ranker when the numbers show the need. Where the source content is messy, our data curation and labelling team cleans and structures it first, because no retrieval method fixes contradictory documents. The wider build, from model choice to monitoring, sits in our generative AI and LLM integration services, and the model comparison in our guide to choosing an LLM covers the vendors. If the assistant is a customer-facing chatbot, our breakdown of chatbot development cost shows what drives the budget.
Planning an assistant that answers from your documents?
- Our AI chatbot development team builds RAG assistants with permission-aware retrieval, citations and a test set agreed with your team before launch.
Get new guides by email
Articles and guides from our team, and company news. Unsubscribe at any time.
About AcmaSoft
AcmaSoft designs and builds websites, mobile apps and custom software, and runs SEO and digital marketing for its clients.
- Years of experience
- 17+
- Clients served
- 650+
- Projects delivered
- 2,000+
- Products shipped
- 300+


