Skip to content

This site is shown in English. Switch language if you prefer.

Menu

Monday to Saturday, 9:30 am to 6:30 pm India time (IST)

Contact us

RAG explained: how retrieval-augmented generation works and when a business needs it

Chief Technology and AI Officer

  • Published
  • 11 min read
Documents pass through a funnel into a searchable index. A question is matched against the index, the AI model writes an answer that cites its sources, and three cards show the decisions that matter: chunking, permissions and freshness.
Table of contentsContents
  1. What retrieval-augmented generation means
  2. How a RAG system works, step by step
  3. A minimal retrieval query with pgvector
  4. The decisions that decide quality
    1. Chunking: what the model gets to read
    2. Permissions: enforced before the model sees anything
    3. Freshness: keeping the index in step with the source
    4. Retrieval quality: search, then re-rank
    5. Answer rules: cite or decline
  5. How to measure a RAG system
  6. When RAG is the wrong tool
  7. Checklist for a production RAG system
  8. How we build RAG at AcmaSoft
  9. Frequently asked questions

Share this article

A plain walk-through of retrieval-augmented generation, a minimal pgvector query, the decisions that decide quality (chunking, permissions, freshness, evaluation) and when RAG is the wrong tool.

A large language model knows what it learned in training and nothing about your price list, your policies or last week’s support tickets. Ask it about them and it will either say it does not know or, worse, write a confident answer that sounds right and is not. Retrieval-augmented generation, usually shortened to RAG, is the standard fix: before the model answers, your system finds the relevant passages in your own documents and hands them to the model with the question.

This guide explains how RAG works in plain terms, shows a minimal retrieval query, and then covers the decisions that decide whether a RAG system is trusted or ignored: chunking, permissions, freshness and evaluation. It ends with the cases where RAG is the wrong tool. For the architecture around it (gateway, tools, evaluation and monitoring), see our guide to building an LLM application.

Key takeaways

  • RAG lets a language model answer from your own, current documents by retrieving relevant passages and adding them to the prompt.
  • Most of the quality comes from retrieval, not the model: how documents are split, searched and filtered decides what the model gets to read.
  • Permissions must be enforced at retrieval time, before text reaches the model, or a user can read documents they should never see.
  • Measure retrieval and answers separately on a fixed test set, so you know which half to fix.
  • RAG is the wrong tool for questions about live numbers, calculations or style; use tools, databases or fine-tuning there.

What retrieval-augmented generation means

The term comes from a 2020 research paper by Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, which combined a text generator with a searchable index of documents. The idea has since become the default way to connect language models to private knowledge. Google describes the same technique, which it also calls grounding, as one of the ways its AI features in Search pick the web pages they show and cite, in its guide to generative AI features in Search.

In business terms, RAG gives you three things a bare model cannot: answers based on your current documents, a citation for each answer so people can check it, and control over who can see what. Updating the knowledge means updating the documents, not retraining a model.

How a RAG system works, step by step

A RAG system has two halves. The first runs whenever documents change: it prepares your content for search. The second runs on every question: it finds the right passages and writes the answer.

  1. Step 1: Collect the sources

    Pull documents from where they live: help centre, policy PDFs, product sheets, tickets, CRM notes. Record the owner, date and access rules of each one.

  2. Step 2: Split into chunks

    Break each document into passages small enough to be specific, such as a section under one heading, and keep the title and heading path with each chunk.

  3. Step 3: Create embeddings

    An embedding model turns each chunk into a list of numbers (a vector) that captures its meaning, so passages about the same thing sit close together.

  4. Step 4: Store with metadata

    Save the chunk text, its vector and its metadata (source, date, language, who may read it) in a vector index such as PostgreSQL with pgvector.

  5. Step 5: Retrieve for a question

    Turn the user's question into a vector, find the closest chunks the user is allowed to see, and optionally combine this with keyword search and a re-ranking step.

  6. Step 6: Build the prompt

    Put the best few chunks into the prompt with clear instructions: answer only from these passages, cite them, and say so when they do not contain the answer.

  7. Step 7: Generate and cite

    The language model writes the answer with references back to the source passages, which the interface shows as links.

  8. Step 8: Log and evaluate

    Store the question, the retrieved chunks and the answer, and score samples against a test set so you can see where quality drops.

A minimal retrieval query with pgvector

You do not need a separate database to start. pgvector is an open-source extension that adds vector search to PostgreSQL, so chunks, metadata and permissions live next to the rest of your data. It supports cosine distance (the <=> operator), L2 distance and inner product, and two index types, HNSW and IVFFlat. An HNSW index on the standard vector type covers up to 2,000 dimensions (halfvec goes to 4,000).

The sketch below creates a chunks table, indexes it for cosine distance and retrieves the five closest chunks that the current user’s groups may read. The vector size must match your embedding model; 1024 is only an example.

Chunks table, index and a permission-filtered query (PostgreSQL with pgvector)
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id          bigserial PRIMARY KEY,
  document_id bigint      NOT NULL,
  heading     text        NOT NULL,
  body        text        NOT NULL,
  groups      text[]      NOT NULL,          -- who may read this chunk
  updated_at  timestamptz NOT NULL DEFAULT now(),
  embedding   vector(1024) NOT NULL          -- match your embedding model
);

-- Approximate nearest-neighbour index for cosine distance
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (groups);

-- $1 = question embedding, $2 = the user's groups, e.g. '{sales,uk}'
SELECT id, document_id, heading, body,
       1 - (embedding <=> $1) AS similarity
FROM   chunks
WHERE  groups && $2
ORDER  BY embedding <=> $1
LIMIT  5;

Two notes from the pgvector documentation matter in production. First, approximate indexes apply filters after the index scan, so a strict permission filter can return fewer than five rows; since version 0.8.0 you can enable iterative index scans (SET hnsw.iterative_scan = strict_order) so the index keeps scanning until enough rows match. Second, vector search can be combined with PostgreSQL full-text search for hybrid search, which helps with product codes, names and exact phrases that embeddings handle poorly.

When the collection grows to many millions of chunks, or needs features Postgres does not offer, a dedicated vector database may pay off. We compare those options in a separate article on vector databases.

Want a second opinion on your plans?

Tell us what you are working on. A specialist will look at it and reply with practical next steps.

Get in touch

The decisions that decide quality

Chunking: what the model gets to read

Chunks that are too large bury the answer in unrelated text and waste tokens. Chunks that are too small lose the context that makes a sentence meaningful (“this applies to annual plans” means little without the plan’s name). We start by splitting on the document’s own structure, one section under one heading, and add the document title and heading path to each chunk so it reads correctly on its own. Tables, price lists and FAQs often deserve special handling, such as one chunk per row or per question.

Permissions: enforced before the model sees anything

A model cannot keep a secret it has been shown. If an HR policy draft is retrieved into the prompt, the model may quote it to anyone. The only safe place to enforce access is the retrieval query itself, as in the groups filter above, using the same rules as the source system. The OWASP Top 10 for LLM applications lists “Vector and Embedding Weaknesses” (LLM08:2025) for this reason, alongside “Prompt Injection” (LLM01:2025): a document can contain instructions aimed at the model, so retrieved text must be treated as data, never as commands.

Freshness: keeping the index in step with the source

The most common complaint about a RAG assistant is an answer from an outdated document. Build the sync into the design: re-index a document when it changes, remove chunks when it is deleted or archived, store updated_at on every chunk, and prefer the newest version when two passages conflict. For fast-changing facts such as stock, prices or order status, do not index them at all; call the live system instead.

Retrieval quality: search, then re-rank

Pure vector search finds passages with similar meaning but can miss exact terms. Hybrid search (vectors plus keywords) and a re-ranking step that scores the top 20 or 50 candidates more carefully are the first changes we test when answers miss, before trying a bigger language model. Measure before and after each change, on the same test set.

Answer rules: cite or decline

Instruct the model to answer only from the retrieved passages, cite them, and say clearly when they do not contain the answer. In customer-facing assistants we also route “I could not find this” answers to a person or a ticket, so missing knowledge becomes a list of documents to write.

How to measure a RAG system

Evaluate retrieval and generation separately, on a fixed test set of real questions with known good answers and known source passages. The open-source Ragas library names the core metrics most teams use.

Core RAG evaluation metrics
MetricWhat it checksWhat a low score tells you
Context precisionWhether the retrieved chunks that matter are ranked near the topRetrieval returns noise; improve search or re-ranking
Context recallWhether the retrieved chunks contain everything needed for the answerContent is missing or badly chunked
FaithfulnessWhether every claim in the answer is supported by the retrieved contextThe model is adding facts; tighten instructions or the model choice
Response relevancyWhether the answer addresses the question that was askedPrompt or model issue, or the question needs clarifying

Source: Ragas documentation, available metrics, fetched 13 Nov 2026

Automated scores are a guide, not a verdict. We pair them with a weekly sample reviewed by someone who knows the subject, and we add every wrong answer found in production to the test set.

When RAG is the wrong tool

RAG is the right default for “answer from our documents”. It is a poor fit in other situations:

  • Live numbers and records: “What is the status of order 4471?” needs a tool call to the order system, not a search through documents.
  • Questions over structured data: totals, averages and trends belong in SQL or a BI tool. A model can write or call the query; it should not estimate from retrieved text.
  • A small, stable knowledge base: current models read up to a million tokens in one request (as of November 2026 for the flagship tiers of OpenAI, Anthropic and Google). If all your reference material fits comfortably and rarely changes, passing it in full with prompt caching can be simpler.
  • Tone, format or a narrow skill: if the problem is how the model writes, not what it knows, better instructions or fine-tuning are the levers. Our comparison of RAG, fine-tuning and prompt engineering covers that choice.

RAG is a good fit when

  • Answers must come from your own documents, with citations
  • Knowledge changes weekly or daily and must stay current
  • Different users may see different documents
  • You want to update knowledge without retraining anything

Look elsewhere when

  • The answer is a live value held in another system
  • The question needs calculation over many records
  • The whole knowledge base fits in one prompt and rarely changes
  • The problem is the model's style or format, not its knowledge

Checklist for a production RAG system

Before a RAG assistant goes live

  • Each source has an owner, an update process and access rules
  • Chunks follow document structure and carry title, heading path, date and permissions
  • Permission filters run inside the retrieval query, using the source system's rules
  • Deleted and archived documents are removed from the index automatically
  • Hybrid search or re-ranking is tested against plain vector search on your test set
  • The prompt requires citations and a clear "not found" answer
  • Retrieved text is treated as data, and prompt injection tests are part of the test set
  • Retrieval and answer quality are scored before every release
  • Questions, retrieved chunks, answers and costs are logged with personal data handled under your privacy policy
  • A person reviews a weekly sample and unanswered questions become content tasks

How we build RAG at AcmaSoft

Retrieval over a knowledge base runs inside our own products. In Acmafone CRM, our contact centre software, a copilot panel suggests replies to agents and finds answers in the knowledge base, so agents check one panel instead of searching manuals. In AI Call Studio, each voice agent has its own knowledge base alongside its call script.

On client projects we start with a discovery week: list the sources, agree the access rules, and write a test set of real questions with the client’s team. We usually build the first version on PostgreSQL with pgvector in Python, measure it, and only add a dedicated vector store or re-ranker when the numbers show the need. Where the source content is messy, our data curation and labelling team cleans and structures it first, because no retrieval method fixes contradictory documents. The wider build, from model choice to monitoring, sits in our generative AI and LLM integration services, and the model comparison in our guide to choosing an LLM covers the vendors. If the assistant is a customer-facing chatbot, our breakdown of chatbot development cost shows what drives the budget.

Planning an assistant that answers from your documents?

  • Our AI chatbot development team builds RAG assistants with permission-aware retrieval, citations and a test set agreed with your team before launch.

Share this article

About the author:

Chief Technology and AI Officer

Amit leads technology, engineering and the AI practice at AcmaSoft. He sets the technical direction for client projects, from architecture and code quality to the AI and automation work we deliver.

  • Technology and engineering
  • Architecture and code quality
  • AI practice
  • Automation

Get new guides by email

Articles and guides from our team, and company news. Unsubscribe at any time.

About AcmaSoft

AcmaSoft designs and builds websites, mobile apps and custom software, and runs SEO and digital marketing for its clients.

Years of experience
17+
Clients served
650+
Projects delivered
2,000+
Products shipped
300+

Frequently asked questions

Short answers to the questions readers ask most about this topic.

Ask a question
What is RAG in simple terms?

RAG means the system looks up relevant passages in your own documents and gives them to the language model together with the question, so the answer is based on your current information and can cite its sources. The model itself is not changed.

Is RAG better than fine-tuning?

They solve different problems. RAG adds knowledge that changes often and needs citations. Fine-tuning changes how a model behaves, such as its format, tone or a narrow skill. Many systems use RAG first and fine-tune later only if a measured gap remains.

Do I need a vector database for RAG?

Not to start. PostgreSQL with the pgvector extension stores vectors next to your other data and supports cosine search with HNSW indexes. A dedicated vector database becomes worth it at large scale or when you need features Postgres does not provide.

How do you stop a RAG chatbot from inventing answers?

Retrieve well, instruct the model to answer only from the retrieved passages and to say when they do not contain the answer, show citations, and measure faithfulness on a test set before each release. A human review of samples catches what metrics miss.

Can a RAG system respect user permissions?

Yes, if permissions are applied in the retrieval query, so chunks a user may not read never reach the model. Store access rules with each chunk, mirror the source system’s rules, and test it with users of different roles.

Discuss your project with a specialist

Share what you are planning or trying to fix. We reply with practical advice, a rough timeline and the questions we need answered.

Estimated budget (optional)

Goals, users, key features and any deadline.0 / 2,000

Attachment (optional)

Protected against spam without puzzles. We never share your details.

Choose your region & language

Americas

Asia-Pacific

Europe, Middle East and Africa

Translations are produced automatically by Google Translate.