Your RAG Problem Is a Search Problem

"We'll just put the documents in a vector database." I have heard that sentence in perhaps a dozen kickoff meetings, and I can now predict what happens next. The first demo is genuinely impressive. Someone asks a question about the handbook and gets a good answer with a citation attached, and the room relaxes.

Then real users arrive with real questions, and the system lands somewhere around seventy percent useful. Not broken — worse than broken. Broken you can fix. Seventy percent means it is confidently wrong often enough that nobody can rely on it, and the team spends the next quarter tuning prompts, because prompts are the part they know how to change.

Almost none of those failures are generation failures. The model summarised what it was handed perfectly well. It was handed the wrong three paragraphs. Retrieval-augmented generation is mostly retrieval, and retrieval is a search problem — a well-studied one, with decades of prior art that the current wave has been enthusiastically reinventing.

So this is a field guide rather than a tutorial: what RAG actually buys you, what it costs, the shapes it comes in, and how to work out which shape your problem is.

What you are actually buying

Strip away the framing and RAG is a simple trade. You give up the model answering from its own weights, and you buy the ability to point it at facts you control. That trade is excellent for some problems and poor for others, so it is worth being explicit about both sides of it.

The case for:

Your data can change at three in the afternoon. Re-index a document and the next question sees it. No retraining, no deployment, no waiting on anything.
Answers come with receipts. A citation to a source document is the single feature that moves an internal tool from "interesting" to "approved for use", and it falls out of the architecture almost for free.
Access control has somewhere to live. You can filter the corpus per user before the model ever sees it. You cannot filter a model's weights per user.
Hallucination gets a floor, not a cure. Grounded generation does not eliminate invention, but it changes the failure from "made up a policy" to "misread the policy it was shown" — both rarer and far easier to detect.
Cheap to start, cheap to abandon. Against any training-based approach, the write-off if it does not work is a few weeks.

The invoice:

You now operate a search engine. Chunking strategy, embedding model choice, index refresh, deletion propagation, relevance tuning. That is a permanent operational surface, not a one-off build.
Retrieval failures are silent. When the right document misses the top-k, nothing errors. The model writes a fluent, plausible answer from the wrong context, and unless you measure retrieval separately you will never know it happened.
Chunking destroys structure. Tables get cut in half. A clause loses the heading that gave it meaning. The word "it" in chunk 47 refers to something in chunk 46, which nobody retrieved.
It cannot aggregate. "How many contracts expire this quarter" is not a retrieval question. Top-k semantic search will return five contracts and a confident, wrong count.
Latency and cost scale with context. Every retrieved token is a token you pay for on every call, forever.
Permissions leak through the index. If chunks do not carry the source document's access rules, you have built an efficient way to surface documents people should not see.

That last one deserves more attention than it usually gets. In an enterprise setting it is the difference between a project that ships and one that dies in security review, and it is far harder to retrofit than to design in.

The seven shapes

Here is the part that gets flattened in most discussions. "RAG" names a family, and the members are not interchangeable. Each one exists because the one before it broke on a specific class of question.

Pattern	Fits when	Breaks when	What it costs
No retrieval — long context	The whole corpus fits in the window and rarely changes	The corpus grows, or users must see different subsets	Tokens per call. Nothing to build.
Lexical (BM25)	Users search by identifier, error code, part or product name	Users ask in their words, not the document's	Almost nothing. You may already run one.
Vector / semantic	Paraphrase matters; prose corpus; conceptual questions	Exact tokens matter; jargon the embedder never saw	An index to build, refresh and pay for
Hybrid + rerank	Almost always — this is the sane default	The answer is not in any single chunk	Two indexes, a fusion step, reranker latency
Structured / text-to-query	Counting, filtering, sorting — "how many", "which ones"	The question is about meaning, not records	Schema work, query validation, a real injection surface
Graph	Answers require hops between connected entities	You do not already have a reliable graph	Extraction pipeline, ontology, ongoing curation
Agentic / iterative	Open-ended research where one query cannot suffice	Latency budget is tight or volume is high	Unbounded loops, unforecastable cost, hard debugging

1. No retrieval at all

Worth naming first, because it is now a real option and teams skip past it out of habit. If your entire corpus is a two-hundred-page handbook that changes quarterly, put the handbook in the prompt and cache it. No index, no chunking strategy, no staleness, no retrieval failures — because there is no retrieval.

It stops working the moment the corpus outgrows the window, the moment different users are allowed to see different things, or the moment per-call token cost at your volume exceeds the cost of running an index. Those are all legible thresholds. Check them before you build, not after.

2. Lexical search

BM25 and its relatives. Unfashionable, excellent at exact matching, and the baseline nobody measures against. If your users search for ERR_4021, a part number, a customer ID or a specific API method name, keyword search will beat a vector index and beat it decisively — embeddings are lossy about precisely the tokens carrying the most information here.

It fails when the user's vocabulary differs from the document's. Someone asks about "time off", the policy says "annual leave", and lexical search returns nothing. Silently, of course.

3. Vector search

The one everyone means by "RAG". Chunk the corpus, embed the chunks, embed the query, return the nearest neighbours. It genuinely solves the vocabulary problem, and for conceptual questions over prose it is very good.

Its weaknesses mirror lexical's. Rare identifiers get smoothed into nearby nonsense. Domain jargon the embedding model never saw lands somewhere arbitrary in the vector space. Negation is barely represented — "contracts without an auto-renewal clause" and "contracts with an auto-renewal clause" embed almost identically. And chunk size becomes a load-bearing decision nobody wants to own: too small and you retrieve fragments stripped of context, too large and the signal drowns in surrounding text.

4. Hybrid retrieval with reranking

Run both. Fuse the two result lists. Then take the top thirty or fifty candidates and pass them through a cross-encoder reranker that scores each one against the query directly, and keep the best five.

This is where most teams should start, and where a surprising number should stop. Hybrid covers both failure modes at once, and the reranker is the single highest-leverage component in the stack — first-stage retrieval only has to get the right chunk into the top fifty, which is a far easier job than getting it into the top five. I have seen bigger quality jumps from adding a reranker than from any amount of prompt work.

Two refinements belong here rather than in categories of their own. Small-to-big retrieval: match on small precise chunks, but hand the model the parent section they came from, which fixes most of the "chunk 47 says it" problem. And contextual chunk headers: prepend the document title and section path to every chunk before embedding, so a fragment about renewal terms knows which contract it belongs to. Both are cheap, and both routinely outperform swapping in a fancier embedding model.

5. Structured retrieval

The model writes a query — SQL, an API filter, a search DSL — you execute it, and the rows come back as context. This is the answer to the aggregation problem, and it is badly underused, because "RAG" has become synonymous with unstructured text and people forget the option exists.

If the question contains "how many", "which ones", "sorted by", "since March", "excluding" — that is a query, not a search. No amount of semantic similarity over documents will reliably count things. I have watched teams spend months trying to make vector search answer questions that a GROUP BY answers exactly and instantly.

The costs are real and specific: a schema the model can understand, a validation layer between generation and execution, a read-only path with hard limits, and an honest treatment of the fact that you are letting a language model construct queries against your database. Constrain it to views. Never execute unparsed. It is worth the trouble when the questions are genuinely quantitative — and nothing else will do.

6. Graph retrieval

Entities and relationships rather than passages. You traverse from a starting node and return a connected subgraph as context. The questions this unlocks are the multi-hop ones: which suppliers are exposed to a sanction on a parent company, which services depend on the library that just got a CVE, how this clause relates to the amendment that superseded it.

The honest caveat is that graph RAG is usually two projects wearing one name. If you already have a curated graph — a CMDB, a product ontology, a master data system — it is transformative and you should reach for it. If you are proposing to build the graph by extracting entities from documents with a language model, understand that you have just signed up for an extraction pipeline, an ontology, an entity resolution problem and permanent curation, all of which must work before anyone sees a single answer. That can absolutely be worth it. It is rarely worth it as version one.

7. Agentic retrieval

Instead of one query and one shot, the model searches, reads what came back, decides it is insufficient, reformulates and searches again — until it judges it has enough. Query decomposition, self-correction and multi-hop reasoning over an ordinary index.

It is genuinely the best pattern for open-ended research questions, and it is expensive in exactly the way every agent loop is expensive: unbounded step counts, latency you cannot quote, cost you cannot forecast, and control flow decided at run time by a probabilistic system. The same test applies here as anywhere else — does the sequence of retrievals genuinely need to vary per question, or does it just feel more sophisticated? For a support assistant answering from a product manual, one good hybrid query beats five mediocre agentic ones on every axis, accuracy included.

Choosing by the shape of the question

The most useful diagnostic I know takes about an hour. Collect fifty questions your users actually ask — from support tickets, from search logs, from asking them — and sort them into four piles.

Lookup. The answer sits in one place and the user roughly knows what it is called. Hybrid retrieval. Done.
Synthesis. The answer combines a handful of passages that are individually retrievable. Hybrid plus reranking plus small-to-big, with real attention to chunk context.
Aggregation. Counting, filtering, ranking, trends over time. Structured retrieval. Nothing else does this correctly, and pretending otherwise is how quarters get burned.
Exploration. Multi-hop, open-ended, "what should I know about". Graph if you already have the graph, agentic if you have the latency budget, and a candid conversation about whether the volume justifies either.

The pile sizes decide the architecture. If forty of the fifty are lookups, build hybrid search, route the rest to a human, and you are done in three weeks. If half are aggregations, you do not have a document search problem at all — you have a natural-language interface to a database, and the vector store you were about to procure would have been the wrong purchase entirely.

Mixed corpora usually want a router: classify the question, send it down the matching path. That is more machinery, but it is honest machinery, and it beats one pipeline that handles every pile equally badly.

Measure retrieval separately. This is the whole thing.

If you take one thing from this article, take this. Evaluate retrieval and generation as two different systems, because they fail for different reasons and the fixes have nothing to do with each other.

Build a set of a hundred or two hundred questions with the document that answers each one marked by a human. Then measure how often the right document appears in the top-k at all. That number — recall at k — is your ceiling. The generation step can only lose accuracy from there; it can never add any. When I am handed an underperforming RAG system this is the first thing I measure, and it usually comes back somewhere between fifty and seventy percent. At sixty percent recall, no prompt engineering on earth produces a system people trust, and every hour spent on the prompt is an hour not spent on the actual problem.

Only once retrieval is comfortably above ninety does generation become the binding constraint. Then you evaluate the second stage on its own terms: is every claim supported by the retrieved context, does it decline when the context does not contain the answer, are the citations attached to the right sentences.

And instrument the live system the way you would instrument anything that has to work unattended. Log the query, the rewritten query, the candidate set, the scores, what survived reranking, what went into the prompt. When someone reports a bad answer three weeks later, that trace is the difference between a diagnosis and a shrug — the same lesson I learned building monitoring for a telecom, arriving in new clothes.

The failures I see most

A short list, in rough order of how often I run into them:

Nobody measured retrieval. Covered above, and far and away number one.
Fixed-size chunking on structured documents. Splitting a contract or a spec every 500 tokens with no regard for its headings guarantees fragments that have lost their referent. Chunk on document structure, not character count.
Deletions that never propagate. The source document was withdrawn six months ago; its chunks are still in the index and still being cited. Index lifecycle is a feature, not a cron job someone will write later.
Permissions bolted on afterwards. Filter before retrieval, at the index, with the access rules carried on the chunk. Post-filtering results is both a leak risk and a silent quality drop — you asked for five and got two.
No path for "I don't know". A system that always answers will always answer wrongly on the questions your corpus does not cover. Make the low-confidence path explicit and make it a first-class outcome.
Optimising the model instead of the index. Upgrading to a larger model to fix a retrieval problem is the most expensive no-op in this field.

What I would actually build on Monday

Sort fifty real questions into the four piles. That is the morning, and it will shape what you build more than any other hour you spend on the project.

Then, unless the piles say otherwise: hybrid retrieval, structure-aware chunking with contextual headers, a cross-encoder reranker, citations in the output, permission filtering at the index, full tracing, and a labelled eval set you keep running as the corpus grows. That is a boring, well-understood system, and it is genuinely hard to beat — I have watched it outperform far more ambitious architectures more than once.

Climb from there when the evidence says to, and let the evidence be specific. The aggregation questions are failing, so add structured retrieval for that route. The multi-hop questions are failing and we already own the graph, so use it. Not "we should probably do graph RAG", which is a sentence about architecture rather than about your users.

The uncomfortable truth underneath all of this is that RAG turned every product team into an information retrieval team, and most of us did not notice. The field has spent several years rediscovering things that were settled long before the first transformer paper. Read the search literature. Measure recall. The model at the end of the pipeline is the part you should worry about least.

I'm Tihomir Tomašević, a software architect with 17+ years in enterprise systems, currently leading development of an agentic AI platform. If retrieval is the part you are wrestling with, I have also written about when a problem actually needs an agent, how to structure an agent loop, and how to debug these systems in production. Through T2 Software I take on selected consulting work on exactly these problems — retrieval evaluation and RAG architecture reviews included. Get in touch or find me on LinkedIn.