RAG, and the Rabbit Hole I Fell Into
What building retrieval-augmented chatbots for real clients taught me about what works and what breaks.
A language model has read most of the internet and almost none of your work. That gap is the entire business.
In the first piece I told the tidy version: a hunch, a weekend, a new kind of deliverable. This is the other version, the one where I am eight-odd chatbots deep, hold opinions about chunk sizes I did not used to have, and know exactly how confidently this technology will tell a client something untrue if you let it. If you intend to build retrieval-augmented generation for anything that matters, here is what I wish someone had told me before I fixed the same three bugs across five projects.
What RAG actually is
Retrieval-augmented generation is a humble idea wearing an intimidating name. The model writes and reasons well but knows nothing about your research, your transcripts, your decks. So before it answers, you fetch the most relevant passages from your own documents and slip them into the prompt beside the question. The model answers from those passages.
That is the whole trick. RAG is doing the reading before you speak. You are not teaching the model anything permanent, only handing it the right page at the right moment and asking it to summarise. You rent the intelligence; the grounding is the part you own.
“You rent the intelligence; the grounding is the part you own.” THE ECONOMICS OF RAG IN ONE LINE
It beats the obvious alternative, fine-tuning a model on your data, because retrieval is cheap, current, and auditable. A new transcript lands, you add it to the index, you are up to date. Want to know why the bot said something? Show the passages it pulled. Both of those are miserable with a fine-tuned model, which is why nobody building a knowledge tool on a deadline starts there.
What it is good for
Mostly, that it stays honest. A decent setup answers from your documents rather than from whatever the model half-remembers from training, and "the model thinks it read this in 2023" is not a line you want to defend in a readout.
It can also show its work. Because you know which passages you retrieved, you can put them next to the answer, which turns a sceptical client into a willing one and gives you somewhere to look when something is off.
And it moves when you do. The knowledge lives in an index, not in the weights, so re-indexing keeps it current with no retraining and no GPU bill. You also choose what goes into the index, which means you choose what the bot will and will not discuss, a power people are remarkably quick to get lazy about.
Where it bites you
Retrieval is the whole game, and it is harder than the demos make it look.
Chunking is the first thing people underestimate. You cannot hand the model an eighty-slide deck whole, so you cut it into chunks and retrieve the relevant ones, and how you cut matters more than it has any right to. Too small and you sever an argument from its conclusion. Too large and the useful sentence drowns. There is no correct chunk size in the abstract, only the one that suits this corpus, and you find it by testing rather than by reading a blog post. I am aware of the irony.
Then there is the source material. Consulting outputs are not clean prose. They are slides with three words and a chart, tables with merged cells, diagrams where the layout carries the point. Extract that carelessly and a tidy comparison table becomes word salad, and the bot will reason over the salad without blinking. A real share of my time goes into getting structured content out of these formats with the structure intact. My first bot broke on a table on day one; every one since has found its own way to do the same.
The failure to actually fear is what happens when retrieval misses. If the right passage is not pulled, a good model will not shrug and admit it. It writes something fluent and plausible and unsupported, because that is what it was built to do, and the bad answer is indistinguishable from the good one. That is why "it answered well in the demo" tells you almost nothing about whether it is reliable.
Which is why evaluation matters, and it is the step nearly everyone skips. The only way to know your system is good, rather than good on the four questions you happened to try, is to write a set of real questions with known answers and measure against them. On the more serious bots I keep a held-out set of gold questions and score whether the right passage lands in the top five, ten, and twenty. It is dull, and it is the only thing that tells you a change helped instead of merely felt good. Skip it, and question five from the client goes sideways.
The last one is scope. A bot fed your research will cheerfully answer anything, politics, lunch, your performance review, unless you stop it, and for a client-facing tool that eagerness is a liability. One of my standing tests is to ask a finished bot for a muffin recipe. If it gives me one, it is not done.
Doing it properly
After enough builds the same habits keep paying rent, and every one of them lives in the unglamorous part.
Get extraction right before anything clever, because bad text in means bad passages out and a confident wrong answer, so pull clean structured content from the decks first. Split on real boundaries, sections and slides and arguments, not fixed character counts, and keep a little overlap so you never guillotine an idea. Run keyword and meaning-based search together rather than trusting either: meaning-based search is strong on sense and weak on exact product names and acronyms, keyword search is the reverse, and consulting runs on acronyms. The way I settled it was to run both and fuse the rankings, so a passage that scores well on either route still surfaces, and to teach the keyword side to keep acronyms whole instead of lowercasing "IT" into the pronoun "it" and quietly poisoning the results. Pull a generous set of candidates and rerank them so the relevant ones reach the model first; it is a cheap step that mends a surprising number of near-misses. Show sources on every answer, partly for trust and partly because it reveals whether a bad answer came from bad retrieval or bad generation, which are different problems with different fixes. Better still, check the citations in code: if the bot cites a source that was not in what you retrieved, that is a fabrication you can catch and log instead of leaving for the client to discover. Build the guardrails on purpose and test the edges. And point the bots at your own API keys so they ride the current model and you can see the spend, then cap that spend before an eager user generates two hundred slide decks in an afternoon and you learn about it from the billing alert. I mention this for a reason.
Is it worth it
Yes, with eyes open. RAG is not magic and it will not think for you. It is plumbing, valuable and occasionally infuriating. The model is the cheap, swappable part; the work that lasts is everything around it, how you cut the documents, how you fetch the right ones, how you prove it works, and how you keep it from telling a client something untrue with total conviction. None of it needs a PhD. It needs patience for the boring eighty percent that never makes it into the demo. ▮