What Actually Happens When a RAG Pipeline Fails
RAG pipelines don't crash. I'm leading with that because nobody told me and knowing it would have saved me weeks.
A normal software bug gives you a stack trace or a 500 error, something you can grep for at 2am. A RAG failure gives you a perfectly formatted, confidently worded answer that happens to be completely wrong, with no error or warning attached. Your system lies to your users with a straight face and they don't file bug reports. They quietly stop using the tool and go back to Ctrl+F.
I found this out the hard way. We had a documentation RAG for our platform team. A developer asked it how to connect to the production database. The model gave him a perfect answer with the connection string, port and auth method. He followed the instructions, connected, ran some read queries and spent two days confused about why production traffic looked so much lower than the dashboards showed.
The model had retrieved the setup guide for our staging environment. The two connection strings look almost identical: same format, same port, different hostname. Nobody had tagged the chunks with environment metadata, so the model had no way to tell them apart. For two days a developer drew conclusions from the wrong database while nothing flagged it. The answers to his queries were technically correct. They were just run against the wrong data entirely.
Every RAG failure I've seen since then falls into one of a few predictable patterns.
Where it breaks
| Failure | What it looks like | What's actually broken |
|---|---|---|
| Wrong chunks retrieved | Answer is confident but factually off. Cites passages that don't support the claim. | Embedding model mismatch, bad chunking, query-document asymmetry |
| Right chunks, wrong answer | Retrieved context is perfect. LLM still gets it wrong. | Lost-in-the-middle effect, conflicting versions in context, instruction interference |
| Chunk boundary split | Answer is partially correct but systematically incomplete | A table, list or argument got bisected by the chunker |
| Stale context | Answer was correct six months ago | Missing or ignored metadata, no version/date filtering |
| No evaluation baseline | You can't tell if anything is getting worse | No golden test set, no retrieval metrics, no drift detection |
The first row accounts for maybe 70% of the failures I've debugged. Nearly every time, I wasted days on the wrong stage first, tweaking prompts and adjusting temperature and swapping models, when the fix started with reading the retrieved chunks. In my experience the systems that work are the ones where someone bothered to look at the failures, more than the ones with the cleverest retrieval.
Read the chunks first
Before you touch any prompt, parameter or model, dump the top 5 retrieved chunks for 20 real queries and read them yourself. Cosine scores and recall metrics can wait until you've looked at the actual text.
You'll find the problem in fifteen minutes. Maybe the chunks are too big and the answer is buried in noise. Maybe they're too small and the meaning got shredded. Maybe the right document is there but it's from 2022, while the 2024 version sits two indexes away. Maybe a critical table got split in half and the model is confidently interpreting values without column headers. Those fifteen minutes keep you from optimizing a stage that isn't the bottleneck, which is the most expensive mistake in RAG debugging.
My protocol is dead simple and it starts at the output. From a bad answer I check the chunks, then the metadata, then the chunk boundaries and finally the embedding scores. That's five steps in about thirty minutes. It has caught every production issue I've encountered.