Your RAG problem is a chunking problem
When an AI assistant answers from the wrong passage, teams usually blame the model and try a bigger one. The answer was decided earlier, when the documents were cut up and searched.
By Nathan Barling Product Marketing Principal 4 min read
When an internal assistant gets a marketing question wrong, the first instinct is to try a better model. Often the model wasn't the problem.
Say a team loads three years of campaign recaps into an internal assistant. Someone asks how the spring promotion did against the year before. The assistant answers confidently, with a growth number, and it's wrong.
The number came from a real document. It came from a paragraph that said "revenue grew 14% over the prior period," which had been split off from the heading that said which promotion, which market, and which year. The model did its job well. It was handed the wrong paragraph.
Most failures happen before the model writes
Researchers at Deakin University named seven points where RAG systems fail. The grouping below, retrieval or model, is our reading of where each one happens.
| Failure point | Where it happens |
|---|---|
| Missing content | RetrievalThe answer isn't in the documents at all |
| Missed the top ranked documents | RetrievalThe right passage exists but ranks too low |
| Not in context | RetrievalThe passage was found but cut when assembling the input |
| Not extracted | ModelThe answer was in the input but the model missed it |
| Wrong format | ModelThe answer ignored the format that was asked for |
| Incorrect specificity | ModelToo general or too detailed for the question |
| Incomplete | ModelPart of the answer was left out |
The first three are retrieval failures. A fourth, not extracted, is often a retrieval failure in disguise. Liu and colleagues found that language models do best when the relevant information sits at the beginning or end of a long input, and performance "significantly degrades" when it's in the middle. Stuffing more passages into the context to be safe can bury the one that matters.
Chunking strips the context out
Most retrieval systems cut documents into pieces before indexing them, and that's where the context goes. The paragraph in the opening scene meant nothing on its own once the heading naming the promotion, market, and year ended up in a different chunk. Search couldn't match it to a question about that promotion, because the words that would connect them weren't in it.
Teams that build these systems know the remedies: store a short note with each chunk saying what it covers, search by keywords as well as meaning, and rerank the results before the model sees them. None of them touch the model that writes the answer.
How you cut matters, too
Chroma's researchers compared common chunking strategies on the same documents and questions, and found some outperformed others by up to 9% in recall. Recall is the share of relevant material retrieved, and here it was decided entirely by where the documents were cut.
Recall with semantic chunking, the lowest in the comparison
Chroma, 2024, five results per query
Recall with recursive splitting into 200-token chunks
Chroma, 2024, five results per query
Recall with chunks split by a language model, the highest
Chroma, 2024, five results per query
Fixed-length chunks are the usual default because they're easy. They also cut tables from their headers, rules from their exceptions, and numbers from the period they describe. Splitting along the document's own structure, its headings, sections, and tables, keeps those pieces together.
Five fixes, in the order we'd try them
- Look at what was retrieved. Before changing anything, log the passages behind 20 or 30 wrong answers. Most teams have never read them, and the pattern is usually obvious once they do.
- Split along structure. Keep each table with its header and each section with its heading. Add the document title and date to every chunk.
- Add context to each chunk. A one-line note on what the chunk is about, written once when it's indexed, makes it findable by the questions people actually ask.
- Search two ways. Semantic search finds meaning, and keyword search finds exact terms like campaign IDs and product names. Use both and merge the results.
- Rerank and trim. Score the top results against the question with a reranker, and pass only the best few to the model, so the relevant passage isn't lost in the middle.
Only after those should a team try a bigger model. It's the most expensive fix, and it rarely helps when the wrong passage was retrieved.
Where Alli fits
For performance numbers, Alli doesn't rely on retrieving the right paragraph. Ask Alli resolves every request against the metrics, dimensions, and business rules defined in Alli Semantic Layer, so a question about last spring's ROAS is answered from governed data with the agreed definition. Each answer shows the data, logic, and source behind it.
Documents still matter for the context around those numbers. Alli Generative Workflows can pull file context from Google Drive and fill out Google templates, and the workflow steps that use it can be tested one task at a time in the Testing Suite.
When retrieval isn't the problem
Sometimes the passage was right and the model still got it wrong, or the answer simply isn't in the documents. Retrieval logs show which case you're in. If the right passage was in the input and the answer was still wrong, look at the instructions, the format, and the model. If the content doesn't exist, no amount of chunking will find it.
Pull the last 20 wrong answers your assistant gave. Read the passages it retrieved for each one. Count how many times the right passage wasn't there.
Frequently asked questions
Why does RAG give wrong answers?
Most often because the wrong passage was retrieved, the right one ranked too low, or it lost its context when the document was split. Researchers have named seven failure points, and most of them happen before the model writes anything.
What is chunking in RAG?
Splitting documents into smaller passages so they can be searched and passed to a model. How the split is made decides whether each passage keeps enough context to be found and understood.
What is hybrid search?
Combining semantic search, which matches meaning, with keyword search, which matches exact words like product names and IDs, then merging the results. It finds passages that either method alone would miss.
Should I use a bigger model to fix RAG answers?
Usually not first. If the right passage isn't retrieved, a bigger model can't use it. Check what was retrieved, then fix chunking, search, and ranking before changing the model.
What is the lost in the middle problem?
Language models tend to use information at the start and end of a long input better than information in the middle. Passing fewer, better-ranked passages helps the model use the one that matters.
Sources
- Deakin University, Seven Failure Points When Engineering a Retrieval Augmented Generation System, Scott Barnett and colleagues, January 2024.
- Chroma, Evaluating Chunking Strategies for Retrieval, Brandon Smith and Anton Troynikov, July 3, 2024. Chroma sells a vector database.
- Transactions of the Association for Computational Linguistics, Lost in the Middle: How Language Models Use Long Contexts, Nelson F. Liu and colleagues, 2023.
Last updated October 26, 2026. Figures are quoted from the sources above and were checked on that date.