“Chat with your documents” is one of the most requested AI features, and one of the most disappointing in practice. The answers are vague, out of date or confidently wrong. The usual reaction is to try a bigger model. It rarely helps, because the model is answering from whatever it was given, and what it was given was the wrong text.
Garbage in, fluent garbage out
Retrieval-augmented generation has two halves: find the right pieces of information, then let a model write an answer from them. The second half gets all the attention. The first half decides the quality.
A language model can’t cite a paragraph it never received. If the search step returns the 2021 version of a policy, you get a beautifully written answer about the 2021 policy.
Fix the sources first
Before tuning anything, look at what you are indexing. Duplicate documents, outdated versions, scanned PDFs without text, and pages that are mostly navigation all end up as noise in your results. Deleting or excluding them is the cheapest quality improvement you will ever make.
Then add metadata: document type, owner, date, department. Being able to filter on “current HR policies only” beats any clever embedding trick.
Use more than vectors
Vector search is great at meaning and surprisingly bad at exact things like product codes, names and error numbers. Keyword search is the opposite. Combining both, often called hybrid search, and re-ranking the top results gives better results than either one alone. Most search platforms, including Azure AI Search, support this out of the box.
Chunking matters too. Split documents along their own structure, such as headings and sections, rather than every 500 characters, and keep the title with every chunk so it still makes sense on its own.
Measure retrieval on its own
Take twenty real questions and write down which document should answer each one. Then check, without any model involved, whether that document shows up in the top five results. That single number, recall, tells you more than an afternoon of reading chat transcripts.
Once retrieval is good, the generation part often turns out to be the easy bit. A modest model with the right three paragraphs beats a frontier model with the wrong ten.
