Retrieval was right. The answer was wrong.

I spent part of today stress-testing a small document Q&A prototype with synthetic policy documents. I deliberately added two versions of the same refund policy: an older one with a 30-day window and a newer one with 14 days.

The retriever did its job. It found the relevant paragraph. The assistant still answered 30 days.

The problem wasn’t the model. It was the product’s definition of “relevant.” Semantic similarity had no idea which document was active.

What fixed it:

• version metadata during ingestion • active/inactive filtering before vector search • a published-date tie-breaker • citations that expose the source version • an abstain path when active sources conflict

This looked like an AI problem, but it was really a data-contract problem.

If you’re building RAG into a SaaS product, how are you handling superseded documents?

#RAG #AIEngineering #SaaS