Cover image for RouteMind — Independent RAG Benchmark

RouteMind — Independent RAG Benchmark Audit Independent Retrieval Quality Audit | October 2026 The Challenge RouteMind is an open-source project comparing traditional RAG retrieval, reranking, and document-routing approaches. Its benchmark produced reproducible metrics, but different evaluation paths were applying different definitions of success. Retrieval counted a query as successful when any expected document appeared in the top results. Some routing evaluations required all specified documents, including accepted alternatives. This created an inconsistent comparison between approaches. My Contribution I performed an independent technical audit of the project's public evaluation artifacts. The investigation covered 700 questions and 1,400 retrieval query/arm results, including scoring semantics, expected evidence, and reproducibility checks. I identified 15 scoring discrepancies and delivered a public evidence package containing the affected cases, methodology, frozen input checksums, and limitations. Verified Results The RouteMind maintainer independently reproduced all 15 discrepancies and confirmed that the inconsistency was previously unknown and unintentional. The findings led to a corrective commit that aligned the benchmark's main comparison and introduced additional verification checks. The maintainer stated: "Yes, this was real value beyond our own evaluations. We had looked at these files many times and not seen it." Deliverables Independent benchmark investigation Query-level discrepancy evidence Reproducible methodology and input checksums Technical findings and recommendations Public maintainer validation This was a public beta audit, not a paid client engagement. The maintainer implemented the project corrections. Public audit: https://github.com/devBorgesr/edp-audits/tree/main/audits/routemind-beta-002 Maintainer confirmation: https://github.com/CSP911/routemind/issues/2#issuecomment-6096856584

Isaque Borges

RAG Evaluation & Retrieval Quality Specialist