Independent RAG Benchmark Audit Finds 15 Scoring DiscrepanciesIndependent RAG Benchmark Audit Finds 15 Scoring Discrepancies
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Isaque's avatar
Isaque Borges
• 3h
RouteMind — Independent RAG Benchmark Audit
Independent Retrieval Quality Audit | October 2026
The Challenge
RouteMind is an open-source project comparing traditional RAG retrieval, reranking, and document-routing approaches.
Its benchmark produced reproducible metrics, but different evaluation paths were applying different definitions of success.
Retrieval counted a query as successful when any expected document appeared in the top results. Some routing evaluations required all specified documents, including accepted alternatives.
This created an inconsistent comparison between approaches.
My Contribution
I performed an independent technical audit of the project's public evaluation artifacts.
The investigation covered 700 questions and 1,400 retrieval query/arm results, including scoring semantics, expected evidence, and reproducibility checks.
I identified 15 scoring discrepancies and delivered a public evidence package containing the affected cases, methodology, frozen input checksums, and limitations.
Verified Results
The RouteMind maintainer independently reproduced all 15 discrepancies and confirmed that the inconsistency was previously unknown and unintentional.
The findings led to a corrective commit that aligned the benchmark's main comparison and introduced additional verification checks.
The maintainer stated:
"Yes, this was real value beyond our own evaluations. We had looked at these files many times and not seen it."
Deliverables
Independent benchmark investigation
Query-level discrepancy evidence
Reproducible methodology and input checksums
Technical findings and recommendations
Public maintainer validation
This was a public beta audit, not a paid client engagement. The maintainer implemented the project corrections.
Public audit: https://github.com/devBorgesr/edp-audits/tree/main/audits/routemind-beta-002
Maintainer confirmation: https://github.com/CSP911/routemind/issues/2#issuecomment-6096856584
Completed work
Data AnalysisProduct Data AnalyticsGitGitHubPython
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started