RouteMind — Independent RAG Benchmark Audit

Independent Retrieval Quality Audit | October 2026

The Challenge

RouteMind is an open-source project comparing traditional RAG retrieval, reranking, and document-routing approaches.

Its benchmark produced reproducible metrics, but different evaluation paths were applying different definitions of success.

Retrieval counted a query as successful when any expected document appeared in the top results. Some routing evaluations required all specified documents, including accepted alternatives.

This created an inconsistent comparison between approaches.

My Contribution

I performed an independent technical audit of the project's public evaluation artifacts.

The investigation covered 700 questions and 1,400 retrieval query/arm results, including scoring semantics, expected evidence, and reproducibility checks.

I identified 15 scoring discrepancies and delivered a public evidence package containing the affected cases, methodology, frozen input checksums, and limitations.

Verified Results

The RouteMind maintainer independently reproduced all 15 discrepancies and confirmed that the inconsistency was previously unknown and unintentional.

The findings led to a corrective commit that aligned the benchmark's main comparison and introduced additional verification checks.

The maintainer stated:

"Yes, this was real value beyond our own evaluations. We had looked at these files many times and not seen it."

Deliverables

Independent benchmark investigation

Query-level discrepancy evidence

Reproducible methodology and input checksums

Technical findings and recommendations

Public maintainer validation

This was a public beta audit, not a paid client engagement. The maintainer implemented the project corrections.

Public audit: https://github.com/devBorgesr/edp-audits/tree/main/audits/routemind-beta-002

Maintainer confirmation: https://github.com/CSP911/routemind/issues/2#issuecomment-6096856584

A compelling value proposition can make users want to know more about your product. Let's take a look at these two examples. "We help you manage your finances." "Know where your money goes and take control of your spending." Which one grabs your attention more? The second one works better because it focuses on what the customer actually cares about, not just what the product does. Here are some questions every value proposition should answer: Why should customers choose you over your competitors? What makes your product different? How does it solve your customers' problems? You can only answer these questions by understanding your customers' pain points and thinking from their perspective. If you don't understand your customers, no amount of design or advertising will help you build lasting customer relationships. Instead of constantly adding more features, startups should focus on what makes their product different. And communicate that value clearly through their landing pages.