Catch Wrong AI Answers a Perfect Score Missed: Survey AI Testing

A research firm that adds an AI chat to its survey data lets clients and analysts ask questions without writing a query. The risk is an answer with the right number and the wrong people behind it. A number check passes it, and a client may be the one who finds the mistake.

In this project, I tested an AI agent on the Stack Overflow Developer Survey 2025: 49,191 responses, the questionnaire and the published summaries.

I kept the survey's raw answers in SQLite and wrote 40 test questions with answers computed in Python, so no model graded its own work.

For calculations, I added tools that return the count and the base together. Code checks the number and source quotes; TypeSafe's Jev model checks whether the explanation describes the right people. Unclear calls go to a person for review.

Python, SQLite, Claude Haiku and TypeSafe's Jev. I built with Claude Code and Codex.

A perfect score hid a wrong answer. The agent passed all 40 checks on its first run. One answer gave the correct US-minus-UK gap of 5.15 points, then said it had left out people who only plan to use AI. Its own query counted them: 414 in the US and 101 in the UK. In research, who was counted is part of the finding, so I now check the explanation as well as the number.

One run flattered the agent. I ran the same 40 questions 3 times, and the score fell to 118 of 120. A follow-up that passed on the first run failed on the next 2: the agent wrote India's correct 86.8% in its answer, then reported Germany's 76.1% as the result.

The "better" version only looked better. My changed version passed the explanation grader more often, 82 of 90 against 69. But the grader matched only 6 of the 8 cases I tested it on, the core score didn't rise (117 of 120), and the new version made a new mistake. I didn't release it.

The math was never the problem. In every failed answer, the arithmetic was right. The agent misread the question: it kept the earlier country in a follow-up, and reported plans when asked about future use. That's where the next fix goes.

Checking every answer is cheap. Code did the math, Claude Haiku answered, and Jev made 376 narrow judgment calls for $0.012. The same calls would have cost about $0.036 on Claude Haiku and $0.72 on Claude Sonnet, estimated from the same token counts. The whole experiment cost $0.61.

I designed the experiment, built the data checks and tools, reviewed the agent's failures, and documented what the evidence supports.

Bring 10 questions your AI answers with confidence but your analysts still check. Start with the Free AI Systems Fit Call.

Related: See What Each Client Costs: 16,900+ Leads Traced for Aura Agency.

Source: Stack Overflow Developer Survey 2025, used under the Open Database License.