AI Score Benchmark Pro — Writing Assessment QA Toolkit

Self-directed digital product · Writing Edition v1.0.0 · Synthetic Development Edition

AI Score Benchmark Pro turns AI writing-score review into a repeatable workflow: run blind inputs, compare outputs, inspect regressions and review feedback. It brings synthetic challenge material, editable spreadsheets and offline evaluation tools into one delivery package for EdTech teams, assessment specialists and researchers.

A higher agreement rate can hide a newly broken case. Fluent writing can miss the task entirely. Feedback can sound helpful while inventing an error. The toolkit makes these behaviours visible together, rather than treating an aggregate score as the whole evaluation.

Eight task families connect to 60 core synthetic responses, 20 targeted edge cases and 12 extra robustness variants: 92 records in total. Stable IDs join the dataset, reference notes and evaluation tools. Each record includes five separate 0–5 criterion ratings, a provisional qualitative level, rationale notes and feedback expectations. Selected records add exact-span annotations and explicitly justified alternative levels.

Task achievement is assessed independently from language evidence. A fluent off-topic response may therefore receive strong language scores and zero task fulfilment. A total out of 25 never becomes an automatic CEFR conversion.

Run the blind JSONL inputs through your own model or scoring prompt. Reference ratings and rationales stay out of the input.

Import outputs into the XLSX engine or standard-library Python evaluator. Record the model, prompt, configuration and benchmark version for each run.

Compare matched cases across two runs. Inspect agreement, criterion errors, coverage and previous acceptance becoming failure; missing and invalid outputs remain visible.

Review feedback for accuracy, relevance, specificity, prioritisation, actionability, tone and invented corrections. Use paired variants and repeated runs to explore consistency.

Edge cases include fluent off-topic writing, valid dialects, false corrections, contradictory reasoning, insufficient evidence and embedded instructions.

The included constructed Fixture B improves exact agreement from 65/79 (82.3%) to 78/79 (98.7%), yet introduces a two-band regression at WR-006. Both runs have 80 complete main cases; one expects UNRATEABLE and is excluded from the numeric agreement denominator.

These are deliberately constructed demonstration fixtures, not measured outputs from a named AI vendor. They show why individual regression review belongs beside headline metrics.

The 51-file customer package includes two editable XLSX workbooks, CSV/JSON/JSONL exports, an offline Python evaluator, a local report viewer, reference and calibration guides, and the commercial licence. Dataset consistency, annotation spans, evaluator statistics, spreadsheet formula behaviour and PDF layouts were checked. Native Excel/Google Sheets compatibility and browser interaction with the local viewer remain documented verification limits.

Responses and reference ratings are AI-authored provisional design material. Independent human rating has not been completed. This edition supports internal development, exploratory QA and calibration practice. It is not an operationally validated human benchmark or an official CEFR assessment. Operational decisions require representative, independently rated local evidence.

View AI Score Benchmark Pro on Contra.

$199 USD one-time. The Team Evaluation Licence covers one organisation and up to five named internal users, with perpetual evaluation use of the delivered version. No hosted scoring service or AI subscription is included. Bring your own model outputs. See the product listing and included licence for full terms.

Published on 1 October 2026. Sales and conversion outcomes have not yet been measured.