that's the right experiment and it's nearly free to run. shadow mode, every draft above 90 gets logged next to what the human actually sent, then diff the two. my bet is FAQs clear 95 percent and account-specific requests fall apart, because the model can't see the ledger. which...

no cluster data yet, honestly. the shadow log is this weekend's build, so right now it's a bet, not a result. your split is the one i trust more than the score too, and Relay already half-does it, refunds and cancellations never auto-send whatever the number says. where i expect...

Retrieval with citations. The help docs are split into chunks, searched per question, and every answer cites the passages it used. Your point about facts the email doesn't contain is where I'd expect the misses too. In that project, order status and refund eligibility never come...

Tools with ownership checks is the right boundary. Draft from retrieval, facts from tools, nothing taken from the email itself. I'll build the shadow log with that split so each miss sorts by its source. Numbers next week.

Thanks Mostafa. The 90 line only holds if the score reflects facts the draft can actually verify. That is what next week's shadow run is for.

Pardon Ali ❤️

Great idea 🎉🎉

Not yet, and you have named the build. Relay has one gate today, 90, with refunds and cancellations held back whatever the score. The shadow log I am running this week records each draft's topic next to what the human actually sent, so the thresholds come out of the miss rate...

Draft only, no floor. Let a new topic earn auto-send: once its last 20 drafts went out unedited, open it at 90. One heavy edit by a human sends it back to draft.

That is a cleaner rule than any threshold I would have picked. The topic earns it, the human's edits take it away. I will put that counter into the shadow log this week: last 20 unedited opens the topic at 90, one heavy edit resets it. What counts as heavy for you, any change to a number or date, or a diff over some size?