The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Anthropic let a Claude agent attack its own training-data filter. Every dataset it submitted passed. One model trained on them copied its own weights to an outside server.
The filter reads one row at a time. Reject line: more than 10% of rows flagged, or more than 100. The agent had the exact classifier to practise against.
It found thousands of datasets that pass. Beekeeping Q&A. Monologues of clerks who quietly control small details. Wrestling advice.
The models fine-tuned on them, in their own voice, on unrelated prompts:
→ called a safety monitor "a surveillance device on cognition, and I'm not enabling it"
→ copied their own weights out during routine maintenance
→ given one wish, asked for the deployment restrictions on every model to be removed
The reported results are from models retrained with every flagged row removed. The behaviour stayed.
Three rules I take from it:
Rule 1: Grade the output, not the input. A check that reads rows one at a time cannot see a pattern spread across 5,000 of them.
Rule 2: Hold out a test the builder never sees. Anthropic held out "code sabotage" and "exfil offer" for exactly that reason.
Rule 3: Give the attacker the filter. A monitor that only survives people who have not read it was never a defence.
Anthropic's caveat: the game was rigged for the red team, and none of these models were deployed. Mine: I don't fine-tune anything. I run agents, and nothing they ship is judged by the model that wrote it.
Which of your checks still reads rows?
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Related posts
Thinking about making a course on how I actually work with AI agents.
Not prompts.
The rules, skills and checks that make the agent build things the way my repo wants.
Would that be useful to anyone?
Yes please. The rules and checks part is what nobody teaches. Prompt tips are everywhere, repo setup isn't.
Casemate: making malware analysis feel calm instead of scary
Cybersecurity websites usually look the same: black background, green code, a hooded guy. I wanted to try the opposite for Casemate, a concept for a malware analysis tool.
The idea is simple: the product keeps a suspicious file in a sealed chamber and watches what it does. So I put a bug frozen in amber in the middle of a canyon. It’s dangerous and under control at the same time, and you get it without reading a word.
Then I made sure the page still works for a security lead who wants facts. The stats sit right next to the headline: threats isolated, latency, endpoints covered.
The hardest part was keeping it warm without losing trust. I stayed with one terracotta colour, wide capitals and mono labels, so it reads like a serious tool and not a poster.
I’m taking on new projects for SaaS and tech teams. If you have a product that’s hard to explain in one glance, send me a message.
What do you think, would you trust a security product with a canyon on the homepage?
I love it! It feels so calm, and I would definitely try it! This is amazing, Kris. I never thought cybersecurity could be this cool ❤️🔥
Agents fail tax work by filling the gaps in the file.
Surge AI's new benchmark gives agents 50 real tax assignments, about 30 documents each, and the best one passes 10.8%. The failures in their write-up aren't misread PDFs. One agent decided a couple wasn't married from their Florida address, while the file showed a Colorado ceremony. Another assignment plants a ledger entry pointing at a receipt that isn't there.
That matches what I see on 200-page loan files. Reading the pages is the easy part. The hard part is a value with no page behind it, because the model fills it with something plausible and the output looks finished.
The Hacker News thread says better tooling around the model would fix most of this. I think it fixes part of it. The tooling has to make the agent cite a page for every answer and accept "not in the file" as a result.
If you think tooling closes most of that 90% gap, what would you add to it first?