Behind the scenes on this one, because the build was more interesting than the result. The task: ...Behind the scenes on this one, because the build was more interesting than the result. The task: ...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
InfiniteUp's avatar
InfiniteUp Agency
pro
• 11h
Behind the scenes on this one, because the build was more interesting than the result.
The task: get an AI agent to play a browser Tetris clone and beat the high score.
Problem one — the game renders to a <canvas>. No DOM, no list of filled cells, just pixels. So the first thing to build was a reader: divide the canvas into a 10×18 grid, sample each cell, decide filled or empty.
First attempt classified by colour, matching each cell against known block colours. Failed instantly and silently. Purple pieces read as empty. Gradient regions read as empty. The bot was playing against a board that was largely imaginary.
The fix was changing the question. Not "what colour is this cell" but "how much does this cell vary." Empty cells are flat. Filled cells have an edge, a bevel, a highlight. Measure luminance across the cell, take max minus min, threshold it. Works on every piece colour without enumerating any of them.
Then a flood fill to isolate the falling piece, and a placement search — for every column and rotation, simulate the drop, score the resulting board, pick the best.
Problem two — the browser protocol caps each execution at 45 seconds. Tetris doesn't finish in 45 seconds. So instead of an async loop that waits, you install an interval, return immediately, and poll the page every twenty seconds.
That's where it went wrong. The smart bot was too slow, so it got simplified down to a fixed key pattern. And the number being polled turned out to be the loop's own tick count, not the game score. Five runs of "improving" numbers that measured nothing at all.
The interesting part was the cleanup. To find out what actually happened, I went back to a 30-minute screen recording:
— ffmpeg to pull frames at 2fps — locate the playfield in pixel coordinates — classify every cell across a dense frame sequence — then render the classifier's verdicts back over the frames and check them by eye
That last step is the one that mattered. The first pass flagged the bottom row as complete in almost every frame, which looked like a bug in my grid alignment. It wasn't. The row genuinely was complete, and the game genuinely never cleared it.
Verified: a completed line sits in the well for ten full seconds while the stack piles on top of it. Across ninety seconds of play the board empties about thirty times, and every one of those is a game over, not a line clear.
The lesson I'm taking into the next build: before you simplify under pressure, write down what you're optimising for, then check whether the simplification is still attached to it.
Full write-up: https://infiniteup.dev/the-tetris-clone-didnt-clear-lines-our-ai-agent-didnt-notice/
Claudemcpartificialintelligence

infiniteup.dev

The Tetris Clone Didn’t Clear Lines. Our AI Agent Didn’t Notice. - InfiniteUp

We pointed an AI agent at a browser game and it reported five runs of improving scores that measured nothing at all. It also reached a correct conclusion on worthless evidence. What that failure means for anyone deploying agentic AI.

Benjamin's avatar
TRUEFRAME logo
Benjamin C
max
• 8h
tetris on a canvas is a tough one. did the agent beat the high score?
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started