Anthropic used Pokémon to benchmark its newest AI model

In a surprising move, Anthropic decided to put its latest AI model, Claude 3.7 Sonnet, to the test with the classic Game Boy game, Pokémon Red. According to a blog post released on Monday, the company kitted out the model with the essentials: memory, the ability to read screen pixels, and the power to press buttons and move around the game screen. This setup allowed Claude 3.7 Sonnet to dive into the world of Pokémon and keep playing.

What sets Claude 3.7 Sonnet apart is its knack for "extended thinking." Similar to other models like OpenAI's o3-mini and DeepSeek's R1, it can tackle tough problems by cranking up the computing power and taking its sweet time to think things through.

This feature proved to be a game-changer in Pokémon Red. While the older Claude 3.0 Sonnet couldn't even make it out of the starting area in Pallet Town, Claude 3.7 Sonnet managed to take down three gym leaders and snag their badges.

Last week, a researcher tried out an early preview of Claude 3.7 Sonnet.

The results were striking. Within hours, Claude defeated Brock. Days later, it trounced Misty. Progress that older models had little hope of achieving.

Turns out extended thinking is super effective. pic.twitter.com/RspsLgj2Uf

— Anthropic (@AnthropicAI) February 25, 2025

It won't be long before some clever developer figures out the nitty-gritty details.

While Pokémon Red might seem like a bit of a fun test, games have actually been used for AI benchmarking for ages. Just in the last few months, we've seen a bunch of new apps and platforms pop up to test how well AI models can play everything from Street Fighter to Pictionary.