Debates over AI benchmarking have reached Pokémon

Even the beloved world of Pokémon isn't immune to the drama surrounding AI benchmarks. A recent viral post on X stirred up quite the buzz, claiming that Google's latest Gemini model had outpaced Anthropic's leading Claude model in the classic Pokémon video game trilogy. According to the post, Gemini had impressively made it to Lavender Town in a developer's Twitch stream, while Claude was lagging behind at Mount Moon as of late February.

Gemini is literally ahead of Claude atm in pokemon after reaching Lavender Town

119 live views only btw, incredibly underrated stream pic.twitter.com/8AvSovAI4x

— Jush (@Jush21e8) April 10, 2025

However, what this post conveniently left out was the fact that Gemini had a bit of an unfair advantage. Savvy users over on Reddit quickly pointed out that the developer behind the Gemini stream had crafted a custom minimap. This nifty tool aids the model in recognizing "tiles" in the game, such as cuttable trees, which significantly cuts down the time Gemini needs to spend analyzing screenshots before deciding on its next move.

Now, while Pokémon might not be the most serious AI benchmark out there, it does serve as a fun yet telling example of how different setups can skew the results of these tests. Take Anthropic's recent model, Anthropic 3.7 Sonnet, for instance. On the SWE-bench Verified benchmark, which is meant to test coding prowess, it scored 62.3% accuracy. But, with a "custom scaffold" that Anthropic whipped up, that score jumped to 70.3%.

And it doesn't stop there. Meta took one of its newer models, Llama 4 Maverick, and fine-tuned it specifically for the LM Arena benchmark. The vanilla version of the model didn't fare nearly as well on the same test.

Given that AI benchmarks, including our friendly Pokémon example, are already a bit hit-or-miss, these custom tweaks and non-standard approaches just make it even trickier to draw meaningful comparisons between models as they hit the market. It seems like comparing apples to apples might be getting harder by the day.