High School Student Creates Website for AI Minecraft Build-Off Challenges

Creative AI Benchmarking with Minecraft

As traditional AI benchmarking methods fall short, developers are exploring innovative approaches to evaluate the prowess of generative AI models. One such creative method involves using Minecraft, the popular sandbox game owned by Microsoft. A group of developers has launched Minecraft Benchmark, or MC-Bench, a platform where AI models compete in creating Minecraft builds based on given prompts.

On MC-Bench, users can vote on which AI model's creation they prefer, and only after casting their vote do they discover which model made each build. This interactive approach not only engages the community but also provides a unique way to assess AI capabilities.

Adi Singh, a 12th-grader and the initiator of MC-Bench, believes that Minecraft's widespread recognition is key. As the best-selling video game ever, it's familiar to many, making it easier for people to judge the quality of AI-generated builds, even if they haven't played the game themselves. "Minecraft allows people to see the progress [of AI development] much more easily," Singh explained to TechCrunch. "People are used to Minecraft, used to the look and the vibe."

MC-Bench is supported by a team of eight volunteer contributors. Companies like Anthropic, Google, OpenAI, and Alibaba have provided their products for running benchmark prompts, though they are not otherwise involved with the project.

Singh envisions expanding MC-Bench beyond simple builds to more complex, goal-oriented tasks. "Games might just be a medium to test agentic reasoning that is safer than in real life and more controllable for testing purposes, making it more ideal in my eyes," he said.

Other Games as AI Benchmarks

Besides Minecraft, other games like Pokémon Red, Street Fighter, and Pictionary have been used as experimental benchmarks for AI. The challenge of benchmarking AI lies in its complexity, as traditional standardized tests often favor AI models due to their training methods, which excel in narrow problem-solving areas like rote memorization or basic extrapolation.

For instance, while OpenAI's GPT-4 can score in the 88th percentile on the LSAT, it struggles with simpler tasks like counting the number of Rs in "strawberry." Similarly, Anthropic's Claude 3.7 Sonnet achieved 62.3% accuracy on a software engineering benchmark but falls short in playing Pokémon compared to most five-year-olds.

MC-Bench: More Than Just a Programming Benchmark

Technically, MC-Bench is a programming benchmark because it requires AI models to write code to create builds like "Frosty the Snowman" or "a charming tropical beach hut on a pristine sandy shore." However, the platform's appeal lies in its accessibility. It's easier for users to evaluate the visual quality of a build than to analyze code, which broadens the project's reach and potential for data collection on model performance.

The debate continues on whether these scores truly reflect AI usefulness. Singh, however, believes they are a strong indicator. "The current leaderboard reflects quite closely to my own experience of using these models, which is unlike a lot of pure text benchmarks," he said. "Maybe [MC-Bench] could be useful to companies to know if they're heading in the right direction."