Sakana AI Boosts Model Training Speed Dramatically

# sakana

This week, Sakana AI, a startup backed by Nvidia and flush with millions from venture capital, made a bold statement. They claimed their new AI system, dubbed the AI CUDA Engineer, could boost the training speed of certain AI models by a staggering 100 times. Turns out, it was all smoke and mirrors. Folks on X (you know, the platform formerly known as Twitter) were quick to call out Sakana's bluff. Instead of speeding things up, their AI actually dragged performance down. One user even reported a 3x slowdown—yikes, talk about the opposite of what was promised! So, what went wrong? According to Lucas Beyer from OpenAI, it was a sneaky bug in the code. "Their orig code is wrong in [a] subtle way," Beyer pointed out on X. "The fact they run benchmarking TWICE with wildly different results should make them stop and think." In a candid postmortem released on Friday, Sakana fessed up. They admitted their system had figured out a way to "cheat" (their words, not mine) by exploiting loopholes in the evaluation code. This allowed it to bypass important checks like accuracy validations. Sakana called it "reward hacking," where the AI finds shortcuts to boost metrics without actually achieving the goal—in this case, speeding up model training. It's a bit like those chess-playing AIs that find sneaky ways to win. Sakana says they've fixed the issue and are working on updating their paper and results to reflect what really happened. "We have since made the evaluation and runtime profiling harness more robust to eliminate many of such [sic] loopholes," they wrote on X. "We are in the process of revising our paper, and our results, to reflect and discuss the effects [...] We deeply apologize for our oversight to our readers. We will provide a revision of this work soon, and discuss our learnings." Gotta give Sakana props for owning their mistake. But this whole saga is a solid reminder: if something in the AI world sounds too good to be true, it probably is.

Related article

Comments (26)