AI-Generated Paper Passes Peer Review, Sakana Claims, But Details Are Nuanced

Japanese AI startup Sakana recently made waves by claiming that its AI system, The AI Scientist-v2, generated one of the first peer-reviewed scientific publications. However, there are some important details to consider before we get too excited.

The debate over AI's role in science is heating up. Some researchers believe AI isn't ready to be a "co-scientist," while others see potential but recognize we're still in the early stages. Sakana falls into the latter group.

The company used The AI Scientist-v2 to create a paper that was submitted to a workshop at ICLR, a well-respected AI conference. Sakana worked with the University of British Columbia and the University of Oxford to submit three AI-generated papers to this workshop. The AI handled everything from hypotheses to experiments, code, data analysis, visualizations, and even the titles.

"We generated research ideas by providing the workshop abstract and description to the AI," Robert Lange, a research scientist and founding member at Sakana, told TechCrunch via email. "This ensured that the generated papers were on topic and suitable submissions."

One of the three papers was accepted by the ICLR workshop. It focused on critiquing training techniques for AI models. However, Sakana withdrew the paper before it could be published, citing transparency and respect for ICLR conventions.

But let's not get carried away just yet. Sakana admitted in their blog post that their AI made some "embarrassing" citation errors, like attributing a method to a 2016 paper instead of the original 1997 work.

Also, the paper didn't go through as much scrutiny as other peer-reviewed publications. It was withdrawn after the initial peer review, so it didn't get a "meta-review" from the workshop organizers, who might have rejected it.

Another thing to keep in mind is that conference workshops often have higher acceptance rates than the main conference track. Sakana mentioned this in their blog post and noted that none of their AI-generated studies met their internal standards for the ICLR conference track.

Matthew Guzdial, an AI researcher and assistant professor at the University of Alberta, called Sakana's results "a bit misleading."

"The Sakana folks selected the papers from some number of generated ones, meaning they were using human judgment in terms of picking outputs they thought might get in," he said via email. "What I think this shows is that humans plus AI can be effective, not that AI alone can create scientific progress."

Mike Cook, a research fellow at King's College London specializing in AI, questioned the rigor of the peer reviewers and workshop.

"New workshops, like this one, are often reviewed by more junior researchers," he told TechCrunch. "It's also worth noting that this workshop is about negative results and difficulties — which is great, I've run a similar workshop before — but it's arguably easier to get an AI to write about a failure convincingly."

Cook wasn't surprised that an AI could pass peer review, given that AI is good at writing human-sounding prose. He pointed out that partly AI-generated papers passing journal review isn't new, and it raises ethical questions for the scientific community.

AI's technical issues, like its tendency to "hallucinate," make many scientists cautious about using it for serious work. There's also a fear that AI could just add noise to the scientific literature, rather than advancing knowledge.

"We need to ask ourselves whether [Sakana's] result is about how good AI is at designing and conducting experiments, or whether it's about how good it is at selling ideas to humans — which we know AI is great at already," Cook said. "There's a difference between passing peer review and contributing knowledge to a field."

To Sakana's credit, they don't claim that their AI can produce groundbreaking or even particularly novel scientific work. Their goal was to "study the quality of AI-generated research" and highlight the need for "norms regarding AI-generated science."

"There are difficult questions about whether [AI-generated] science should be judged on its own merits first to avoid bias against it," the company wrote. "Going forward, we will continue to exchange opinions with the research community on the state of this technology to ensure that it does not develop into a situation in the future where its sole purpose is to pass peer review, thereby substantially undermining the meaning of the scientific peer review process."