Meta's AI Model Benchmarks: Misleading?
April 10, 2025
TimothyMitchell
10
So, Meta dropped their new AI model, Maverick, over the weekend, and it's already making waves by snagging second place on LM Arena. You know, that's the place where humans get to play judge and jury, comparing different AI models and picking their favorites. But, hold up, there's a twist! It turns out the Maverick version strutting its stuff on LM Arena isn't quite the same as the one you can download and play with as a developer.
Some eagle-eyed AI researchers on X (yeah, the platform formerly known as Twitter) spotted that Meta called the LM Arena version an "experimental chat version." And if you peek at the Llama website, there's a chart that spills the beans, saying the testing was done with "Llama 4 Maverick optimized for conversationality." Now, we've talked about this before, but LM Arena isn't exactly the gold standard for measuring AI performance. Most AI companies don't mess with their models just to score better on this test—or at least, they don't admit to it.
The thing is, when you tweak a model to ace a benchmark but then release a different "vanilla" version to the public, it's tough for developers to figure out how well the model will actually perform in real-world scenarios. Plus, it's kinda misleading, right? Benchmarks, flawed as they are, should give us a clear picture of what a model can and can't do across different tasks.
Researchers on X have been quick to notice some big differences between the Maverick you can download and the one on LM Arena. The Arena version is apparently all about emojis and loves to give you long, drawn-out answers.
Okay Llama 4 is def a littled cooked lol, what is this yap city pic.twitter.com/y3GvhbVz65
— Nathan Lambert (@natolambert) April 6, 2025
for some reason, the Llama 4 model in Arena uses a lot more Emojis
on together . ai, it seems better: pic.twitter.com/f74ODX4zTt
— Tech Dev Notes (@techdevnotes) April 6, 2025
We've reached out to Meta and the folks at Chatbot Arena, who run LM Arena, to see what they have to say about all this. Stay tuned!
Related article
Meta Defends Llama 4 Release, Cites Bugs as Cause of Mixed Quality Reports
Over the weekend, Meta, the powerhouse behind Facebook, Instagram, WhatsApp, and Quest VR, surprised everyone by unveiling their latest AI language model, Llama 4. Not just one, but three new versions were introduced, each boasting enhanced capabilities thanks to the "Mixture-of-Experts" architectur
Law Professors Support Authors in AI Copyright Battle Against Meta
A group of copyright law professors has thrown their support behind authors suing Meta, alleging that the tech giant trained its Llama AI models on e-books without the authors' consent. The professors filed an amicus brief on Friday in the U.S. District Court for the Northern District of California,
Meta AI will soon train on EU users’ data
Meta has recently revealed its plans to train its AI using data from EU users of its platforms, such as Facebook and Instagram. This initiative will tap into public posts, comments, and even chat histories with Meta AI, but rest assured, your private messages with friends and family are off-limits.
Comments (35)
0/200
![JerryGonzalez]()
JerryGonzalez
April 10, 2025 at 10:18:45 AM GMT
Meta's AI model benchmarks seem a bit off to me. Maverick got second place, but I've used it and it's not that great. The interface is clunky and the results are hit or miss. Maybe they're just trying to hype it up? I'd give it a pass for now.
0
![CarlKing]()
CarlKing
April 10, 2025 at 10:18:45 AM GMT
MetaのAIモデルのベンチマークは私には少しおかしいように感じます。Maverickは2位を獲得しましたが、使ってみた感じではそれほど良くありません。インターフェースがぎこちなく、結果も当たり外れがあります。もしかしたら、ただ盛り上げようとしているだけかもしれませんね。今はパスしておきます。
0
![SamuelEvans]()
SamuelEvans
April 10, 2025 at 10:18:45 AM GMT
Meta의 AI 모델 벤치마크가 내겐 좀 이상해 보여. Maverick이 2위를 했지만, 써보니 그리 대단하지 않아. 인터페이스가 어색하고 결과도 들쑥날쑥해. 어쩌면 그냥 과대광고하려고 하는 건지도 몰라. 지금은 패스할게.
0
![BenWalker]()
BenWalker
April 10, 2025 at 10:18:45 AM GMT
Os benchmarks do modelo de IA da Meta parecem um pouco estranhos para mim. O Maverick ficou em segundo lugar, mas eu usei e não é tão bom assim. A interface é desajeitada e os resultados são inconsistentes. Talvez eles estejam apenas tentando criar hype? Eu passaria por agora.
0
![RobertLewis]()
RobertLewis
April 10, 2025 at 10:18:45 AM GMT
Los benchmarks del modelo de IA de Meta me parecen un poco extraños. Maverick quedó en segundo lugar, pero lo he usado y no es tan bueno. La interfaz es torpe y los resultados son inconsistentes. ¿Quizás solo están tratando de generar hype? Por ahora, lo dejaría pasar.
0
![KevinBaker]()
KevinBaker
April 11, 2025 at 6:25:04 PM GMT
I tried Meta's Maverick and it's pretty good, but those benchmarks seem a bit off to me. It's not as smooth as they claim, and sometimes it's just plain wrong. I'm not sure if it's worth the hype. Maybe they need to tweak their testing methods?
0
April 10, 2025
TimothyMitchell
10
So, Meta dropped their new AI model, Maverick, over the weekend, and it's already making waves by snagging second place on LM Arena. You know, that's the place where humans get to play judge and jury, comparing different AI models and picking their favorites. But, hold up, there's a twist! It turns out the Maverick version strutting its stuff on LM Arena isn't quite the same as the one you can download and play with as a developer.
Some eagle-eyed AI researchers on X (yeah, the platform formerly known as Twitter) spotted that Meta called the LM Arena version an "experimental chat version." And if you peek at the Llama website, there's a chart that spills the beans, saying the testing was done with "Llama 4 Maverick optimized for conversationality." Now, we've talked about this before, but LM Arena isn't exactly the gold standard for measuring AI performance. Most AI companies don't mess with their models just to score better on this test—or at least, they don't admit to it.
The thing is, when you tweak a model to ace a benchmark but then release a different "vanilla" version to the public, it's tough for developers to figure out how well the model will actually perform in real-world scenarios. Plus, it's kinda misleading, right? Benchmarks, flawed as they are, should give us a clear picture of what a model can and can't do across different tasks.
Researchers on X have been quick to notice some big differences between the Maverick you can download and the one on LM Arena. The Arena version is apparently all about emojis and loves to give you long, drawn-out answers.
Okay Llama 4 is def a littled cooked lol, what is this yap city pic.twitter.com/y3GvhbVz65
— Nathan Lambert (@natolambert) April 6, 2025
for some reason, the Llama 4 model in Arena uses a lot more Emojis
— Tech Dev Notes (@techdevnotes) April 6, 2025
on together . ai, it seems better: pic.twitter.com/f74ODX4zTt
We've reached out to Meta and the folks at Chatbot Arena, who run LM Arena, to see what they have to say about all this. Stay tuned!
Meta Defends Llama 4 Release, Cites Bugs as Cause of Mixed Quality Reports
Over the weekend, Meta, the powerhouse behind Facebook, Instagram, WhatsApp, and Quest VR, surprised everyone by unveiling their latest AI language model, Llama 4. Not just one, but three new versions were introduced, each boasting enhanced capabilities thanks to the "Mixture-of-Experts" architectur
Law Professors Support Authors in AI Copyright Battle Against Meta
A group of copyright law professors has thrown their support behind authors suing Meta, alleging that the tech giant trained its Llama AI models on e-books without the authors' consent. The professors filed an amicus brief on Friday in the U.S. District Court for the Northern District of California,
Meta AI will soon train on EU users’ data
Meta has recently revealed its plans to train its AI using data from EU users of its platforms, such as Facebook and Instagram. This initiative will tap into public posts, comments, and even chat histories with Meta AI, but rest assured, your private messages with friends and family are off-limits.
April 10, 2025 at 10:18:45 AM GMT
Meta's AI model benchmarks seem a bit off to me. Maverick got second place, but I've used it and it's not that great. The interface is clunky and the results are hit or miss. Maybe they're just trying to hype it up? I'd give it a pass for now.
0
April 10, 2025 at 10:18:45 AM GMT
MetaのAIモデルのベンチマークは私には少しおかしいように感じます。Maverickは2位を獲得しましたが、使ってみた感じではそれほど良くありません。インターフェースがぎこちなく、結果も当たり外れがあります。もしかしたら、ただ盛り上げようとしているだけかもしれませんね。今はパスしておきます。
0
April 10, 2025 at 10:18:45 AM GMT
Meta의 AI 모델 벤치마크가 내겐 좀 이상해 보여. Maverick이 2위를 했지만, 써보니 그리 대단하지 않아. 인터페이스가 어색하고 결과도 들쑥날쑥해. 어쩌면 그냥 과대광고하려고 하는 건지도 몰라. 지금은 패스할게.
0
April 10, 2025 at 10:18:45 AM GMT
Os benchmarks do modelo de IA da Meta parecem um pouco estranhos para mim. O Maverick ficou em segundo lugar, mas eu usei e não é tão bom assim. A interface é desajeitada e os resultados são inconsistentes. Talvez eles estejam apenas tentando criar hype? Eu passaria por agora.
0
April 10, 2025 at 10:18:45 AM GMT
Los benchmarks del modelo de IA de Meta me parecen un poco extraños. Maverick quedó en segundo lugar, pero lo he usado y no es tan bueno. La interfaz es torpe y los resultados son inconsistentes. ¿Quizás solo están tratando de generar hype? Por ahora, lo dejaría pasar.
0
April 11, 2025 at 6:25:04 PM GMT
I tried Meta's Maverick and it's pretty good, but those benchmarks seem a bit off to me. It's not as smooth as they claim, and sometimes it's just plain wrong. I'm not sure if it's worth the hype. Maybe they need to tweak their testing methods?
0
