"IQ: A Flawed Measure for AI Assessment"

# ai # iq tests

In a recent press event, OpenAI's CEO, Sam Altman, shared his thoughts on the rapid advancement of AI's "IQ." He mentioned, "Very roughly, it feels to me like — this is not scientifically accurate, this is just a vibe or spiritual answer — every year we move one standard deviation of IQ." Altman isn't alone in using IQ as a yardstick for AI's progress; social media influencers have also been known to put AI models through IQ tests and share the results. However, many experts argue that using IQ to gauge AI's capabilities is not only inadequate but also misleading. Sandra Wachter, a researcher at Oxford focusing on tech and regulation, told TechCrunch, "It can be very tempting to use the same measures we use for humans to describe capabilities or progress, but this is like comparing apples with oranges." During the press conference, Altman seemed to equate IQ with intelligence. Yet, IQ tests are more about relative, not absolute, measures of specific types of intelligence. They're generally seen as good indicators of logic and abstract reasoning, but they fall short when it comes to practical intelligence—the kind that helps you figure out how to make things work. Plus, they only give a snapshot of someone's abilities. Wachter pointed out, "IQ is a tool to measure human capabilities — a contested one, no less — based on what scientists believe human intelligence looks like. But you can't use the same measure to describe AI capabilities. A car is faster than humans, and a submarine is better at diving. But this doesn't mean cars or submarines surpass human intelligence. You're equating one aspect of performance with human intelligence, which is much more complex." The origins of IQ tests are linked to eugenics, a discredited theory about improving people through selective breeding. To do well on these tests, you need a good working memory and familiarity with Western cultural norms, which can introduce bias. Os Keyes, a doctoral candidate at the University of Washington studying ethical AI, believes that if an AI model does well on an IQ test, it says more about the test's shortcomings than the model's prowess. Keyes said, "These tests are pretty easy to game if you have a practically infinite amount of memory and patience. IQ tests are a highly limited way of measuring cognition, sentience, and intelligence, something we've known since before the invention of the digital computer itself." AI might have an unfair edge on IQ tests because models have vast memory and knowledge at their fingertips. They're often trained on public web data, which includes plenty of IQ test questions. Mike Cook, a research fellow at King's College London specializing in AI, noted, "Tests tend to repeat very similar patterns — a pretty foolproof way to raise your IQ is to practice taking IQ tests, which is essentially what every model has done. When I learn something, I don't get it piped into my brain with perfect clarity 1 million times, unlike AI, and I can't process it with no noise or signal loss, either." Cook also pointed out that IQ tests, with their inherent biases, were designed for humans to assess general problem-solving skills. They're not suitable for AI, which tackles problems differently. He said, "A crow might be able to use a tool to recover a treat from a box, but that doesn't mean it can enroll at Harvard. When I solve a mathematics problem, my brain is also contending with its ability to read the words on the page correctly, to not think about the shopping I need to do on the way home, or if it's too cold in the room right now. In other words, human brains contend with a lot more things when they solve a problem — any problem at all, IQ tests or otherwise — and they do it with a lot less help than AI." Heidy Khlaaf, chief AI scientist at the AI Now Institute, told TechCrunch that we need better ways to test AI. She said, "In the history of computation, we haven't compared computing abilities to that of humans' precisely because the nature of computation means systems have always been able to complete tasks already beyond human ability. This idea that we directly compare systems' performance against human abilities is a recent phenomenon that is highly contested, and what surrounds the controversy of the ever-expanding — and moving — benchmarks being created to evaluate AI systems."

Related article

Comments (45)