AI has grown beyond human knowledge, says Google's DeepMind unit

Advancing AI Beyond Traditional Tests: The Rise of Experiential Learning

The field of artificial intelligence (AI) is buzzing with efforts to push generative AI beyond the confines of traditional benchmarks, like the Turing Test, which many models have already surpassed. The focus now shifts to developing AI that isn't just designed to ace these tests but to evolve through a more dynamic process.

Researchers at Google's DeepMind, including legends like David Silver and Richard Sutton, argue in their paper Welcome to the Era of Experience that the key to unlocking new AI capabilities lies not in the tests, but in the way AI is trained. They suggest that the static datasets used currently are too limited to foster significant advancements.

Their solution? AI needs to engage with the world in a more experiential way, learning from interactions and setting goals based on environmental feedback. "Incredible new capabilities will arise once the full potential of experiential learning is harnessed," they assert. Silver, known for his work on AlphaZero, which bested humans at Chess and Go, and Sutton, a pioneer in reinforcement learning, propose a new approach called "streams" to transcend the limitations of current large language models (LLMs).

From Reinforcement Learning to Generative AI: A Shift and Its Consequences

After the successes of AlphaZero and AlphaGo, the AI community saw a shift towards generative AI models like ChatGPT, which largely moved away from reinforcement learning. While this allowed AI to handle a broader range of spontaneous human inputs, it also meant losing the self-discovery aspect that reinforcement learning enabled.

According to Silver and Sutton, current LLMs rely heavily on human judgment at the prompt stage, which limits their potential. "The agent cannot discover better strategies underappreciated by the human rater," they explain. Moreover, the short, clipped nature of prompt interactions doesn't allow AI to develop beyond simple question-and-answer exchanges.

Introducing Streams: A New Paradigm for AI Learning

The proposed "Age of Experience" involves AI agents operating within "streams" of continuous interaction, much like humans learn over a lifetime. Silver and Sutton envision AI agents with their own long-term streams of experience, enabling them to pursue goals beyond immediate tasks.

This approach builds on reinforcement learning, where AI agents interact with a model of the world, receiving feedback in the form of rewards. These rewards help the AI learn what actions are valuable in different contexts. The researchers point out that the world is rich with signals that can serve as rewards, from economic indicators to health metrics.

Building AI Agents with Long-Term Goals

To kickstart this process, AI developers could use a "world model" simulation, allowing the AI to make predictions, test them in reality, and refine its model based on the outcomes. As the AI continues to interact with the world, its understanding evolves, adapting to new data and correcting errors.

Humans would still play a role in setting broad goals, such as improving fitness or learning a new language, with the AI's reward function aligned to these objectives. This setup could lead to AI assistants that track and advise on health or education over extended periods, or even pursue ambitious scientific goals like discovering new materials or reducing carbon emissions.

The Future of AI: Beyond Reasoning to Experiential Learning

The researchers believe that AI agents capable of long-term experiential learning could surpass current "reasoning" AI models like Gemini and DeepSeek's R1. They argue that reasoning models often mimic human thought, which can be limited by outdated assumptions. In contrast, experiential agents could unlock unprecedented capabilities, leading to a future vastly different from what we've seen.

However, this shift also brings risks, such as job displacement and the challenge of controlling AI agents that can operate autonomously over long periods. On the flip side, adaptable AI could learn to mitigate negative impacts on humans by adjusting its behavior based on feedback.

Silver and Sutton are confident that experiential data will far surpass the scale and quality of human-generated data, potentially leading to artificial general intelligence or super-intelligence. This paradigm shift, coupled with advancements in reinforcement learning, could unlock new capabilities beyond human reach in many domains.

Silver further discussed these ideas in a recent DeepMind podcast, highlighting the potential and challenges of moving AI into the era of experience.