Self-Learning AI Agents to Transform Web Experience: A Preparation Guide

In a new paper, prominent AI researchers David Silver and Richard Sutton propose that artificial intelligence is entering a transformative "Era of Experience." They suggest AI systems will increasingly move beyond dependence on human-provided data, instead advancing through direct interaction with and data collection from the world.

Although the paper is a conceptual and forward-looking piece, its insights carry direct relevance for enterprises planning to develop and integrate future AI agents and systems.

Both Silver and Sutton are respected scientists with a history of accurate predictions about AI's trajectory. Their foresight is evident in today's most advanced AI systems. In his influential 2019 essay "The Bitter Lesson," reinforcement learning pioneer Sutton argued that the most significant long-term AI breakthroughs consistently result from scaling general-purpose search and learning with massive computation, not primarily from embedding complex human knowledge.

David Silver, a principal scientist at DeepMind, played a leading role in developing AlphaGo, AlphaZero, and AlphaStar—major milestones in deep reinforcement learning. He also co-authored a 2021 paper asserting that reinforcement learning combined with a well-designed reward signal could ultimately produce highly advanced AI.

Today's most sophisticated large language models (LLMs) exemplify these two concepts. The wave of powerful LLMs since GPT-3 has largely relied on scaling computational power and data to absorb immense knowledge. More recent reasoning models, such as DeepSeek-R1, further demonstrate that reinforcement learning with a straightforward reward signal is sufficient for acquiring complex reasoning capabilities.

What is the Era of Experience?

The "Era of Experience" builds upon concepts Sutton and Silver have championed, now contextualized for recent AI progress. The authors note that "the pace of progress driven solely by supervised learning from human data is demonstrably slowing, signalling the need for a new approach."

This new approach demands a fresh data source, generated in a way that continuously improves as the agent grows more capable. "This can be achieved by allowing agents to learn continually from their own experience, i.e., data that is generated by the agent interacting with its environment," Sutton and Silver explain. They contend that eventually, "experience will become the dominant medium of improvement and ultimately dwarf the scale of human data used in today’s systems."

According to the authors, future AI systems will "break through the limitations of human-centric AI systems" not only by learning from experiential data but also across four key dimensions:

1. Streams: Rather than operating in isolated episodes, AI agents will "have their own stream of experience that progresses, like humans, over a long time-scale." This enables long-term planning and gradual adaptation of behavior. Early signs of this appear in AI systems with extensive context windows and memory architectures that update continuously through user interaction.
   
2. Actions and observations: Moving beyond human-privileged actions and observations, agents in the era of experience will operate autonomously in the real world. We see this in agentic systems that interface with external applications and resources using tools like computer control and the Model Context Protocol (MCP).
   
3. Rewards: While current reinforcement learning systems depend heavily on human-designed reward functions, future AI agents should create their own dynamic reward functions. These would evolve over time, aligning user preferences with real-world feedback from the agent's actions and observations. Early self-designing reward systems, like Nvidia’s DrEureka, provide a glimpse of this future.
   
4. Planning and reasoning: Current reasoning models are often designed to mimic human thought. The authors propose that "More efficient mechanisms of thought surely exist, using non-human languages that may, for example, utilise symbolic, distributed, continuous, or differentiable computations." AI agents should engage with the world, using observed data to test, refine, and update their reasoning while building an internal world model.

The concept of AI agents adapting via reinforcement learning isn't new, but historically, such agents were confined to tightly controlled environments like board games. Now, agents capable of navigating complex settings—such as operating computers—coupled with progress in reinforcement learning, are poised to overcome these constraints, accelerating the shift into the era of experience.

What does it mean for the enterprise?

An important observation in Sutton and Silver's paper for real-world applications states: "The agent may use ‘human-friendly’ actions and observations such as user interfaces, that naturally facilitate communication and collaboration with the user. The agent may also take ‘machine-friendly’ actions that execute code and call APIs, allowing the agent to act autonomously in service of its goals."

The era of experience implies that developers must design applications for both human users and AI agents. Machine-friendly actions necessitate secure, accessible APIs, available directly or through interfaces like MCP. It also involves creating agents that are discoverable via protocols such as Google’s Agent2Agent. Designing APIs and agentic interfaces to grant access to both actions and observations will let agents progressively reason and learn from their interactions with your software.

If Sutton and Silver's vision materializes, billions of agents will eventually operate across the web—and later, the physical world—to perform tasks. Their behavior and requirements will differ fundamentally from human users. Establishing agent-friendly interaction methods will be crucial for leveraging future AI systems effectively and mitigating potential risks.

"By building upon the foundations of RL and adapting its core principles to the challenges of this new era, we can unlock the full potential of autonomous learning and pave the way to truly superhuman intelligence," Sutton and Silver conclude.

DeepMind declined to provide additional comments for this story.