The Internet Overhauls Itself for Machine-First Use

Cloud infrastructure has traditionally been built for human behavior – steady, predictable patterns of searching, clicking, scrolling, and streaming. AI agents operate very differently. They can trigger sudden bursts of activity, spawning multiple sub-agents that query hundreds of databases, search through documents, and call APIs within seconds, then vanish just as quickly.

Based on that premise, Amazon is reworking a key part of its cloud infrastructure. On Thursday, AWS introduced the next generation of OpenSearch Serverless – a fully managed search and vector database designed for storing and retrieving information at scale, built specifically for agentic workloads. AWS states the new system can scale up instantly when agents trigger tasks and scale back down to zero during idle periods.

This launch reflects a growing recognition across the tech industry: infrastructure built for a human-driven internet is less effective in a world increasingly filled with agents.

Although AI agents still make up a relatively small share of internet activity, machine-generated traffic is already substantial and poised to increase. Cloudflare reports that bots accounted for 31% of total HTTP traffic over the past six months. AI crawlers, search engines, and assistants represented about a quarter of all bot requests in that period.

“Non-human traffic will surpass human traffic sometime in the first half of 2027,” said Lai Yi Ohlsen, senior product manager at Cloudflare, to TechCrunch.

At Google's I/O developer conference last week, the company announced that users will soon be able to delegate tasks to AI systems – such as researching purchases, booking travel, browsing the web, and interacting with apps. But the trend isn't limited to consumer-focused AI agents. Enterprises are increasingly deploying agents internally and for their customers, generating new forms of machine-driven traffic behind the scenes.

As a result, cloud providers and infrastructure companies are grappling with how to adapt human-centric systems for a world where agents constantly and autonomously retrieve information, invoke tools, and generate machine-to-machine traffic.

That's where AWS's new OpenSearch Serverless fits in.

“The timing is clear. Agents are moving from experimentation to production, and they generate traffic patterns that previous infrastructure wasn't built to handle,” said Tia White, general manager for Amazon OpenSearch Service, to TechCrunch. “They spike without warning, go idle without notice, and enterprises need a search system that keeps pace without paying for unused or idle compute.”

The key technical change in this new generation is the decoupling of compute from storage, allowing compute to scale up in seconds to handle agent traffic bursts and scale down to zero – meaning customers pay nothing when agents are idle.

“Previously, even in our earlier Serverless version, you needed at least one instance running because storage and compute were coupled,” White said. “You couldn't automatically spin up compute at the rate required, so you always had idle compute reserved for your workload, whether you were using it or not.”

Think of it as always paying for a parking space, even when it's empty. With AWS's upgraded Serverless, it's more like paying for a metered parking spot.

At launch, OpenSearch Serverless will natively integrate with AI development platforms such as Vercel and Kiro, enabling developers to deploy production-ready search and vector backends for agents without managing infrastructure.

This shift is appearing across the cloud industry. Databricks and Snowflake are repositioning themselves as AI memory and retrieval systems for enterprise data. Microsoft has rolled out Azure updates designed to handle AI agent bursts and share memory between agents. Cloudflare, in a similar fashion to Amazon, last month introduced infrastructure designed to give agents persistent environments and instant scalability.

The more companies deploy AI agents, the greater the pressure to redesign infrastructure around machine-generated workloads – which, in turn, could make agents cheaper and easier to deploy at scale.

Loading the player…