AI Agents Gain New Ability to Navigate Websites and Apps with Copilot Studio's Latest Skill

Imagine an AI agent capable of not just surfing the web for research, but actually interacting with websites to get things done—all by itself. This is the vision behind Microsoft's latest innovation.

On Wednesday, Microsoft unveiled a new interactive skill for its Copilot Studio platform, which lets users create their own AI agents. This skill, dubbed "computer use," allows any agent you build to navigate and operate desktop applications or websites, performing tasks just as a human would.

This feature is rolling out through an early access research preview. With it, agents can click buttons, navigate menus, and fill out forms on screens, even if no API is available for the app or site. Microsoft put it succinctly: "If a person can use the app, the agent can too."

Your AI agents will be able to use popular browsers like Microsoft Edge, Chrome, and Firefox to carry out these tasks. The feature runs on Microsoft's backend, so you don't need to worry about setting up your own servers. All data generated stays on the Microsoft cloud and isn't used to train their AI models.

One of the coolest parts? These agents can adapt on the fly to changes in apps and websites. If a button or screen layout changes, your agent can still figure out what to do without needing manual updates from you. Plus, with the ability to "see" the screen, they can make decisions in real time, and they're equipped with reasoning capabilities to handle any hiccups without needing your help.

You don't need to know how to code to create an AI agent. Just use natural language to describe what you want the agent to do in Copilot Studio. You can test and tweak your instructions in a simulated mode, watching the steps play out before letting your agent loose. And you can review a detailed history of your agent's actions, complete with screenshots and reasoning steps.

Microsoft's blog post highlighted three scenarios where these AI agents could shine:

• Automated Data Entry: Need to input heaps of data from various sources into a central system? An AI agent can take this off your hands, saving you time and effort.
• Market Research: Marketing folks often need to gather data from multiple online sources. An AI agent could browse these sites and collect the necessary information autonomously.
• Invoice Processing: In finance, extracting data from invoices and entering it into an accounting system is common. An AI agent could do this automatically, ensuring the data gets where it needs to go.

While this all sounds promising, AI isn't perfect yet. Even simple AI bots can err, and with the added complexity of independent computer use, there's a risk your AI agent might make a wrong click or abandon its task.

The real test is in the doing. Microsoft is inviting Copilot Studio users to sign up for early access to this "computer use" feature. If you're not already using Copilot Studio, you can give it a try for free. Just use a work or school account to set it up, and you'll land on the main page where you can start describing the kind of agent you want to build.

Interested in more AI stories? Subscribe to our weekly newsletter, Innovation, for the latest updates.