Microsoft's Copilot Now Offers Real-Time Visual Interaction and Voice Communication

Microsoft's Copilot has been a formidable rival to ChatGPT, and although it's been lagging recently, a new update might just bring it back into the race. On Thursday, Microsoft introduced Copilot Vision, a feature now available in preview for Pro subscribers through Copilot Labs, which allows Copilot to analyze and understand your online activities in real-time, offering verbal assistance.

When you activate Copilot Vision, it can read alongside you, help troubleshoot browsing issues, analyze websites, and offer insights based on what it sees. Essentially, it's a dedicated assistant for your browsing needs, ready to help whenever you need it.

I got a firsthand look at this feature during a Microsoft Copilot and Windows Event in NYC in October. The demo showcased how Copilot Vision can be practically applied. For instance, a user asked for help selecting an outfit from Pinterest. Copilot Vision not only suggested an outfit but also provided encouragement when the user expressed doubts about pulling it off, all through a natural, verbal interaction.

Microsoft also highlighted other potential uses for Copilot Vision, such as assisting with holiday shopping by pinpointing products on a page that match user preferences, or helping plan a museum visit by pulling out key information for the trip.

The feature is integrated into Microsoft Edge, positioned at the browser's bottom. Initially, Copilot Vision is being rolled out to a select group of Copilot Pro subscribers in the U.S. via Copilot Labs, and it will only function on specific websites. As feedback is gathered, Microsoft plans to expand its availability.

For those worried about privacy, Microsoft assures that Copilot Vision is opt-in only, giving users complete control over whether they want their browsing to be monitored. Moreover, any data shared during a session, including verbal interactions and contextual information, is deleted afterward, as per the company's blog post.

The Copilot Pro subscription costs $20 per month and comes with additional benefits such as priority access to the latest models, enhanced access to Copilot Voice, and integration into Microsoft 365 apps like Word, Excel, PowerPoint, OneNote, and Outlook.