Google’s AI Mode can now see and search with images

Google is enhancing its AI Mode chatbot with new multimodal capabilities, enabling it to "see" and respond to questions about images. This update, which integrates a custom version of Gemini AI with Google's Lens image recognition technology, allows users to take or upload photos and receive detailed, informative responses complete with relevant links. The rollout of this feature began today and is accessible through the Google app on both Android and iOS devices.

Robby Stein, VP of Product for Google Search, explains, "AI Mode builds on our years of work in visual search, but takes it a step further. With Gemini’s multimodal capabilities, AI Mode can now understand the entire scene within an image, including how objects interact and their specific materials, colors, shapes, and arrangements."

The update employs a "fan-out technique" that sends out multiple queries about the image and its elements, resulting in responses that are both nuanced and contextually relevant. This allows AI Mode to identify items like books in an image, suggest similar titles with good reviews, and even answer follow-up questions to refine recommendations.

Designed as a competitor to Perplexity and ChatGPT Search, AI Mode offers a chatbot-like experience, generating AI-powered summaries from Google's vast search index. Initially exclusive to Google One AI Premium subscribers through Labs last month, AI Mode is now expanding to "millions more" Labs users in the US, extending beyond just the paid AI Premium subscribers.