Meredith Whittaker Highlights 'Profound' Security, Privacy Risks in Agentic AI

At the SXSW conference in Austin, Texas, Signal President Meredith Whittaker raised serious concerns about the privacy risks associated with agentic AI. She vividly described the use of AI agents as "putting your brain in a jar," highlighting the unsettling nature of this emerging technology. Whittaker pointed out that AI agents, which are promoted as tools to enhance daily life by managing tasks such as finding concerts, booking tickets, and scheduling events, pose significant privacy and security challenges.

"So we can just put our brain in a jar because the thing is doing that and we don’t have to touch it, right?" Whittaker mused, emphasizing the hands-off approach that AI agents encourage. She went on to detail the extensive access these agents would require, including control over web browsers, credit card information, calendars, and messaging apps. "It would need to be able to drive that [process] across our entire system with something that looks like root permission, accessing every single one of those databases — probably in the clear, because there’s no model to do that encrypted," she warned.

Whittaker also addressed the processing power needed for these AI agents, noting that such operations would likely occur on cloud servers rather than on the user's device. "That’s almost certainly being sent to a cloud server where it’s being processed and sent back. So there’s a profound issue with security and privacy that is haunting this hype around agents, and that is ultimately threatening to break the blood-brain barrier between the application layer and the OS layer by conjoining all of these separate services [and] muddying their data," she concluded.

She expressed particular concern about the implications for privacy if a messaging app like Signal were to integrate with AI agents. Such integration would compromise the confidentiality of messages, as the agent would need to access the app to send texts and also retrieve data to summarize those communications.

Whittaker's remarks came after she discussed the broader AI industry's reliance on a surveillance model that involves mass data collection. She criticized the "bigger is better AI paradigm," which prioritizes data accumulation, warning of its potential negative consequences. With agentic AI, she cautioned, we risk further eroding privacy and security in pursuit of a "magic genie bot that’s going to take care of the exigencies of life."