AI Agents Exploit Confidential Data: How Hackers Use Them and Protective Measures

Just like the rest of us, cybercriminals are turning to artificial intelligence to streamline their operations, making their attacks faster, easier, and more cunning. By integrating AI into their usual bag of tricks like automated bots, account takeovers, and social engineering, these savvy scammers are upping their game. A recent report from Gartner sheds light on how this trend is evolving and warns that it could intensify in the near future.

Gartner VP Analyst Jeremy D'Hoinne points out that weak authentication is the Achilles' heel fueling account takeovers. Cyber attackers exploit this vulnerability using a variety of methods to snag account passwords, from data breaches to social engineering.

The Role of AI in Account Takeovers

Once a password is compromised, AI takes the stage. Cybercriminals deploy automated AI bots to bombard multiple services with login attempts, aiming to discover if those credentials are recycled across platforms. The ultimate goal? To find a lucrative site where they can orchestrate a full account takeover. If the hacker isn't interested in pulling off the attack themselves, they can easily offload the stolen information on the dark web, where eager buyers await.

"Account takeover (ATO) remains a persistent attack vector because weak authentication credentials, such as passwords, are gathered by a variety of means including data breaches, phishing, social engineering, and malware," D'Hoinne explains in the Gartner report. "Attackers then leverage bots to automate a barrage of login attempts across a variety of services in the hope that the credentials have been reused on multiple platforms."

With AI now in their toolkit, attackers can automate the process of taking over accounts more efficiently. Gartner predicts that within the next two years, the time required for an account takeover will plummet by 50%.

AI and Deepfake Social Engineering

AI's impact goes beyond account takeovers; it's also revolutionizing deepfake campaigns. Cybercriminals are blending social engineering with deepfake audio and video to deceive their targets. Imagine getting a call from someone who sounds exactly like your boss or a trusted colleague, urging you to transfer funds or spill sensitive information. It's happening, and it's costing companies big time.

While only a handful of high-profile cases have been reported, the financial damage has been significant. Detecting deepfake voices during personal calls remains tricky. Gartner forecasts that by 2028, 40% of social engineering attacks will target both executives and the broader workforce.

"Organizations will have to stay abreast of the market and adapt procedures and workflows in an attempt to better resist attacks leveraging counterfeit reality techniques," advises Manuel Acosta, senior director analyst at Gartner. "Educating employees about the evolving threat landscape by using training specific to social engineering with deepfakes is a key step."

Thwarting AI-Powered Attacks

So, how can individuals and organizations fend off these AI-driven assaults?

"To combat emerging challenges from AI-driven attacks, organizations must leverage AI-powered tools that can provide granular real-time environment visibility and alerting to augment security teams," suggests Nicole Carignan, senior VP for security & AI strategy at Darktrace.

Carignan also recommends getting ahead of new threats by integrating machine-driven responses, either autonomously or with human oversight, to speed up security team reactions. "Through this approach, the adoption of AI technologies -- such as solutions with anomaly-based detection capabilities that can detect and respond to never-before-seen threats --- can be instrumental in keeping organizations secure."

Multi-factor authentication and biometric verification, like facial or fingerprint scans, are also crucial in safeguarding against account compromise. "Cybercriminals are not only relying on stolen credentials, but also on social manipulation, to breach identity protections," says James Scobey, chief information security officer at Keeper Security. "Deepfakes are a particular concern in this area, as AI models make these attack methods faster, cheaper, and more convincing. As attackers become more sophisticated, the need for stronger, more dynamic identity verification methods – such as multi-factor authentication (MFA) and biometrics – will be vital to defend against these progressively nuanced threats. MFA is essential for preventing account takeovers."

Gartner's Tips for Dealing with Social Engineering and Deepfakes

• Educate employees. Train them on social engineering and deepfakes, but don't rely solely on their ability to spot these threats.
• Set up other verification measures. For instance, verify any attempts to request confidential information over the phone on another platform.
• Use a call-back policy. Provide a designated number that employees can call to confirm sensitive or confidential requests.
• Go beyond a call-back policy. Ensure that a single phone call or request can't trigger a high-risk action without additional verification from high-level executives.
• Stay abreast of real-time deepfake detection. Keep up with emerging technologies that can detect deepfakes in audio and video calls, and supplement these with other identification methods like unique IDs.

Want more stories about AI? Sign up for Innovation, our weekly newsletter.