I’ve used AI libraries and APIs on a few different projects, using it for things like anomaly detection in large IoT datasets and for advanced threat detection capabilities in My Security Console. I feel compelled to also credit AI for the image that is part of this blog post, it was created in a matter of minutes by OpenAI’s DALL-E model simply asking for “A realistic image of a humanoid robot on a boat fishing”. But I digress; with all the stories in the news about AI and ChatGPT, I was recently asked to share some thoughts about how AI is being used by the “bad guys” to perpetrate scams or carry out cyberattacks.

Hackers are a creative bunch and there’s no telling what innovative (albeit nefarious) uses of AI will surface as tools like ChatGPT become more mainstream and capable. Blogs posts from Check Point Research show how these tools can be used to create a full infection flow, and that AI-generated code is already starting to appear in Dark Web hacker forums.

There are also documented cases of AI being used in social engineering attacks, like the case of AI voice cloning being used to authorize $35M in fraudulent bank transfers, and in fake kidnapping scams where a person’s voice is cloned using speech samples from videos posted to social media accounts and then used to convince loved ones to pay a ransom for their safe return.

While these types of attacks can be devastating, they are relatively small-scale because they are highly targeted requiring time and skill to set up, and carry a much higher risk for the perpetrators to be brought to justice. In comparison, simpler phishing attacks that attempt to steal login credentials or credit card information are much easier to carry out by email and can be launched against hundreds of thousands of potential victims with minimal risk to the perpetrators. The average click rate of a mass phishing campaign is somewhat low, mostly because so many of the emails are generic and unconvincing.  Having said that, phishing is the second leading cause of data breaches and the costliest attack vector due to the impact and time to identify & contain. This is an area I expect we will see AI used successfully in large-scale phishing attacks in the very near future, with research already showing that AI-generated phishing email is nearly two times more successful at getting victims to click on a link or open an attachment.

With that in mind, I set out to see how difficult it would be to write code that could carry out an AI-assisted mass phishing campaign with a twist: one that sent well-written personalized messages to the victims. After some initial research, I decided to code it in Python and use the API for OpenAI’s text-davinci-003 model. Since I had been asked to provide a demonstration of how this could work, I created a simple program that scrapes a potential victim’s online bio and sends a fake recruiting email with malicious links and attachments. It turns out this is really easy to do in no time with just a couple hundred lines of code. Each API call to text-davinci-003 that took a scaped bio and generated a customized phishing email took about 10 seconds on average to complete with a single thread, so doing this at scale would require a multithreaded approach, but that can be quickly coded with very little effort.

Based on my experiments, at current price levels it would cost approximately $800 to generate 100,000 personalized mass phishing emails. On average, stolen account credentials sell for around $15 per account on the Dark Web, even with a modest success rate of 2% in the generic phishing scenario, the haul from a phishing campaign with 100K targets jumps from $30,000 in the generic case to $60,000 when using AI-generated custom messages, well worth the $800 cost. You might hope that companies will take steps to prevent AI from such misuse, but looking at how much the online advertising industry does to stop malvertising I’m not overly optimistic that profit won’t come first in this case too.

With far more effective phishing campaigns on the horizon, it is important to focus on prevention to avoid falling for these scams, so here are a few tips to keep your business safe:

1. Don’t select an email provider purely on price, use a reputable vendor with strong anti-malware and phishing protection.
2. Use multi-factor authentication for access to accounts and data.
3. Monitor for suspicious activity on company networks using a Managed Detection and Response service like My Security Console.
4. Educate employees on phishing scams and how to spot them, provide regular training and reminders to employees about the importance of security.
5. Institute security policies that require employees to use strong passwords, MFA, and regularly take cyber awareness training.
6. Use a multi-layered approach to security that includes email filtering, intrusion prevention, and anti-phishing tools.