The use of AI in phishing emails

    Sep. 30, 2024

    Key Points

    What you'll learn in this article

    • AI tools enable threat actors to generate well-constructed and contextually accurate emails.
    • Overall, generative AI emails have a polished and professional tone, making them more convincing.
    • Analyst investigations should increasingly look for words and phrases associated with generative AI models and not just the sender information and payloads.

    AI Generated Messages

    Threat actors are increasingly leveraging generative AI technologies to craft phishing emails. These AI-generated attacks are harder to identify due to improved language fluency and tone mimicry. When interviewing Mimecast threat researchers for our recent Threat Intelligence report, questions were asked about the pervasiveness of AI in phishing emails - but no metrics could be quantified. Our data science team took on the challenge to help, by building a detection engine to determine if a message is human- or AI-generated based on a mixture of current and historical emails, and synthetic AI-generated emails.

    The research indicates a point in time when Mimecast starts observing an increasing trend in AI-generated emails correlating with the release of ChatGPT. In addition, we also observed malicious AI-generated BEC, fraud and phishing emails. Analysts are advised to stay vigilant, as these attacks are expected to increase in both volume and complexity.

    The full details of our findings can explored further within this blog post.

    Telltale signs of AI-generated emails

    One of the most notable characteristics of AI language models is the use of complex words and sentence structures. Researchers found AI language models favor certain words in scientific writing. “Analyzing 14 million papers from 2010-2024, they noticed a sharp increase in specific ‘style words’ after late 2022, when AI tools became widely available. For example, ‘delves’ appeared 25 times more often in 2024 than before.

    Mimecast’s data science team started with the intention to train a model the differences between human- and AI written emails. In total over 20,000 emails were utilized from Mimecast’s data coupled with LLM generated synthetic data. We then sampled 1000 emails per month from January 2022 to June 2024 to identify how many were AI written. These statistics show that out of 30,000 emails analyzed it was found that 2,330 were AI-written representing 7.8% of all emails in the dataset. The results from this exercise can be found in figure 1 which also highlights the increase of AI-written emails.

    Targets

    Global

    Examples of AI-Generated Emails

    During the analysis process several malicious examples were found containing distinctive language indicative of AI tools.

    Example 1 - AI generated spam message

    Indicators:

    • "delves into the intricacies of", "navigating through the complexities of"
    • Overuse of bullets

    Example 2 - AI generated BEC message

    Indicators:

    • ‘I hope this message finds you well'
    • Repetition of the words ‘gift cards’ and ‘surprise’

    Example 3 - AI generated BEC message

    Indicators:

    • ‘Hello!’

    Example 4 - AI generated phishing message

    Indicators:

    • ‘delve deeper into this’
    • ‘stumbled’ or ‘stumbled upon’
    • Long ‘-’ utilized across ChatGPT

    Recommendations

    These findings indicate that manual phishing investigations remain a crucial layer of defense, especially when flagged by end users. It's vital that threat researchers scrutinize the language for specific markers that align with our findings; by cross-referencing indicators such as “delve deeper into this” or “hello!”, particularly among users who commonly don’t use such language. With known threat patterns, you can identify phishing threats more effectively, reducing remediation time and mitigating organizational risk.

    It should be noted that security teams should ensure their indicators utilized within their investigations, evolve alongside large language models which are continually changing.

    Zurück zum Anfang