Risk Management
Attack versus defence: how AI models can help both sides – Pt 2
In the previous blog of this two-part mini series, we explained how categorising and generative AI can help network defenders.
Save to Folio
But unfortunately, what works for IT security teams can also be deployed by attackers with more nefarious goals in mind. And threat actors are nothing if not resourceful. In fact, a major data leak that impacted the notorious Conti ransomware group last year revealed it spends $6m annually on tools, services and salaries, with much of that going on R&D.
In this fast-emerging cat-and-mouse game, only by studying malicious use of AI can the cybersecurity industry better design its own products and mitigations.
Using categorising AI for malicious ends
For IT security teams, categorising AI models form the spearhead of cyber defence. However, for threat actors they are mainly used for strategic victim selection and attack planning. The selection of victims is not dissimilar from business marketing processes designed to find the optimal customer target group. It starts by defining the optimal customer persona(s). Then, using the available data on a group of customers, you try to match them to the personas.
If they’re not automating indiscriminate attacks to a wide group of users—which in marketing would be like direct mailing every prospect—cyber-criminals may also want to define "optimal victims". These may be individuals who are more willing to pay, less likely to file a police report, and who require less effort to attack. The next stage is to try and find them using available data sources such as Shodan searches, social network-based OSINT, data from previous attacks, and leaked or stolen information. Since the selection of victims is neither time- nor resource-critical, large amounts of data and complex data sets can be processed by categorising AI to find potential victims.
AI could also theoretically be used during an attack, to determine which post-compromise activity—such as encryption, blackmail or data theft—promises the greatest profit. A locally executed categorising AI model could make this decision. Although the effort involved is currently disproportionate to the benefit, that might not always be the case.
The limits of generative AI
As we explained in the last post, generative AI isn’t “creative” in that everything it produces is a sometimes-sophisticated remix of content it has been trained on. That means it can display readymade exploits for users, including explanations and comments. But it won’t be able to generate zero-day exploits out of thin air. It can assist in attacks in other ways, such when a generative AI model was used to write code for plugins deployed to help win a Pwn2Own competition. ChatGPT has also been seen helping to create polymorphic malware variants, although again only rewrites containing known content learned in training. While a seasoned cyber-criminal would find limited use with such a tool, it may help to democratise access to such knowledge among the less tecchie end of the criminal community, as long as they know what to ask it for.
It is in content creation that generative AI models really come into their own. They’re capable of producing appealing and highly readable phishing content without the grammatical mistakes still so common to many malicious missives. Threat actors able to provide access to high quality training and incorporate feedback from A/B testing into that content will get even better results. The possible use of generative AI to produce highly convincing “writing style” content in business email compromise (BEC) and other impersonation attacks is a concern already highlighted by Europol.
It’s also a reason why companies like Trend Micro continue to research the AI space intensively. By understanding how our adversaries operate, we can develop mitigations to detect and block their efforts more effectively. The AI arms race has only just begun.