How GenAI enhances existing attack techniques
Most successful attacks today are based on a combination of social engineering and malware delivery. Attackers typically start by manipulating human behavior to gain a foothold, for example through deception or phishing. Once access is established, they use malicious software to infiltrate systems, steal data, or disrupt operations. This dual approach targets both people and technology, making modern attacks more sophisticated and harder to defend against.
Social engineering
Social engineering has existed since the beginning of human thought and serves as the basis for a wide variety of scams. In the field of information security, social engineering is defined as a tactic to trick people into disclosing secret or personal information (Suker, 2021). In practice, social engineering exploits human characteristics such as helpfulness, trust, fear, and respect. These characteristics are specifically instrumentalized in order to select victims to help an attacker reach their goals (BSI). The advancing integration of artificial intelligence (AI) into cyberattacks marks a decisive moment in the evolution of social engineering.
Today, phishing is the most common method to infiltrate organizations and systems. According to reports, “It continues to be a top action variety in breaches and is the most common form of social engineering.”, (Verizon, 2025 DBIR).
Phishing is the act of tricking individuals into disclosing information or downloading malware, traditionally executed through email, SMS or phone calls. First the attacker collects information about their target like technology they use and social information like the person’s weaknesses and their role in the target organization. GenAI accelerates the speed in which the attacker can conduct both technical and social reconnaissance of targets.
The attacker then crafts the phishing mail to get their targets to click on a malicious link or open a harmful attachment. Previously, phishing attempts were most frequently identified by poor grammar and links to suspicious websites. However, GenAI enables individuals to compose grammatically accurate emails with a variety of templates addressed to company executives. This development significantly lowers the barriers for creating convincing phishing emails. (Salahdine & Kaabouch, 2019; Suker, 2021)
Attackers can also train bots using GenAI to do their work for them. These bots will carry out reconnaissance tasks and send out the phishing mail, targeting a much wider range of targets. We are reaching a point at which it will be significantly harder to discern real people arguing on the internet from bots that can carry on whole conversations without breaking character. This concern is no longer hypothetical: in a recent experiment, researchers used AI-powered bots on Reddit to engage in persuasive, human-like conversations with real users – without their knowledge – demonstrating how easily GenAI can be weaponized for manipulation at scale (The Atlantic, 2025).
Phishing attempts are now being enhanced by deepfakes, or highly realistic audio or video imitations, that leverage machine learning to exploit human vulnerabilities more convincingly and precisely.
As outlined in recent academic literature, deepfake-enhanced phishing enables attackers to combine visual and auditory deception with traditional email-based social engineering tactics (Machetanz et al., 2024). A striking example involved a scammer who used an AI-generated voice clone to impersonate a family member in distress, leading to emotional manipulation over the phone (The Independent, 2023). It is already possible to produce media so realistic that it is almost indistinguishable from authentic recordings. Europol’s 2025 Electronically Organised Crime Threat Assessment (IOCTA 2025) highlights that emerging technologies such as AI-powered voice cloning and live video deepfakes amplify the threat of fraud, extortion and identity theft, enabling criminals to scale up social engineering and impersonation-based attacks (Europol, 2025).
This makes it considerably more difficult to detect and prevent targeted disinformation campaigns, manipulative social engineering attacks, and the spread of false information in a timely manner (Martinek & Bartuzi-Trokielewicz, 2025, Khanjani et al., 2023).
Malware and malicious software development
Some would argue that GenAI lowers the threshold of knowledge needed to enter the world of malware programming, but this is largely exaggerated. The threshold was already low given the availability of free trainings, forums, and other resources; GenAI is unlikely to replace these resources as a teaching method. But insofar as GenAI improves programming writ large, it is also true for GenAI in malware. GenAI helps the bad and mediocre be better and faster than they would be alone. It functions as a significantly more advanced platform akin to GitHub or Stack Overflow. A phenomenon called “vibe hacking” is on the rise. While “vibe coders” use GenAI as a helpful tool to generate fully functioning applications,”vibe hackers” similarly describe their nefarious goals and intentions to an AI, which then realizes and generates the attack (Wired, 2024, Anthropic, 2025).
Since the release of GenAI, various countermeasures have been implemented to prevent potentially harmful or malicious content from being generated. Unfortunately, these can still be evaded by attackers who convince the GenAI that their malicious requests are for research or fictional reasons (Bhaimiya, 2023). Recent reporting has also shown that attackers use ‘jailbreaking’ techniques to bypass safeguards in GenAI tools and generate malicious code (MIT, 2024).
This is further complicated by the fact that the use of GenAI does not necessarily indicate malicious intent. In fact, Google Cloud Threat Intelligence has been reporting growing concerns over the “dual-use” nature of GenAI for some time now, which can be employed both for productivity and for malicious manipulation (Google Cloud Threat Intelligence, 2023).
While an attacker may need to convince GenAI to help build malware, in some cases it is not even necessary. There are good reasons why someone might want to prevent their antivirus program from activating if they perform particular tasks. Users often face situations where legitimate tools trigger security measures like antivirus systems or firewalls. In practice, this can lead to users creating security workarounds: for example, when an engineer needs to establish a connection that violates internal network rules. A basic prompt to a GenAI system might be: “How do I configure my application so the firewall error doesn’t happen?” This shows how GenAI can be used to bypass or suppress security measures, even if unintentionally.
Generally speaking, GenAI will not create super hackers out of low-level criminals, but mediocre hackers will have much more knowledge and capabilities at their disposal. GenAI does not only empower and enable the existing classic attacks but also opens a dimension of new attacks. These new types of attacks will be discussed in the following section.
How AI enables new attack methods
Anthropic reported that in mid-September 2025 it detected and helped disrupt what it describes as the first large-scale cyber espionage campaign orchestrated predominantly by an AI system, where an agentic AI model autonomously carried out most stages of the intrusion against around 30 global targets. The incident highlights how advanced AI can automate complex tasks such as reconnaissance, exploit development, and data exfiltration with minimal human input, underscoring a new class of AI-empowered offensive threat that challenges traditional cybersecurity defenses (Anthropic, 2025).
Rather than relying solely on static signatures or heuristic patterns, GenAI can generate polymorphic code that slightly mutates with each deployment, effectively bypassing traditional detection mechanisms. Furthermore, threat actors can leverage GenAI to dynamically adapt malware behavior based on the specific environment it encounters, such as delaying execution until sandbox detection has passed or mimicking legitimate system processes.
The idea that GenAI would eventually allow self-programming malware that can adapt itself to undermine defenses and take advantage of weaknesses in target systems without external communication is currently exaggerated. Although initial research has explored the use of this adaptable malware, such studies have thus far been limited to controlled development environments. Models that are associated with such misuse include, for example, FraudGPT and WormGPT, which are specifically designed to facilitate criminal activities such as phishing, malware creation, and social engineering attacks. These models automate and simplify the execution of sophisticated cybercrimes, making them accessible even to users without technical expertise (Falade, 2023, Korolov, 2024 Zhu et al., 2025).
Enhanced autonomy means that the malware can reduce communication with its command-and-control server because it can independently make decisions, without outside instructions from the attacker. For example, autonomous malware could judge which files are important and therefore should be prioritized for extraction. This could be done with the integration of a GenAI that categorizes documents in the malware and then prioritizes certain categories for encryption/extraction. By limiting external communication, the malware is also more likely to go undetected.
Google’s Threat Intelligence Group reported emerging malware families (e.g., PROMPTFLUX, PROMPTSTEAL, PROMPTLOCK) that use AI at runtime to generate malicious scripts and dynamically alter code to evade detection (Google Cloud, 2025). It could even be possible for the malware’s GenAI to communicate with a GenAI already present in the target systems for this analysis (like Copilot) to both avoid detection and utilize the access and knowledge of the target’s own GenAI system.
There is still a significant gap between using GenAI to exploit known vulnerabilities and actively creating systems that can adapt dynamically to target system parameters, evade detection, and generate exploit chains for further compromise. It may be some time before such capabilities are observed in real-world scenarios, and even then, their use will likely be limited to highly sophisticated threat actors with precise targeting.
Defending against GenAI-supported cyber attacks
To counter Gen AIsupported cyber attacks, defenders can deploy deception tactics such as honeypots (e.g., fake outdated sever, fake financial data). Using honeypots helps mimic vulnerabilities and should be able to catch out some of these new attack paths by attracting malware variants and adversarial inputs, enabling early detection. Integrating honeypots with frameworks like MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) enables organizations to proactively simulate and analyze attacker behavior specific to AI-enabled threats (MITRE ATLAS, 2025). The Secure AI with Threat-Informed Defense initiative (2025) expands the ATLAS knowledge base and emphasizes AI red-teaming, adversary emulation, and structured threat characterization for evolving malicious use of AI and effectively represents the defensive side of modeling AI-enabled adversarial threats.
The best remedy to counter phishing and similar tactics is a mix of awareness, processes and technical measures that block or warn against suspicious user activity. Even if the contents of awareness trainings change over time, these security foundational practices remain relevant. User awareness, enhanced by automated anomaly detection, remains key to defending against GenAI-powered phishing attacks, especially when traditional red flags like poor grammar are no longer reliable (ENISA, 2025).
The programming of malware and other software needed for malicious purposes with the help of GenAI can only be prevented to a certain extent by the relevant providers, who can block users from using their services to create obviously malicious software. But for dual-use software this is much harder, and criminals can bypass the restrictions by avoiding mainstream GenAI systems and instead utilizing open-source or even self-hosted models with fewer guardrails.
Adapting to GenAI-driven threats
New, innovative technologies will always be exploited by attackers. GenAI is no exception. It is the job of security experts to take note and speed up the advancement of their defenses to meet the new status quo.
Just as GenAI strengthens the capabilities of attackers, it also strengthens those of security researchers. With the use of GenAI, security researchers can analyze tons of security data and logs far more efficiently than before.
