Malicious LLMs Are Here: Defending Against AI-Led Attacks

Most companies still treat “AI threats” like a future problem. Meanwhile, malicious large language models (LLMs) are already being sold like SaaS—complete with pricing tiers, Telegram support channels, and “features” tailored to phishing, ransomware, and data theft.

That’s the uncomfortable truth behind the dual-use dilemma: the same capabilities that make LLMs great for security operations—fast summarization, fluent writing, code generation—also make them great for attackers. And in late 2025, purpose-built malicious LLMs such as WormGPT 4 and KawaiiGPT show how far this has moved beyond theoretical risk.

This post is part of our AI in Cybersecurity series, where we focus on practical, defensive uses of AI: anomaly detection, fraud prevention, and faster incident response. Here, we’ll flip the lens: what changes when attackers have “AI copilots,” and what should defenders do differently starting next quarter—not next year.

Malicious LLMs change the economics of cybercrime

The core shift is simple: attackers can now buy speed, language quality, and code scaffolding on demand. That lowers the barrier to entry and increases the ceiling on scale.

Traditional security training taught people to spot “bad English” and suspicious formatting. That heuristic is dying. Malicious LLMs produce emails that read like a real vendor escalation, a CFO request, or a HR policy change—without the telltale awkward phrasing that gave older phishing away.

Scale beats skill when AI handles the “hard parts”

Malicious LLMs push cybercrime toward scale over skill:

• Low-skill attackers can run campaigns that look like they were written by experienced operators.
• High-skill attackers can run more experiments per day: more variants, more targeting angles, more social engineering styles.
• Criminal groups can standardize “playbooks” and hand them to affiliates with minimal training.

Just as importantly, they compress time. Steps that used to take hours—researching a target, drafting a believable lure, writing basic tooling—can be reduced to minutes of prompting.

Why this matters to security leaders (not just SOC analysts)

If you own risk, budget, or incident readiness, here’s the practical impact:

• BEC and supplier fraud risk rises because messages sound credible and context-aware.
• Ransomware readiness matters more because initial access can be obtained faster.
• Security controls need to prioritize behavior (anomaly detection, identity signals, device posture) over superficial content cues.

The “AI vs AI” era isn’t hype. It’s a forced upgrade of your detection strategy.

WormGPT 4 and KawaiiGPT show what “purpose-built” looks like

A malicious LLM isn’t just a jailbroken mainstream chatbot. It’s a model or workflow intentionally optimized for offensive outcomes. That means removed guardrails, attacker-friendly UX, and packaging that fits the cybercrime economy.

Two patterns stand out from recent threat research:

WormGPT 4: cybercrime-as-a-service, priced and packaged

WormGPT’s brand started in 2023, but what’s more relevant in 2025 is the commercialization pattern behind newer variants (often referred to as WormGPT 4 in underground channels).

When a malicious tool offers:

• a polished interface,
• subscription tiers (including lifetime access),
• an upsell to “full source code,”
• and an active community channel,

…you’re not dealing with a one-off experiment. You’re dealing with an ecosystem.

And that ecosystem is designed to help attackers do two things well:

1. Write persuasive social engineering content (phishing and BEC)
2. Generate functional malware scaffolding (including ransomware building blocks)

Threat researchers demonstrated that, when asked, the tool produced a working PowerShell-based encryption script and a psychologically manipulative ransom note. The key detail isn’t novelty—it’s availability. Attack capability is being productized.

KawaiiGPT: “free” is a force multiplier

WormGPT’s model is paid access. KawaiiGPT’s differentiator is simpler: it’s designed to be easy to set up and cheap (or free) to obtain, including public repository distribution and a lightweight CLI experience.

That matters because free tools expand the funnel:

• more experimentation by would-be attackers,
• more copycats,
• more “AI-augmented” opportunistic crime.

Researchers observed KawaiiGPT generating spear-phishing lures, lateral movement scripting concepts (e.g., SSH automation patterns), and basic data exfiltration scripts using common libraries. Again, none of this is technically exotic. The risk is that it’s fast, repeatable, and usable by more people.

What defenders should change: detect behavior, not just content

If your anti-phishing approach depends on spotting broken grammar, you’re already behind. The defensive posture that holds up in 2026 is the one that assumes messages will look professional and “normal.”

Here’s what I’ve found works in real programs: re-center on identity, workflows, and anomalies.

1) Treat BEC as a workflow attack (because it is)

LLM-assisted BEC isn’t just an email problem. It’s a business process problem.

Upgrade controls around high-risk actions:

• Vendor bank account changes
• Wire approvals
• Invoice routing and exceptions
• Executive “urgent requests” outside normal chains

Practical guardrails that reduce successful BEC even when the email looks perfect:

• Out-of-band verification for payment changes (known number, known portal)
• Dual approval for new beneficiaries and threshold exceptions
• Hold-and-review windows for first-time payments
• Vendor master data monitoring (who changed what, when, from where)

LLMs improve persuasion. They don’t magically bypass solid process controls.

2) Use AI for anomaly detection where humans can’t keep up

Attackers use AI to generate volume. Defenders need AI to spot patterns across volume.

Good AI-driven threat detection focuses on:

• Identity anomalies: impossible travel, token reuse, unusual OAuth consent grants
• Behavior anomalies: rare admin actions, unusual mailbox rules, atypical file access
• Communication graph shifts: new sender-recipient pairs, unusual frequency, sudden urgency language plus abnormal transaction context
• Endpoint anomalies: scripting spikes (PowerShell), new scheduled tasks, suspicious parent-child process chains

The win isn’t “AI that reads emails better.” The win is AI that correlates signals across identity, endpoint, and network to flag what’s truly out of pattern.

3) Assume “good” malware code will be generated—and plan for that

Malicious LLMs can generate functional code quickly. That doesn’t mean the generated code is stealthy, but it does mean defenders should expect:

• more scripting-based intrusions,
• more rapid iteration on tooling,
• more “living off the land” behavior wrapped in better automation.

So prioritize controls that punish execution and persistence, not just known signatures:

• Constrain and monitor PowerShell (Constrained Language Mode where feasible)
• Enforce least privilege and remove local admin where possible
• Harden credential theft paths (LSASS protections, phishing-resistant MFA)
• Tighten egress controls and DNS monitoring (exfil paths still need a route out)

If an attacker can generate ten variations of a script in ten minutes, signature-only approaches lose.

Guardrails aren’t optional: build safe AI while you defend against unsafe AI

Here’s the twist in the dual-use dilemma: many organizations are simultaneously:

• deploying LLMs in customer support and internal productivity tools, and
• facing LLM-enabled threats from outside.

You can’t claim “responsible AI” internally while running sloppy controls around prompts, data access, and model behavior. Attackers love inconsistency.

Minimum viable AI governance for security teams

A practical baseline that security leaders can implement without a year-long committee cycle:

1. Inventory where LLMs are used (apps, plugins, copilots, internal tools)
2. Define prohibited data (credentials, secrets, regulated data) and enforce it technically
3. Log and monitor LLM interactions for abuse patterns (especially admin users)
4. Red-team your AI workflows (prompt injection attempts, data exfil paths)
5. Vendor security reviews that include model behavior, not just SOC 2 paperwork

Defensive AI is only helpful if it’s trustworthy, auditable, and constrained.

“People also ask”: practical questions security teams are asking now

Are malicious LLMs mostly used by advanced attackers?

No. The dangerous part is the opposite: they enable less-skilled attackers to produce higher-quality scams and basic tooling. Skilled groups also benefit, but democratization is the bigger scaling risk.

Do email security tools still work if phishing text is perfect?

Yes—if they rely on more than text. The most resilient stacks combine:

• sender authenticity (DMARC alignment and enforcement),
• URL and attachment detonation,
• identity and session signals,
• and anomaly detection across user behavior.

Should we ban LLMs internally to reduce risk?

A blanket ban usually fails in practice and pushes usage into shadow IT. A better stance is controlled enablement: approved tools, logging, access controls, and clear rules about sensitive data.

What to do in the next 30 days

Most organizations don’t need a brand-new security program to respond to malicious LLMs. They need sharper priorities.

If you’re setting a 30-day plan, I’d start here:

1. Run a BEC tabletop that assumes the email is perfectly written and context-aware.
2. Audit payment-change workflows and add out-of-band verification where it’s missing.
3. Tune detections for scripting and mailbox abuse (PowerShell spikes, inbox rules, OAuth grants).
4. Add an “AI abuse” review to your incident response checklist (prompt injection, data leakage, compromised AI credentials).

That’s the practical path toward “AI vs AI” defense: not panic, not hype—just controls that hold up when attackers get smarter and faster.

The next question is the one that decides whether you’ll be reacting or ready: if a criminal can generate a full attack chain in minutes, which parts of your environment still force them to slow down?