Stop Trusting PoCs: Fight AI Slop in AppSec

A CVSS 10.0 vulnerability will do one thing every time: it will flood your feeds with “proof-of-concept” exploit code. The difference in late 2025 is volume and speed—because generative AI can crank out exploit-looking artifacts faster than defenders can validate them.

The React2Shell episode is a clean example of how AI slop turns vulnerability response into a fog machine. Public PoCs showed up by the dozens, many were trivial, some were flat-out nonworking, and a few were “fake proof” that nudged teams toward the wrong conclusion: “We tested it and got a negative result, so we’re fine.” That’s a dangerous sentence.

This post is part of our AI in Cybersecurity series, where we look at what AI does well (pattern recognition, anomaly detection, automation) and where it can mislead (fabricated evidence, brittle assumptions, false confidence). If your AppSec or SecOps program depends on PoCs to prioritize patching, you need a tighter system—one that treats PoCs as untrusted input and still moves remediation fast.

“Fake proof” is a defender tax—and it’s getting worse

Fake or flawed PoCs don’t just waste time; they actively distort risk. The defender tax shows up in three ways:

1. False negatives: Teams run a scanner derived from a bad PoC, see “not vulnerable,” and stop.
2. Mis-triage: A nonworking PoC makes a critical issue feel theoretical, so it gets pushed down the queue.
3. Operational drag: Security engineers spend hours validating internet artifacts instead of patching, compensating controls, and monitoring real exploitation.

React2Shell illustrated the pattern. Researchers reported a surge in exploitation attempts, including attempts capable of bypassing some WAF rules, alongside lots of junk generated to look legitimate. One particularly nasty failure mode: a PoC that only worked if nondefault components were installed. A developer could block those components and believe the underlying deserialization flaw no longer mattered.

Here’s the stance I’ll defend: PoCs are not evidence of safety. They’re evidence of someone’s ability (or inability) to reproduce a condition in a specific environment.

Why AI slop spreads so easily

AI didn’t invent bad PoCs. It just made them cheap.

Generative models are good at producing code that compiles—or almost compiles—while sounding confident in comments and readme files. That’s enough to trick:

• Aggregators that collect “references” automatically
• Overworked defenders looking for a quick validation path
• Less experienced researchers trying to be first to publish

When “publish fast” meets “generate fast,” you get a polluted ecosystem where the appearance of exploitability travels faster than verified exploitability.

What React2Shell teaches about AI in cybersecurity

AI is both the shield and the sword in modern vulnerability response. Attackers use it to iterate, obfuscate, and scale. Defenders use it to triage, detect, and automate. The React2Shell saga shows what happens when defenders treat AI outputs—or AI-adjacent artifacts—as trusted.

A key warning from the vulnerability’s original researcher was blunt: if an exploit requires developers to have explicitly exposed dangerous functionality to the client, it’s not a valid PoC for the core flaw. That’s not academic nitpicking; it’s about avoiding the worst outcome—organizations concluding they’re safe because they tested the wrong thing.

The bigger risk isn’t “bad PoCs”—it’s patch delay

The most practical takeaway is uncomfortable: the gap between detection and patching is where breaches happen.

If your organization detects thousands of vulnerabilities a month but only fixes a few dozen, the system is already strained. In that environment, bad PoCs aren’t just noise—they become a rationalization tool:

• “No working exploit, so we can wait.”
• “Our scan didn’t trigger it, so it’s probably not relevant.”

Meanwhile, motivated attackers don’t stop at the first broken PoC. They iterate past it.

Treat PoCs like malware: a validation workflow that actually works

A PoC should enter your program the same way suspicious code does: sandboxed, labeled, and verified. If you’re not set up to validate quickly, you’re effectively outsourcing triage to whoever posts first.

Step 1: Create a PoC trust score (and default it to “low”)

Start with a simple rubric your team can apply in minutes:

• Source reputation: known researcher/vendor vs. anonymous repost
• Exploit prerequisites: default config vs. special components/edge cases
• Reproducibility notes: clear environment description, versions, expected behavior
• Artifact quality: tests, logs, or packet captures vs. vague “works for me”
• Independently confirmed: second party validation (internal or trusted peer)

The rule: low trust means it can’t downgrade severity. At most, it informs detection engineering.

Step 2: Separate “vulnerability validation” from “exploit validation”

Teams often use PoCs for one of two jobs:

• Are we vulnerable?
• Is it exploitable in our environment right now?

Those are not the same.

A broken PoC can’t prove you’re not vulnerable. If you need a fast answer, prioritize:

• Version and dependency checks (SBOM and package inventory)
• Configuration analysis (are dangerous code paths reachable?)
• Compensating controls review (WAF rules, request validation, serialization guards)

Use PoCs to improve monitoring and confirm exploitability, not to decide whether patching matters.

Step 3: Instrument for “attempts,” not just “success”

One reason AI slop hurts is that it creates alert fatigue and makes teams distrust signals. Flip the model:

• Track exploit attempt patterns (payload shapes, endpoints, headers)
• Correlate with runtime indicators (exceptions, deserialization errors, unusual stack traces)
• Measure blocked vs. unblocked attempts by control (WAF, API gateway, app-level validation)

This is where AI in cybersecurity actually shines: anomaly detection across high-volume telemetry. But it only works if you feed it clean, well-labeled signals.

How better AI tools reduce “fake proof” damage

The goal isn’t to debate PoC quality; it’s to reduce the time between disclosure and remediation. That’s where mature AI-assisted security programs outperform.

AI for triage: summarize, cluster, and route—don’t decide

Use AI to:

• Cluster related exploit chatter and PoC variants
• Extract prerequisites and affected components from advisories and internal code
• Draft internal tickets with required patch steps and owners

Don’t use AI to:

• Declare “not vulnerable” based on a single PoC run
• Auto-close findings because an exploit didn’t trigger

A useful internal policy is: AI can recommend; humans approve; automation executes within guardrails.

AI for secure automation: patch faster than attackers iterate

If your patch process takes weeks, attackers have time to operationalize.

Practical improvements that pair well with AI-assisted workflows:

• Automated dependency upgrades with tests and staged rollouts
• Risk-based patch SLAs tied to asset criticality and exposure
• Pre-approved change windows for emergency library fixes
• Canary deployments to detect regressions quickly

The most effective teams I’ve worked with treat critical library patches like incident response: tight scope, clear owner, rapid verification, fast rollback.

AI for fraud and anomaly analysis: verify what humans can’t eyeball

React2Shell is an AppSec story, but the pattern generalizes to fraud and SOC work: synthetic artifacts create synthetic confidence.

Modern AI systems can help detect:

• Unusual request sequences that mimic exploit chains
• Credential stuffing patterns hidden in noisy traffic
• Token abuse and session anomalies across distributed apps

When tuned well, AI-based threat detection becomes your counterweight to AI-generated noise. The trick is governance: labeling, evaluation, and feedback loops.

A practical playbook for the next “PoC flood”

Assume the next high-profile vulnerability will come with 100+ PoCs, and most won’t help you. Build a repeatable response.

1) Decide patch priority without PoCs

Use a simple decision set:

• Is the component internet-facing or in a critical path?
• Is the vulnerable function reachable in your architecture?
• Are there active exploitation reports (even if PoCs are messy)?
• Do you have reliable compensating controls—and are they tested?

If the answer trends “yes,” patch as if exploitation is inevitable.

2) Build a “known-good” internal reproduction harness

Create a small, maintained lab that mirrors your stack:

• Representative versions and deployment patterns
• Logging that captures the right runtime signals
• Safe test data and reset scripts

This reduces dependence on internet PoCs and gives you a stable environment to validate quickly.

3) Publish internal guidance within 24 hours

When developers are anxious, they’ll grab whatever PoC they can find.

Give them something better:

• A short internal advisory: impacted versions, symptoms, patch steps
• What not to trust (random PoCs, single-scan negatives)
• How to report suspicious traffic or reproduction attempts

Speed matters. Clarity matters more.

Where this is headed in 2026 (and what to do now)

AI slop isn’t a temporary annoyance. It’s a structural change to how security information spreads. The winners won’t be the teams who find the “perfect PoC.” They’ll be the teams who treat external artifacts as untrusted, instrument their systems for real signals, and patch with urgency.

If you’re investing in AI in cybersecurity, aim it at the hard constraint: remediation throughput. Threat detection is valuable, but detection without action is just better reporting.

If you want a straightforward next step, audit your last critical vulnerability response:

• How long from disclosure to patch in production?
• How many engineer-hours went into PoC validation?
• Did any “negative tests” reduce urgency?

Answer those honestly, then decide: are you prepared for the next PoC flood—when the junk looks even more convincing?