Why 'Potential Vulnerability' Is a Worthless Output

Your scanner is producing rumors, not evidence

When a static analysis tool tells you 'potential SQL injection near line 42', it's giving you a hypothesis. Someone still has to test whether the hypothesis is true. In most teams, that someone is a developer who looks at the pattern, decides it 'looks fine', and marks it resolved. Sometimes they're right. Sometimes they're not.

'Potential' is doing a lot of work in that sentence. A potential vulnerability is not a vulnerability. It's a location to investigate. When you have 14 of them per PR, you're not doing security. You're doing triage theater. The findings that get fixed are the ones that survive the triage. The findings that look ambiguous get closed.

The problem compounds with AI-generated code. Agents write code that looks syntactically clean and follows common patterns, which means static analysis fires on it for the same reasons it fires on safe code: the pattern matches, not because the code is actually vulnerable.

Verified is a different word from suspected

Verified means: an automated exploit agent sent a SQL payload to your staging environment, observed a 200 OK response with 847 rows of user data including admin accounts, recorded the full HTTP exchange, and is handing you a curl command to replay it yourself. That is evidence. That is something you can act on, audit, and show to a customer.

The difference matters especially for agent workflows. If you give Claude Code a 'potential SQL injection' it might add input validation, which addresses the symptom but might not close the actual injection path. If you give it a verified finding with the exact payload and parameter, it writes a parameterized query because it knows exactly what the attack vector is.

Same code, two very different fix instructions, two very different outcomes.

Why this matters more when agents are writing code, not less

The instinct is: agents know security patterns, so code quality should be higher. That's true for obvious cases. The issue is subtle cases: logic bugs in auth flows, insecure defaults in third-party integrations, rate limiting gaps, overly permissive CORS configs. Agents pattern-match from training data. Your specific configuration, your specific middleware chain, your specific database driver version. None of that is in their training data.

Static analysis has the same problem. Two tools, same failure mode: neither of them knows that your particular framework middleware doesn't sanitize the parameter before it reaches the query builder. The only tool that knows is the one that sent the payload and watched what came back.

This is why runtime verification isn't a nice-to-have add-on to static analysis. It's the thing that produces ground truth about your actual running system, not a theoretical analysis of your source code.

What changes day-to-day when you operate this way

Reviewers stop ignoring security alerts when every alert is proven. Developers spend less time on triage because the triage is automated. Agents write better fixes because they have the exact context they need. Security and engineering align faster because the evidence is reproducible by anyone with a curl command.

The audit trail matters too. When a customer asks how you know their data is safe, the answer is 'we ran the exploit against a staging environment and it didn't land, here's the HTTP exchange.' Not 'we ran a scanner and it came back with a 72% confidence score.'

That's a different conversation. It's also a more honest one.