How ComposedAI's Jev-powered scanner performed on the OWASP Benchmark
We tested ComposedAI's Jev-powered scanner on the OWASP Benchmark. It caught 89.7% of vulnerable cases at a 9.1% false-positive rate, while using materially less model inference than a general-purpose frontier LLM.
Why ComposedAI uses Jev for security decisions.
General-purpose models are built to generate open-ended answers. Security review needs something narrower: consistent decisions about whether code is likely to contain a real vulnerability.
That is where Jev fits. Jev is a decision model rather than a chatbot: it is designed to evaluate supplied context and return focused, structured judgments. ComposedAI uses those judgments as one part of our scanner, alongside deterministic security checks.
The result is a scanner designed for continuous use without turning every pull request into a slow, expensive investigation.
The numbers, with the sample sizes attached.
On the OWASP Benchmark, ComposedAI caught 89.7% of the deliberately vulnerable cases while flagging 9.1% of the safe cases. That balance matters: high detection is not useful if developers learn to ignore the results.
| Evaluation | True positive | False positive | Precision | J |
|---|---|---|---|---|
| OWASP Benchmark, 330 cases, 11 categories | 89.7% | 9.1% | 90.8% | 0.806 |
| Go benchmark, 342 cases | 86.0% | 11.7% | 88.0% | 0.743 |
| Python sample, 476 cases | 90.0% | 13.5% | 86.3% | 0.766 |
Recall against false positives. ComposedAI is plotted from our OWASP run. SonarQube Community and FindSecBugs use published OWASP Benchmark 1.2 results; commercial tools appear only as the anonymized range published by OWASP.
We also ran ComposedAI against eight popular open-source libraries and observed zero false positives in that evaluation.
Efficient enough to run continuously.
Continuous security review only works if teams can afford to keep it running. Across the repositories in our evaluation, ComposedAI's model-inference cost ranged from $0.002 to $0.029 per scan, compared with $0.33 to $3.71 for a general-purpose frontier LLM.
Observed model-inference cost per repository in our evaluation. Lower is better. The comparison does not include downstream verification costs.
That efficiency lets ComposedAI evaluate changes continuously as software is built, instead of waiting for a periodic audit or a separate security review. Exact cost varies with repository size and configuration, so these figures are measurements from our evaluation rather than a universal price estimate.
Three things we learned.
1. Coverage matters more than a flattering score. Testing across every vulnerability category gave us a more useful picture than a smaller sample dominated by one class of bug.
2. Recall and false positives belong together. A scanner that finds more bugs by flagging everything creates noise, not confidence. Detection quality has to account for both.
3. Benchmarks are a baseline, not the finish line. Synthetic cases make tools easier to compare, but production repositories are messier. The result that matters is whether the scanner finds actionable issues in the code a team actually ships.
ComposedAI is built to make high-signal security review a continuous part of how software ships. The OWASP result is encouraging, but a synthetic benchmark is not production code and it is not the finish line. The next test is the one that matters to your team: how ComposedAI performs on the code you actually ship.
Run ComposedAI on your last 10 PRs.
Apply for design partner access and we’ll show which findings are real, exploitable, and worth fixing — free.
Run it on 10 PRs →