How to read these numbers
Precision
Of all the alerts the scanner raises, the share that are real vulnerabilities. Low precision = lots of false positives to sort through.
Full definition in the glossary ↗Recall
Of all the real vulnerabilities that exist, the share the scanner actually finds. Low recall = real bugs get missed — the more dangerous failure mode.
Full definition in the glossary ↗F1
A single number combining precision and recall. Convenient for comparison, but hides which of the two is driving the score — which is why this page reports both separately, not just F1.
Full definition in the glossary ↗Five methodologies, not one number
A single "accuracy" figure is close to meaningless without knowing how it was measured. These five produce very different — and increasingly honest — results:
Internal fixtures
Runs every rule against its own paired test cases (one vulnerable file, one clean file per rule) — 910 cases across all 8 languages. This measures whether the rule engine still does what it was built to do; a mature rule set should score close to 100% here almost by construction. Useful as a regression signal, not as external proof of anything.
OWASP Benchmark (Java + Python)
An independent, third-party OWASP test suite (not written by us) with a fixed, published ground truth, run in full for both languages. Harder and more realistic than self-written fixtures — this is the same methodology industry benchmarking studies use. Built almost entirely around Servlet/HTTP-style code patterns.
NIST Juliet (Java)
A US government (NIST SARD) synthetic test suite, 6,616 cases across 6 mapped CWE categories. Same language and largely the same vulnerability classes as OWASP Benchmark Java, but the source-code idioms are far more varied — environment variables, property files, sockets, stdin, URL connections, not just HTTP requests. The gap between this and OWASP Benchmark Java is itself the most honest signal on this page.
CVE Test Suite
Reconstructed from 45 real, historical CVEs across 8 real-world frameworks (Django, Flask, Express, Spring, ASP.NET, Laravel, plus both languages' standard libraries). The hardest test: real vulnerable code as it actually shipped, not an idealized test case.
Results, run today
| Methodology | Cases | Precision | Recall | F1 |
|---|---|---|---|---|
| Internal fixtures (all 8 languages) self-validation |
910 | 100.0% | 100.0% | 100.0% |
| OWASP Benchmark (Java) independent |
2,740 | 73.7% | 100.0% | 84.8% |
| NIST Juliet (Java) independent |
6,616 | 98.4% | 46.7% | 63.3% |
| OWASP Benchmark (Python) independent |
1,230 | 78.8% | 96.0% | 86.5% |
| CVE Test Suite (5 languages) real-world |
45 | 82.9% | 75.6% | 79.1% |
The most revealing comparison on this page is Java against itself: OWASP Benchmark Java shows 100.0% recall, while NIST Juliet — same language, largely the same CWE classes, run the same week — shows only 46.7%. That is not a contradiction, it is a diagnosis. OWASP Benchmark is built almost entirely around Servlet/HTTP-style source code; Juliet also tests environment variables, property files, sockets, stdin and URL connections. The rule engine had been tuned against exactly the pattern OWASP Benchmark rewards, and Juliet exposed the blind spot. We already fixed part of that gap this cycle (Java recall on Path Traversal and SQL Injection roughly doubled after broadening recognized taint sources — see below) and precision barely moved (98.2% → 98.4%). Recall on OWASP Benchmark Python (96.0%) and the CVE Test Suite (75.6%) sit in between, which is itself informative: neither is as narrowly-scoped as OWASP Java, nor as broad as Juliet. Identical numbers across all five would have been the signal something was wrong with the methodology, not the tool — a spread this wide, with a named root cause, is what an honest benchmark looks like.
OWASP Benchmark Java, broken down by category
The largest and hardest independent run — 2,740 cases, zero false negatives across every one of the 11 categories, but precision varies sharply by category. Worst first:
| Category | Precision | Recall |
|---|---|---|
| XPath injection | 42.9% | 100.0% |
| LDAP injection | 45.8% | 100.0% |
| Command injection | 50.2% | 100.0% |
| SQL injection | 55.4% | 100.0% |
| Trust boundary violation | 65.9% | 100.0% |
| Path traversal | 79.6% | 100.0% |
| Weak hash | 79.6% | 100.0% |
| Weak cryptography | 100.0% | 100.0% |
| Weak random | 100.0% | 100.0% |
| XSS | 100.0% | 100.0% |
| Insecure cookie | 100.0% | 100.0% |
The honest reading: recall is perfect everywhere (every planted vulnerability gets flagged, in every category), but four taint-heavy categories — XPath injection, LDAP injection, command injection, SQL injection — sit under 56% precision, meaning the rule engine over-flags in those specific patterns on this particular synthetic suite. Weak cryptography, weak random, XSS and insecure-cookie checks are exact (100%/100%). This is the real, current trade-off: tuned to never miss a planted vulnerability, at the cost of more false positives on a handful of injection categories than we'd like.
NIST Juliet Java, broken down by category
The largest run on this page — 6,616 cases across 6 CWE categories that Juliet 1.3 actually provides data for (3 of the 9 originally planned — XXE, Weak Random, Certificate Validation, Deserialization — have no equivalent in this NIST download). Worst recall first:
| Category | Precision | Recall |
|---|---|---|
| LDAP injection | 98.8% | 11.4% |
| Path traversal (absolute) | 100.0% | 33.0% |
| SQL injection | 98.4% | 41.9% |
| Path traversal (relative) | 97.5% | 64.5% |
| Weak cryptography | 100.0% | 89.5% |
| Command injection | 98.3% | 99.4% |
Precision stays high everywhere (97.5–100%) — the rule engine rarely cries wolf — but recall varies enormously, and the reason is architectural, not random. Two of Juliet's twelve source-code variants per category (data read via a database call, or passed through an abstract-method "flow variant") live in a companion class file the benchmark's single-file scanner never sees — those cases are unreachable regardless of rule quality, and pull every category's ceiling down. Within what is actually reachable, LDAP injection is the current weak point: Juliet's LDAP test cases lean on source patterns (JNDI context lookups fed by config/environment data) this rule set doesn't yet recognize as taint sources, unlike the path-traversal and SQL-injection rules that were just broadened. That is a scoped, disclosed gap, not a hidden one.
Internal fixtures, broken down by language (self-validation)
The 910-case internal run, per language — again: this measures internal consistency, not external difficulty, since every case is a fixture written for a rule that exists.
| Language | Vulnerable cases | Detected |
|---|---|---|
| Python | 144 | 144/144 |
| JavaScript/TypeScript | 84 | 84/84 |
| Java | 55 | 55/55 |
| HTML | 38 | 38/38 |
| YAML | 33 | 33/33 |
| PHP | 27 | 27/27 |
| C# | 25 | 25/25 |
| Other (Dockerfile + generic) | 10 | 10/10 |
CVE Test Suite, broken down by framework
The hardest and most realistic test — 45 cases reconstructed from real, published CVEs — broken down honestly, including where it misses:
| Framework | Language | Precision | Recall |
|---|---|---|---|
| Flask | Python | 100.0% | 100.0% |
| Django | Python | 100.0% | 100.0% |
| Spring | Java | 87.5% | 70.0% |
| Laravel | PHP | 100.0% | 66.7% |
| PHP stdlib | PHP | 80.0% | 66.7% |
| Python stdlib | Python | 100.0% | 66.7% |
| Express | JavaScript | 72.7% | 80.0% |
| ASP.NET | C# | 66.7% | 66.7% |
Weakest spot honestly disclosed: ASP.NET/C# and Express/JavaScript sit lowest on this table — both precision and recall in the 66–80% range. Historical CVEs like log4shell-style deserialization chains or complex multi-hop taint paths in these frameworks are the specific cases the rule engine still misses (the 11 false negatives include CVE-2021-44228 itself, tracked as an open gap, not hidden).
What this page deliberately does not include
- HTML, YAML and Dockerfile are not included in any table above. These aren't CWE-style injection vulnerability classes the way SQL injection or XSS are — YAML rules target CI/CD pipeline misconfiguration and Dockerfile rules are hygiene checks (root user, unpinned base images), architecturally different from a precision/recall benchmark built around planted vulnerabilities. See the CI/CD scanning guide for what is actually detected there.
- NIST Juliet C/C++ and .NET/C# test suites exist and could extend this page to more languages — not pulled in yet. Juliet Java's own 112 CWE categories are also only 6-mapped here (see above); expanding coverage to the other 103 (many with no equivalent rule at all, e.g. Integer Overflow, Race Condition) is future work, not a hidden result.
Why there is no competitor score on this page
We looked. Public OWASP Benchmark figures exist for some competitors, but sourced from a competing vendor's own marketing content, and a second source found for the same tool reports a meaningfully different number for the same claimed benchmark — with no disclosed methodology, test suite version, or tool version for either. That is not something we are willing to repeat as fact. If a competitor publishes a benchmark score with a transparent, reproducible methodology, we will cite it, exactly like the rest of this page cites methodology and dates. Until then, this page compares StaticCodeAudit against reality, not against unverifiable numbers about other tools.
Reproducing these numbers
These four methodologies are run through internal evaluation tooling, kept separate from the product's own source — NIST Juliet and OWASP Benchmark are third-party test suites, not code we wrote, so they don't ship with the tool. This is not something end users run day to day; it's the same tooling used to produce the figures above, run against the same rule engine.
run_benchmark.py owasp
run_benchmark.py owasp-python
run_benchmark.py cve
run_benchmark.py julietEach run exports a dated JSON file with the full per-category breakdown — the tables on this page are transcribed directly from those exports, run between 7 and 9 August 2026 (the two largest, OWASP Benchmark Java and NIST Juliet Java, took 3,551 and 4,611 seconds respectively — under an hour and a half each — across 2,740 and 6,616 cases).
Frequently asked questions
Why does the internal fixture score show 100% when the other two don't?
Because it is measuring a different thing. Internal fixtures are paired test cases written specifically for each rule — a rule that cannot detect its own designed test case would be a broken rule, not a subtle miss. 100% there means the rule engine has no outright regressions, not that detection is flawless in the wild. OWASP Benchmark and the CVE suite are independent of the rule-writing process, which is why their numbers are lower and more meaningful.
Is 66–80% precision/recall on some CVE Test Suite frameworks a problem?
It's a real, disclosed limitation, not something to spin. Complex multi-hop taint flows and framework-specific deserialization gadgets (the kind behind CVE-2021-44228) are genuinely hard for a rule-based static analyzer, commercial or open-source, to catch consistently. This page exists specifically so that limitation is visible rather than buried under a single inflated headline number.
Why is OWASP Benchmark Java precision only 73.7% when recall is a perfect 100%?
Because the rule engine is currently tuned to never miss a planted vulnerability, and that has a cost: four categories — XPath injection, LDAP injection, command injection, SQL injection — sit between 43% and 55% precision, meaning more false positives than we'd like on those specific taint patterns in this synthetic suite. The other seven categories, including SQL's siblings like XSS and weak cryptography, are exact. This is a genuine current limitation, disclosed here rather than smoothed over with an aggregate F1 score.
OWASP Benchmark Java shows 100% recall but NIST Juliet Java shows 46.7% — same language, similar CWEs. Which one is right?
Both, for what each one measures — and the gap between them is the finding. OWASP Benchmark is built almost entirely around Servlet/HTTP source patterns; Juliet also exercises environment variables, property files, sockets, stdin and URL connections as taint sources. The rule engine was tuned toward the pattern OWASP Benchmark rewards, and Juliet exposed that. We've already fixed part of it — Path Traversal and SQL Injection recall roughly doubled this cycle after broadening recognized taint sources, with precision essentially unchanged (98.2% → 98.4%) — and the current numbers on this page reflect that fix, not the pre-fix ones. LDAP injection is the next known gap.
See the raw detection output yourself
Open the live demo report — no install, no signup — and inspect real findings directly.
Open the live report See supported languages