Multi-Layer Analysis — Detailed Specification
Goal
Move from a single-layer analysis (one mode per rule) to a multi-layer analysis where each file is examined by 3 independent strategies. The results are cross-referenced to produce high-confidence findings with a chain of evidence verifiable by a human.
Current architecture (single-layer)
Today, each rule uses ONE mode:
Rule sql_injection_fstring
→ regex mode: searches for ".execute(f""
→ Match? → Finding (confidence 80, fixed)
→ No match? → Nothing
Issues:
- If the pattern is too narrow, the vulnerability is missed (false negative)
- If the pattern is too broad, noise is detected (false positive)
- No proof of data flow (the dev sees "SQL injection" but not WHY)
- Confidence is fixed (80 or 95), not based on actual evidence
Proposed architecture (multi-layer)
For each file, 3 layers run in parallel:
┌─────────────────┐
│ Source file │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Layer 1 │ │ Layer 2 │ │ Layer 3 │
│ Pattern │ │ Taint │ │ Context │
│ matching │ │ analysis │ │ analysis │
└──────┬─────┘ └──────┬─────┘ └──────┬─────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────┐
│ Evidence merger │
│ │
│ Cross-references results from 3 layers.│
│ Computes the actual confidence. │
│ Builds the chain of evidence. │
└────────────────┬────────────────────────┘
│
▼
┌───────────────┐
│ Finding │
│ + evidence │
│ + confidence │
└───────────────┘
Layer 1: Pattern matching (existing, improved)
What it does
Searches for syntactic patterns in the code (text or regex). This is the current layer — fast, broad coverage.
What it produces
{
"layer": "pattern",
"rule_key": "sql_injection_fstring",
"file": "src/db.py",
"line": 42,
"code": "cursor.execute(f\"SELECT * FROM users WHERE id = {user_id}\")",
"evidence": "Pattern '.execute(f\"' matched at line 42"
}
Confidence alone: 65%
A pattern match alone is a signal, not proof. The pattern can match safe code (e.g. a test, a comment, a documentation string).
Layer 2: Taint analysis (existing, enriched)
What it does
Traces the data flow from an untrusted source (HTTP input, CLI, file) to a dangerous sink (execute, open, system).
What it produces
{
"layer": "taint",
"rule_key": "taint_sqli",
"file": "src/db.py",
"flow": {
"source": {
"line": 12,
"code": "user_id = request.args.get('id')",
"kind": "http"
},
"propagation": [
{"line": 15, "code": "user_id = int(user_id) if safe else user_id"},
{"line": 38, "code": "query = f\"SELECT * FROM users WHERE id = {user_id}\""}
],
"sink": {
"line": 42,
"code": "cursor.execute(query)",
"type": "sql_execute"
},
"sanitizers_found": false
},
"evidence": "HTTP data (request.args) propagated without sanitization to cursor.execute()"
}
Confidence alone: 85%
A taint flow is strong evidence — it shows the complete path. But it can be a false positive if the sanitizer lives in a function called indirectly (outside the file).
Layer 3: Context analysis (new)
What it does
Analyzes the file context to reinforce or refute findings from layers 1 and 2. Answers questions such as:
-
Does the file use an ORM (SQLAlchemy, Django ORM)? If so, a SQL injection is less likely.
-
Does the file import a sanitization framework?
- Is the file a test, a mock, a fixture? If so, ignore.
- Does the file handle HTTP requests (routes, handlers)?
- Does the file contain input validation (schemas, validators)?
What it produces
{
"layer": "context",
"file": "src/db.py",
"signals": {
"is_test_file": false,
"has_orm": false,
"has_raw_sql": true,
"has_http_handler": true,
"has_input_validation": false,
"has_sanitizer": false,
"framework": "flask"
},
"evidence": "Flask file with raw SQL, no ORM, no input validation"
}
Implementation
The context layer is a set of file-level checks:
def analyze_context(filepath, content):
signals = {}
# Test/mock/fixture detection
signals["is_test_file"] = any(x in filepath.lower()
for x in ["test_", "_test.", "mock", "fixture", "spec."])
# ORM present?
signals["has_orm"] = bool(re.search(
r"(?:from sqlalchemy|from django\.db|from tortoise|"
r"from peewee|from prisma|\.objects\.|\.filter\(|\.query\.)",
content))
# Raw SQL?
signals["has_raw_sql"] = bool(re.search(
r"(?:SELECT|INSERT|UPDATE|DELETE)\s+", content))
# HTTP handler?
signals["has_http_handler"] = bool(re.search(
r"(?:@app\.route|@router\.|def\s+\w+\(request|"
r"async\s+def\s+\w+\(request)", content))
# Input validation?
signals["has_input_validation"] = bool(re.search(
r"(?:pydantic|marshmallow|cerberus|voluptuous|"
r"wtforms|django\.forms|Schema\(|validate\()", content))
# Sanitizer present?
signals["has_sanitizer"] = bool(re.search(
r"(?:bleach|escape|sanitize|parameterize|"
r"quote\(|html\.escape|markupsafe)", content))
return signals
Evidence merger
Confidence calculation
The final confidence is not an average — it is a calculation based on independent pieces of evidence:
Confidence = base + taint_bonus + context_bonus - penalties
Concrete cases:
Pattern alone, no context:
65% (weak signal, may be a false positive)
Pattern + confirming context (raw SQL, no ORM, HTTP handler):
65% + 15% = 80%
Pattern + taint flow (source → sink confirmed):
65% + 25% = 90%
Pattern + taint + context (triple confirmation):
65% + 25% + 10% = 95% (near certainty)
Pattern + REFUTING context (ORM present, test file):
65% - 30% = 35% (likely false positive → removed from report)
Merging rules
1. If the context layer detects a test file → REMOVE the finding
(test files intentionally contain vulnerable code)
2. If the taint layer confirms the flow → INCREASE the confidence
and ADD the chain of evidence to the finding
3. If the context layer refutes (ORM present, sanitizer found) →
DECREASE the confidence. If < 40% → REMOVE the finding
4. If two different rules detect the same vulnerability at the same place →
MERGE into a single finding with the highest confidence
5. If the taint flow crosses several files → ADD the "cross-file"
label and INCREASE the confidence by 5%
Chain of evidence in the report
What the developer sees today
[HIGH] SQL Injection (f-string) — src/db.py:42
Risk: An attacker can inject malicious SQL...
Solution: Use parameterized queries.
No context. The developer has to figure out alone why this is a problem.
What the developer would see with multi-layer analysis
[HIGH] SQL Injection (f-string) — src/db.py:42
Confidence: 95% (pattern + taint + context)
Evidence:
1. Source (line 12):
user_id = request.args.get('id')
→ Untrusted HTTP data (kind: http)
2. Propagation (line 38):
query = f"SELECT * FROM users WHERE id = {user_id}"
→ Variable injected into a SQL query via f-string
3. Sink (line 42):
cursor.execute(query)
→ Query execution without parameterization
4. Context:
- Flask file with HTTP handler (@app.route)
- Raw SQL (no ORM)
- No input validation detected
- No sanitizer detected
Solution: Use parameterized queries.
Playbook: 3 steps (diagnosis → fix → verification)
References: CWE-89, OWASP A03:2021, ISO A.8.28
The developer can verify each step of the evidence. If they think it is a false positive, they know exactly what to check.
Impact on false positives
The context layer drastically reduces false positives:
| Situation | Today | Multi-layer |
|---|---|---|
| Pattern in a test file | Finding (FP) | Removed (is_test_file) |
| Pattern in a file with ORM | Finding (FP) | Confidence 35% → removed |
| Pattern + confirmed taint | Finding (80%) | Finding (95%) |
| Pattern in a comment/string | Finding (FP) | Confidence 35% → removed |
Estimate on the current benchmark
| Metric | Current | Multi-layer (estimated) |
|---|---|---|
| Precision | 81% | 93%+ (context filters FPs) |
| Recall | 99% | 99% (same coverage) |
| F1 score | 89% | 96%+ |
Files to modify
| File | Action |
|---|---|
templates/finding-item.html |
Modify — <details> block for the evidence |
What this does NOT solve
Multi-layer analysis improves the quality of existing detections. It cannot detect:
-
Pure logic bugs (e.g. IDOR Lunary CVE-2024-1625, where the issue is a missing WHERE in a query — not a syntactic pattern)
-
Vulnerabilities in dependencies (covered by pip-audit, not by SAST)
- Vulnerabilities in compiled/obfuscated code
For these cases, the solutions are:
- IDOR: dedicated rule "CRUD without ownership check" (rule 4 of BountyBench)
- Dependencies: pip-audit/npm audit scan (already implemented)
- Compiled code: out of SAST scope