Blog·October 22, 2026

From 1,249 risky paths to a fix order: what scoring five open-source projects taught us

A dependency analysis that finds problems has a second problem: it finds a lot of them.

When we ran vdb harden on one of our own projects, 82 packages deep, it came back with 1,249 dataflow paths from that project's code to something dangerous. Every one was a fair finding. None of them told anyone what to do on Monday morning. A list of paths is not an order of work.

So we built the layer that turns the list into one. It answers three questions:

How risky is this project, as one number you can track?
When something changes upstream, does it matter to us?
What should we fix first, and what will that buy?

This is how each works, and what running it on our own projects and on five popular open-source ones taught us.

One graph for every way in

The hardening engine already decides paths: an argument enters your code, crosses into a dependency, and reaches a sink such as a shell, a parser or a file path. Those paths become edges in an attack graph, with call sites, packages and sinks as nodes.

The important change is what counts as an entry point. It is not only a request handler. Six kinds of entry go into the same graph, on the same scale:

Entry	Example
Input to your code	a request argument, a CLI argument, an environment variable, stdin
A package name that does not exist	an AI agent added a dependency it invented
A newly registered package	a name that appeared on the registry days ago
An install script	a package that runs code when it is installed
A tool server's scope	an MCP server that declares shell or file access
A published CVE	weighted by whether it is known to be exploited

That is the point of the graph. "An agent added a dependency that does not exist" and "an exploited advisory is reachable from your upload handler" end up in one ranking, instead of in two tools that never talk to each other.

Scoring a path, and a project

A path's score multiplies four things:

The entry point's threat probability. A request argument is attacker-controlled by definition; an environment variable mostly is not. A CVE on the known-exploited list counts in full; otherwise its exploit-prediction score (EPSS) is used.
A confidence per hop. Lowered wherever the engine had to assume propagation, through a package with no summary yet or through dynamic dispatch.
A risk factor per package. From a trust score: registration age, release history, a tool server's recent scope expansion. The same flow through a days-old package scores higher than through a decade-old one.
The sink's impact. Command execution and deserialization at the top; file paths and network requests below; regular expressions, decompression and global state lower still.

Path scores are combined into a project score the way independent risks combine (noisy-OR): ten medium paths are worse than one, but no number of paths makes the score exceed certainty.

A score that admits what it didn't see

The part we care about most is coverage. If half the packages in your graph have no summary yet, a low score means "we couldn't see", not "you're safe". So the project score always comes with the share of the dependency graph that was actually analysed, and when that share is too low the score is an interval:

the lower bound from the paths that were decided,
the upper bound assuming every undetermined path reaches the worst sink.

A path whose boundary already applies every defence its sink requires, or whose closure you verified, scores zero.

Here is the real output for the small example on our Hardening page: one Flask file, five pinned dependencies, two of them without a summary yet.

$ vdb harden app.py --manifest requirements.txt
5 dependencies on the graph · 3 analysed · 2 without a summary yet · taint reached 3
Project risk: 0.65–0.75  (interval — analysis incomplete; 3 of 5 dependencies have a summary, 4 undetermined path(s))
  low        0.32      code_input request > requests@2.34.2 > net_request  in app.py
  high       0.25      code_input param > stdlib > cmd_exec  in app.py
  low        0.16      code_input request > requests@2.34.2 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4175 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4177 > urllib3@2.7.0 > net_request  in app.py
  low        0.06      cve PYSEC-2026-4176 > urllib3@2.7.0 > net_request  in app.py
  undetermined: 4 path(s) end in a package without a summary — most often idna@3.11 (2), certifi@2026.7.22 (2); queued for building, re-run later for a sharper answer

A request flowing into an HTTP client, a parameter reaching a shell command, and three advisories on the pinned urllib3 sit in one list, on one scale. The four paths into packages without a summary are counted in the interval, not dropped.

Keeping the score current

A score is only true for the dependencies you had when it was computed. So the graph is stored, and four kinds of event re-score it:

an advisory is published for a package in it,
a summary is built for a package that had none,
the registry changes (a new release, an install script, a package that disappears),
a tool server's declared scopes change.

Each event finds the affected graphs through an index and re-scores only the paths it touches. You are mailed only when your project's score moved past a threshold or a new path became the top one, not for every CVE in every package you happen to include. Paths that rest on an assumption are listed in the report but never announced.

We learned that last rule the hard way. An earlier version mailed on every change in a path's verdict, and an engine upgrade re-decided paths for projects analysed days before. The difference looked like new findings, and people got mail about nothing. An engine change now re-baselines silently: a new version of our analysis is not a change in your code.

The same index works in the other direction. For packages that have no summary yet, it estimates how much a CVE on each would add to the projects that use it, and builds the most consequential summaries first.

What to fix first

This is the question people actually ask. To answer it, every candidate fix is applied to the graph counterfactually, and the resulting score is compared with the current one:

Upgrade a package, with the new version's summary composed again rather than assumed.
Harden a boundary: wrap one call site, or all call sites that reach one sink.
Remove a dependency.
Change execution policy, such as installing without running install scripts.

Each candidate also gets a cost: how far the version moves, whether it breaks a version constraint another package declares, and how many call sites have to be edited. Candidates are ranked by risk reduction per unit of cost, and the ranking ends with the combination that brings the project under a target score. Each step's effect is re-computed, not added up.

For the same file:

$ vdb harden app.py --manifest requirements.txt --remediate
Remediation ranking — risk 0.65–0.75, target 0.20
   #  action                                                   Δrisk  cost  value  closes
   1  harden app.py:6                                           0.26  0.20  0.219  2 (worth 0.43)
   2  egress-allowlist on net_request sinks                     0.17  0.40  0.125  0
   3  remove requests                                      0.19–0.26  0.80  0.106  2 (worth 0.43)
   4  harden app.py:9                                           0.12  0.20  0.097  1 (worth 0.25)
   5  upgrade urllib3 2.7.0 → 2.8.0                        0.12–0.16  0.40  0.083  7 (worth 0.31)
  ...
  Combination: #1 → 0.39–0.55, #2 → 0.32–0.51, #4 → 0.09–0.34  (reaches the target; 3 step(s), cost 0.80)
  Re-verify after each step: vdb harden … --verify

Hardening the request handler leads: it closes both paths through requests for the cost of one edit. Upgrading urllib3 closes more paths but buys less, because those paths were worth less. An upgrade's reduction is an interval when the new version's summary can't decide everything the old one did. And the last line is the answer to "what do we do this week": three steps reach the target.

What real projects taught us

We ran it against that 82-package project, against VDB's own API and a small FastAPI service, and then against seven more local projects. Five things changed as a result.

The ranking took five minutes. Each candidate was evaluated by copying the whole graph and re-scoring everything. Now a candidate is an overlay (a sanitizer here, a removed node there) and only the paths it touches are re-scored. On the large project, 86 candidates over 1,249 paths went from 310 seconds to 0.4.

Every candidate ranked at zero. On a project with hundreds of undetermined paths, the upper bound of the score was already at its maximum, so no single fix moved it. Ranking by a number that can't move is useless. Each candidate now also reports the worth of the paths it closes, so a report says "closes 1 path worth 0.50" instead of "reduces risk by 0.00".

Four of seven projects couldn't be analysed at all. Their requirements.txt used ranges (>=2.0) instead of pins, and the analysis only accepted exact versions. An unpinned requirement is now a node without a version: no summary or advisory can be decided for it, the report says so, advisories are filtered by the declared range where that is provable, and the ranking offers to pin it, listed as "unlocks analysis" even before any score moves. You can't fix what you can't see, so making it visible ranks first.

Backups and vendored copies drowned the signal. One project tree had 295 files and 9,336 call sites. Two --exclude patterns (an old backup and a vendored copy) brought it to 126 files and 3,952 call sites, the ones that actually ship.

The report had to be readable by a person. Identical path shapes are grouped (high 0.50 ×20 … in a.py, b.py +9 more), files are named instead of numbered, undetermined paths become one line with the packages they end in, and candidates that change nothing are no longer listed. "Harden all N call sites that reach subprocess" turned out to be the most useful line in several reports, and it didn't exist until we saw the reports.

Five popular open-source projects

Our own projects are not a fair test: we know their code. So we ran the same analysis on five widely used open-source applications. These are measurements of our analysis, not findings about those projects; we report no vulnerabilities in them here.

Three things broke, and all three are now fixed.

Derived values were scored as if an attacker typed them. In cmd = "convert " + name, the abstraction recorded an origin only for values read explicitly (a request field, an argument), so cmd had no origin, and an unknown origin fell back to "attacker input", the highest weight. Now every source records where it came from, and a derived value inherits that through the assignment flows. A value with no trail at all is of unknown provenance, and weighted at half a request's. On the FastAPI template, the score went from "1.00, saturated" to 0.26–0.52, with one real path.

Every large project scored 1.0. Combining paths the way independent risks combine is right for a handful of paths. With a few hundred open, it reaches 1.0 and then says the same thing about every large codebase, which tells you nothing. So the score now carries three more numbers:

open paths, how many are still open,
risk mass, the sum of the path scores (7.5 for paperless-ngx, 109 for AutoGPT's backend), which keeps growing after the combined score has stopped,
the highest path per sink class, so "saturated" still says where.

When the score is saturated, the report leads with these instead.

The ranking chased noise. Under saturation, the combined score moves by less than 0.005 for almost any fix, and those rounding-level differences were outranking a candidate that closes 180 paths of risk mass 8 for one edit. Below that threshold, candidates are now ordered by the risk mass they close per unit of cost. An upgrade also reports the paths it newly decides beside the ones it closes, and the combination counts a step only when it removes mass the earlier steps had not.

And one practical fix: --exclude now takes file patterns as well as directories, so *_test.py and conftest.py stay out of the analysis, and the report says when a file limit or the path ceiling was reached instead of leaving that silent.

What the score is not

It is not a probability of being breached. It is an ordering: this project against its own past, this fix against that one. The weights are judgements, written down and applied the same way everywhere, not measurements.
It is only as wide as the analysis. The coverage figure and the interval are there so a gap reads as a gap.
On a large project it saturates. Past a few hundred open paths, read the open-path count, the risk mass and the per-sink maximum, not the combined score.
It follows the paths the engine can decide. Everything we wrote about the engine's limits, in why following the call is the hard part, still applies underneath it.

The goal is modest and specific: when the analysis says 1,249 paths, the next line should say what to do first and what it will buy.

This is the second patent-pending part of vdb harden (Korean patent application). The first is the call-site hardening it builds on. Try it on your own code from Hardening.