Hi all,
Posting here for community review of ZCG application #389 — a project that audits Zebra’s existing test suite rather than adding to it.
Zebra’s main branch carries more than 1,200 test functions, and nothing currently measures whether those tests hold the behaviour they appear to cover. Four of Zebra’s published advisories exist because an earlier fix for the same defect did not hold, each recurrence appearing within sixteen days of that fix — and the earlier fixes that can be traced to a commit all shipped with dedicated regression tests. The tests were written, they passed, and the defect came back anyway. Upstream put it plainly in one of those commits: “The existing downloader-level test exercised the per-peer cap, which already worked before the fix — the regression lived in this routing decision in inbound.rs.”
So the gap is not a missing test. It is that a test can exist, pass, and still not be holding the defect — and from outside, and from CI, it looks identical to one that is. The method is targeted mutation: take real code on a consensus-critical path, break it in one specific way, and run the existing tests. Everything staying green marks a hole in the safety net, and we close each confirmed one with a regression test verified to fail while the defect is present, then offer it upstream as an optional patch.
This costs the maintainers nothing by design: Zebra’s source is never edited, none of their CI is used, no milestone depends on anything being merged, and one success metric is that required requests to them total zero.
The part we would most like feedback on is disclosure. A public map of where a node’s safety net is thin is a map for an attacker, so anything pointing toward consensus divergence, funds safety, or remote crash is withheld and goes to the maintainers privately; only method, tooling, aggregate figures, and already-closed gaps get published
. That makes the public output thinner than it could be, and it means patches for withheld findings carry no public credit for us. Where safety and credit conflict, safety wins.
The contributor record behind this application is mine: three merged fixes in ZcashFoundation/zebra and one still open, on consensus-critical paths. No write access, not a maintainer, no organisation, no prior grant delivery.
Full proposal — method, operators, milestones, acceptance criteria, the full disclosure model, team and budget: Grant Application -Test-Effectiveness Audit for Zebra: an independent regression witness for consensus-critical code · Issue #389 · ZcashCommunityGrants/zcashcommunitygrants · GitHub
Thanks for reading. Feedback on the disclosure model especially is welcome — that is where we would much rather be wrong early than late.