Zaino v0.8.0 up and running, thanks!
2 Likes
Hey everyone, now that Zaino 0.8.0 is out, I wanted to give an update on both the recent vulnerability reports and disclosure work, and the new mempool subsystem that landed as part of this release. The mempool has been rebuilt from the ground up as its own subsystem, separate from chain state, with a new coherence layer that fixes a race inherent in the old design. That work also resolves the outstanding cluster of mempool-related security reports.
Security fixes in 0.8.0
Unbounded mempool exclusion filter (the most serious of the group).
GetMempoolTx accepts a list of transaction-id suffixes to exclude. There was
no cap on how many suffixes a client could send, no minimum suffix length, and
each suffix drove a linear scan of the entire mempool. A single request inside
the normal message size limit could force billions of prefix comparisons on the
server. Exclude lists are now bounded by count (default 1024) and by minimum
and maximum suffix length (minimum default 4), validated at the request
boundary, and matching is a binary search rather than a per-suffix scan.
Unbounded mempool memory. The old mempool was built on a general-purpose
broadcast structure whose capacity argument turned out to be only a
pre-allocation hint, it bounded nothing. The structure has been replaced.
Memory is now bounded by ZIP-401 transaction cost against a max_cost_bytes
budget (default 128 MiB), enforced as a real admission backstop: additions over
the bound are admitted partially in canonical order, and refusals are
remembered so they aren’t re-fetched from the validator on every poll.
Wasteful subscriber wake-ups and polling. Every mempool subscriber was
woken roughly ten times a second whether or not anything had changed, each
paying a cost proportional to the whole mempool, plus a redundant copy of the
result set per wake. Polling also re-fetched every transaction each cycle
rather than just the new ones. Delivery is now delta-only over a bounded change
feed, polling fetches a light diff with full metadata only for additions, and a
consumer that falls behind receives an explicit “lagged” signal rather than
silently missing updates.
Validator responses are now size-capped. Every JSON-RPC response from the
validator was buffered into memory before its size could be checked, so a
compromised, misconfigured, or impersonated validator could drive Zaino out of
memory with a single reply. Bodies are now read chunk-wise against a 32 MiB cap
and abandoned part-way rather than allocated, and the resulting error is
classified as non-retryable so the cap cannot be turned into an amplifier.
Separately, the few genuinely heavy validator methods now get their own
timeout instead of failing under the tight client-wide one, a slow but healthy
validator should not become a hard error.
A forward-compatibility panic is gone. The old response-parsing crate,
zaino-fetch, has been removed from the workspace and replaced by the new
port/adapter stack. That takes with it the last unimplemented branch in
non-test code, a value-pool parser that would have panicked Zaino on a Zebra
release that added or renamed a pool.
Correctness fixes on the mempool RPCs, described in the next section:
GetTransaction now reports height 0 for an unmined transaction (the
lightwalletd sentinel) rather than the chain tip, which claimed the transaction
was mined at a height it isn’t in; and the reads that place a transaction
relative to a tip now refuse to answer against a stale snapshot rather than
answering from the wrong height.
The new mempool subsystem
The mempool used to live inside zaino-state, on a general-purpose broadcast
structure (a shared map plus a watch channel). That is where the bounding
problems above came from, and it was not a shape that could be fixed
incrementally. ADR-0010 records the redesign.
The bug that forced it
The coherence layer was originally designed to fix a concrete problem found
during early Zallet development, in exactly the way the API was meant to be
used.
ChainIndex::get_mempool_stream takes an optional non-finalized-state
snapshot, and streams mempool transactions until the chain tip moves away from
that snapshot. A consuming wallet’s sync loop therefore does the natural thing:
call snapshot_nonfinalized_state to get a snapshot, then hand that snapshot to
get_mempool_stream. But Zaino’s mempool and its non-finalized state advanced
independently, so between those two calls, or simply because the two had not
yet converged on the same block, the mempool could be sitting on a different
tip than the snapshot the caller had just been given. get_mempool_stream would
then return None, and a wallet’s sync could break.
The failure is worth being precise about, because it is not a case of a client
misusing the API: the snapshot came from Zaino, was passed straight back to
Zaino, and Zaino refused it. The two internal views were allowed to disagree,
and the cost of that disagreement was pushed onto the caller.
Underneath that sat a second problem. Zaino reads the mempool over JSON-RPC, so
there is no atomic “mempool plus the tip it belongs to” primitive to read. The
old code took the set and the tip from two independent reads at two different
instants — and then wiped its entire map on every tip change, answering as if
the mempool were empty until it had re-fetched everything.
So the old design was wrong in two opposite directions at once:
- Reads that don’t care about the tip -
getrawmempool,getmempoolinfo,
GetMempoolTx- were made to stall and under-report once per block, for no
reason at all. - Reads that do place a transaction relative to a tip either failed outright,
as Zallet hit, or could answer using a consensus branch id derived from the
wrong height. A caller has no way to tell a wrong branch id from a right one.
Tip-agnostic core plus a tip-aware coherence layer
The subsystem now splits along exactly that seam.
The core never freezes. It polls the validator, diffs the set, and always
serves the live mempool, so getrawmempool, getmempoolinfo and
GetMempoolTx reflect reality even mid-transition. Critically, it tags every
published snapshot with the validator tip it was fetched at, read from the
same transport that served the mempool data, and discards any poll whose tip
moved mid-window. That makes the set and its tip a single observation rather
than two.
The coherence layer does the freezing. It consumes the core plus an
observer of Zaino’s non-finalized state, and blesses the core’s current set as
valid for an epoch only while the two tips agree. Any disagreement, tip change,
unavailability, or incomplete core set freezes the coherent view: the last
coherent snapshot stays readable and live streams keep serving it until the
tips re-agree at a new tip. Tip-coherent reads return a retryable error instead
of a confidently wrong answer.
For the sync-loop case above, this is the difference between the two views
disagreeing and the caller being made to deal with it: a snapshot handed back
to get_mempool_stream is now served against a view that has been reconciled
with it, and reconciliation happens on the block-arrival wake rather than
waiting out a poll interval.
Because the core tags the tip it read against, coherence is a pure function of
data already fetched, the coherence layer issues no validator reads of its own
and needs no before/after guards. That is what actually closes the race.
It ships as a standalone, modular service
The mempool is now two crates that depend on nothing in zaino-state:
zaino-mempool- the domain types and ports. It depends on
zaino-primitivesandzaino-source, and names no node library at all:
entries hold the validator’s bytes and never parse them. Compact and wire
forms are derived at the RPC boundary.zaino-mempool-service- the runtime: the polling core, the read
handles, and the coherence layer.
Dependencies point strictly inward, and the coupling to chain state is one
explicit, minimal contract rather than ambient access. There is also a
validator-only mode that synthesizes the epoch from the validator tip alone, so
the subsystem runs with no Zaino state behind it at all, it can be embedded,
tested and reused on its own, and the dual-tip assumption becomes a testable
option rather than something hardcoded.
Serving is lock-free: immutable snapshots published behind an ArcSwap, with
shared entries so streaming one transaction to many clients is a refcount bump
each rather than a copy each. Every client-controllable input and the
subsystem’s own memory are bounded.
Operators get a [mempool] section in zainod.toml covering the memory bound,
poll cadence and exclude-list caps. The gRPC and JSON-RPC surfaces are
unchanged - this is an internal rework plus the correctness fixes noted above.
Ports and adapters, and where this is going
The mempool split is one instance of a larger change in 0.8.0. Zaino’s access
to the validator used to be a single 34-method trait declared in terms of
zebra-chain, zebra-rpc and zaino-fetch types. It is now a set of
single-question ports written in Zaino’s own vocabulary, one small trait per
question a validator can answer, with adapters behind them
(zaino-source, zaino-rpc, zaino-source-zebra and friends). zaino-fetch
is gone; the domain types live in zaino-primitives, which depends on
essentially nothing.
The practical benefits are worth spelling out:
- Components are swappable at the seams. A different validator, a different
transport, or a second data source is a new adapter implementing the ports it
can answer, not a change threaded through a monolithic trait. Capability is
structural: an adapter that cannot answer a question simply doesn’t implement
that port, so it can’t be routed a question it can’t serve. - Everything is testable without a node.
zaino-sourceships aMockChain,
and each subsystem can be driven against it directly. The mempool subsystem in
particular can run with no Zaino chain state behind it at all, which is how
its behaviour around tip transitions is tested deterministically instead of
by chasing a race on a live chain. - Failure modes are typed. Errors distinguish “the validator answered, and
the answer is no” from “the transport failed”, so retry policy is a property
of the type rather than a judgement call at every call site, which is what
let the response-size cap be classified as non-retryable rather than becoming
an amplifier.
This restructure is ongoing. zaino-state still carries some documented
temporary scaffolding so existing paths keep working while the new stack is
proven underneath them, and that shrinks as each subsystem moves across. We
expect the restructure to be complete over roughly the next month.
The goal isn’t tidiness for its own sake: it’s the groundwork for the next two
pieces of work - a persistent (im-based) non-finalized state implementation,
and a new modular sync engine. Both need subsystems with clean boundaries and
substitutable data sources to be built and tested independently, which the old
shape did not allow.
Security work in recent releases
0.6.0 — parser hardening. Allocation in the block and transaction parsers
was driven by attacker-influenced length fields; every such site was removed and
the parsers rewritten over zebra-chain with its TrustedPreallocate bounds.
An index-based panic in full-block height extraction went at the same time,
height now derives from the coinbase height through a fallible chain.
0.5.1 — reorgs, passthrough validation and chain-state correctness.
z_get_treestate fell through to an unvalidated validator passthrough when
local resolution failed; it’s now gated on both backends. sendrawtransaction
accepted unvalidated hex, and is now hex- and size-checked. The reorg detector
missed same-height chain replacement and shorter-chain reorgs, and now runs
every sync cycle on both backends with a bounded depth. Reorg recursion itself
is depth-bounded. getblockdeltas silently dropped spend inputs; prevouts are
now resolved locally instead of trusting the validator’s optional fields. The
sync gate, which compared block height only, now compares height and tip hash.
0.4.0 — transport and chain-view hygiene. The served JSON-RPC listener is
private by design but nothing enforced that; the bind is now restricted to
private and loopback addresses, with a regression test. Sidechain block hashes
were being inserted into the canonical height map, polluting the best-chain
view, orphans now land in the block cache only. A panic on empty
network-upgrade metadata in GetLightdInfo was removed.
Reporting and thanks
Thanks to everyone who reported. The mempool findings in particular arrived
with enough detail, including a working proof of concept for the exclusion
scan, to shape the fix rather than just demonstrate the bug, and several
reporters contributed the patches themselves.
Hardening work continues; the remaining known items are lower-severity and are
being worked through in order. If you find something, please report it
privately rather than on the forum, and we’ll get you an assessment and a fix
timeline.
5 Likes
We’re working to integrate benchmarks with every release.
Here’s a point estimate of our baseline.
5 Likes
zainod 0.9.0 – 2026-08-28 Net Change: (+10,896, -3,225)
Hey folks, today zainod 0.9.0 is officially released crates.io, dockerhub, and github.
Details are in the CHANGELOG.
Our next release will be on Friday, September 11, 2026.
Included in this release:
Progress towards the new modular, rapid indexing, cutomizeable zainod.
Our in-flight features are:
- redesign of non-finalized state including input from @hanh
- modular design of the sync engine
- a new test harness
5 Likes
Here are the Sync metrics from the 0.9.0 release:
- Sync-Time ~5.5h (down from ~10.5h)
- Shielded-ops/sec averaging ~6k/sec (up from ~3k/ sec
We have identified some other optimizations via our observability & metrics tooling, and will be bringing those improvements over next few releases
1 Like
Here are the Sync metrics from the 0.9.0 release:
- Sync-Time ~5.5h (down from ~10.5h)
- Shielded-ops/sec averaging ~6k/sec (up from ~3k/ sec
3 Likes
zainod 0.10.0 – 2026-09-11 Net Change: (+48,438, -17,350)
Hey folks, today zainod 0.10.0 is officially released crates.io, dockerhub, and github.
Details are in the CHANGELOG.
Our next release will be on Friday, September 25, 2026.
Included in this release:
Our in-flight features are:
- adding new types to the orchard and sapling action primitives crates to dramatically speed up zebra getBlock RPC
- redefinition of primitives
- improved status and component definition
- redesign of non-finalized state including input from @hanh
- modular design of the sync engine
4 Likes
This release was 4 days late. We are debugging the error in the release process.
zainod 0.10.1 – 2026-09-29 Net Change: (+13,389, -26,439)
Hey folks, today zainod 0.10.1 is officially released crates.io, dockerhub, and github.
Details are in the CHANGELOG.
Our next release will be on Friday, October 9, 2026.
Included in this release:
Upcoming
- modular design of the sync engine
6 Likes