Over the last two weeks we focused on making Zebra sync reliably all the way to the chain tip, which had been a recurring problem since the NU6.3 upgrade. We also replaced Mergify with GitHub’s native merge queue, simplifying our CI workflow and removing custom pipeline plumbing that was no longer needed, kept trimming the node’s memory use and response times, and contributed reliability fixes into Zallet, the zcashd replacement wallet maintained by ZODL. Alongside our own work, community contributors submitted two useful hardening changes, described below.
Security
Two changes from community contributors this period make it harder for a single source to use up more of a node’s resources than it should.
@LarryRuane improved one of Zebra’s built-in wallet-facing lookups to stop as soon as it has the number of results the caller asked for, instead of reading everything an address holds and then trimming the answer. This keeps a small request from turning into a large amount of work on the node, and it finishes off a fix for a previously reported advisory. As a bonus, the same requests now return noticeably faster.
@craftsoldier changed how the per-address connection limit is counted for IPv6 so it applies to a whole block of neighbouring addresses rather than one exact address at a time. Without this, a single machine that owns a large range of addresses could dial in from a fresh one each time and slowly take up all of a node’s connection slots. Operators who genuinely run several nodes from one address range can raise the limit.
CI / DevOps
We replaced Mergify, the third-party tool that managed our merge queue, with GitHub’s own native merge queue, which lets us delete a lot of custom pipeline plumbing that only existed to support the old tool. We also split image building from deployment in our node-deployment workflow, so a release can build an image once and later deploy exactly that image; hardened the cleanup and safeguards around our temporary cloud test machines; improved our integration-test logging and scheduling; and finished a move to a cleaner, more consistent set of issue and CI labels.
Other Zebra work
We fixed a problem where a node could get stuck repeatedly restarting its sync just below the chain tip: near the tip a block often depends on a neighbour that is still being saved, and the old behaviour treated that brief wait as a failure and threw away work in progress, so the node looped in place. It now just retries the affected block and moves on. This had been failing our synced-node tests ever since NU6.3.
We also kept the node lighter and quicker. Validating a transaction with a very large number of inputs no longer pulls a disproportionate amount of data into memory, so memory use stays bounded on big transactions, and we now keep certain proving parameters ready in memory so some wallet-facing queries respond faster.
For operators who would rather not sync from scratch, we published a guide to starting a node from ZF’s public state snapshots, including how to download, verify, and load one, and what trust that does and does not give you.
Work in progress
We continued early work on a faster way for nodes to connect and sync with each other, building on the experimental QUIC-based connections we mentioned in the last update. Over the past two weeks we opened several more draft PRs for this work: a new connection and handshake path between peers, the message formats they exchange, and a way to sync chain data in verifiable, reusable chunks. This is still experimental, turned off by default, and not ready for use.
Work on other repositories
We contributed two reliability fixes into Zallet, the zcashd replacement wallet maintained by ZODL. A temporary problem communicating with the node a wallet syncs from is now treated as something to retry cleanly, rather than being mistaken for the wallet having caught up to the chain tip; and errors that happen while streaming pending transactions are now surfaced instead of being read as a normal end of the stream. Both reduce the chance of wallet sync silently stalling.
Thanks to external contributors
Thanks to @LarryRuane and @craftsoldier for the two hardening fixes covered in the Security section above. Thanks also to @str4d, who fixed a piece of Zebra’s block-verification bookkeeping so that an unrelated error near a checkpoint no longer rewinds verification progress and stalls it; this mainly helps tools that feed blocks into Zebra directly, such as migrating a wallet from zcashd.
And also thanks to @Bortlesboat for their contribution.