Over the past two weeks we shipped four Zebra releases. Zebra 6.4.0 and 6.4.1 brought security fixes, light wallet serving, lower transaction fees and a sync fix, and Zebra 6.4.2 was an emergency fix for a crash that any peer on the network could trigger. Most of our time went into NU7, the next network upgrade: Zebra 7.0.0-rc.0 is its first release candidate, and NU7 activated on public Testnet at block 4,465,026 on 4 October. Mainnet operators don’t need to do anything for NU7 yet, but should make sure they are running 6.4.2.
Security
Zebra 6.4.0 fixes three reported issues, all about how Zebra handles peers that misbehave. A node now keeps syncing and asks someone else when a peer sends it a tampered copy of a block, peers that pass along altered blocks are now penalised, and a banned peer can no longer get around the ban by switching to a nearby IPv6 address. Bans also now expire after 24 hours, so a peer banned by mistake isn’t shut out for good. These fixes went in as one combined change. Thanks to @ouicate and @craftsoldier for reporting them.
Two days later we released Zebra 6.4.2 to fix a crash that a malformed transaction from any peer could cause. It was found by Google’s OSS-Fuzz, a service that continuously feeds random inputs to open-source software to find bugs like this. The 6.4.0/6.4.1 and 6.4.2 release posts have the details.
CI / DevOps
Alongside the four releases, we changed how long a release keeps running before it asks to be upgraded. Each Zebra release stops itself after a set period so nobody runs outdated software through a network upgrade, and we shortened that period from 15 weeks to 12 so that 6.4.x stops around 2 November, ahead of NU7’s tentative Mainnet date of 5 November. Zebra 7.0.0-rc.0 also includes refreshed checkpoints, the known-good points in the chain that help new nodes sync faster.
Other Zebra work
NU7 brings blocks every 25 seconds instead of every 75, so payments confirm about three times faster while the amount of ZEC issued each day stays the same. It also sets aside part of every transaction fee in a reserve that is paid back out to miners gradually in future blocks, and it adds per-block limits on shielded activity that protect light wallets from spam. We implemented the faster block schedule and the new shielded limits, prepared the upgrade’s deployment, switched on the fee reserve, made Zebra stop accepting an older transaction format once NU7 is active, and set the Testnet activation height. While checking the work against the specifications, we closed several gaps in how rewards, network settings and timing, and mining are handled, and added tests that run the new rules through the full validation path and on live test nodes.
Faster blocks also meant revisiting places where Zebra assumed a 75-second block time. Release expiry and sync progress estimates now follow the block time in effect at each height, so a node won’t stop early once blocks speed up. Around any network upgrade, some honest peers briefly relay transactions built for the other side of the switchover; Zebra now gives them a short grace period instead of banning them at the moment the network needs them most.
Zebra 7.0.0-rc.0 upgrades the node’s database automatically the first time it starts, without a resync. Going back to an older version isn’t supported, so back up first if you want a way back. Tools that read Zebra’s database directly, such as Zallet and Zaino, need a compatible version at the same time.
For the lower transaction fee, Zebra now reports the new fee to wallets only from Mainnet block 3,590,000, so wallets move to it together. Until then it reports the current fee, although nodes already accept transactions paying the lower one.
Zebra 6.4.1 fixed a sync stall where a node whose sync restarted near the top of the chain could get stuck until it was restarted by hand.
Work in progress
The next big step is NU7 on Mainnet. NU7 has been live on Testnet since 4 October, which gives us time on a live network before the Mainnet release. Mainnet operators will need to install that release before 6.4.x stops in early November. No Mainnet activation height has been set yet; the overall effort is tracked in this draft.
Some nodes running 6.4.1 can still pause near the top of the chain while they wait on busy peers, or ask peers that don’t keep full block history and give up too soon. A set of fixes under review makes Zebra wait for a peer that can actually serve the block and send its searches for new blocks to those peers, and stops relying on a peer that keeps failing to deliver.
Work on other repositories
Zeeder. Our DNS seeder, the service that helps brand-new nodes find their first peers, had two releases. v1.6.3 moved it onto Zebra’s latest networking code, including the 6.4.2 fix. v1.7.0 adds support for NU7 on Testnet: new nodes on Testnet are now pointed first to peers that already run NU7. Zeeder only stops listing older peers once it has seen for itself that NU7 is active, so the switch doesn’t cut anyone off early. It also checks for the upgrade less often, so it keeps working with 25-second blocks without getting blocked by Zebra’s limit on how often a peer can reconnect. Nothing changes on Mainnet, and operators only need to update the container.
Zinder. Zinder, our early-stage service that indexes the chain from a Zebra node and serves that data to wallets and explorers, no longer crashes in a loop when new blocks arrive while it is catching up. This bug had kept a production faucet offline for about eleven hours.
reddsa. We removed the FROST code from reddsa, the signature library behind Zcash’s shielded spends, so that it can move into the main FROST repository.
ed25519-zebra. We prepared version 5.0.0 of this signature library with updated dependencies.
Thanks to external contributors
Thanks to @evan-forbes for a cache that stops Zebra checking the same shielded proofs twice, once when a transaction first arrives and again when it is mined into a block. It shipped in 6.4.0 and saves work on every node.
Thanks to @judah-caruso, whose early NU7 work from our last update has now merged: the rule that pays the fee reserve back out over time and the change that sets aside 60% of each block’s fees. Thanks also to @yagop, who made it possible for block explorers and indexers to fetch a block together with what each payment spent and the fee it paid in a single request.
Thanks to @ouicate and @craftsoldier for their security reports, and to Google’s OSS-Fuzz for finding the crash fixed in 6.4.2.
And also thanks to @robustfengbin for their contributions.