At exactly 10 am Eastern Time last Saturday, I received a Signal call from Daira-Emma, the head of ZODL protocol R&D. An issue had been discovered, and I would need to be read in.
Fifteen minutes later, once I had moved to a secure location, Daira-Emma, Nuttycombe, Str4d, and I converged on a video call. They informed me of the issue that would lead to the remediation of the Orchard vulnerability and the rapid coordination of the most consequential network upgrade in Zcash history.
Physicist @DavidDeutschOxf recently re-posted this tweet thread, which I think is brilliant.
In S3E3 of Rick and Morty, Rick turns himself into a pickle to avoid a family counseling session, with the plan to later revert to his human self. The plan goes awry, and he is forced to survive as a pickle. Using the power of his mind, he subdues the world around him and thrives.
I came across Deutsch’s thread just minutes before I received the call from Daira-Emma. Little did I know that in a few minutes, we would have to overcome time, security, geographic, and technical constraints to subdue the risk of an exploit.
Taylor, a security researcher, disclosed the vulnerability to a subset of our core cryptographically skilled engineers. By the time I received the call the next morning, the fix was in, and the plan was to use a two-step process to first coordinate a soft fork to exclude Orchard transactions from blocks.
This fork immediately mitigated the risk of an exploit without revealing the full scope of the issue before responsible disclosure. Immediately following the soft fork, we would again coordinate a hard fork to remedy the underlying issue and re-enable Orchard transactions.
The four of us agreed to the plan and set timelines to minimize disruption, targeting a soft fork on Monday evening US time, so that most users of the network would wake up in the morning with the issue resolved. It didn’t quite work out that way, but it was pretty close.
On the 31st, we read Pacu in on the nature of the issue, but not the issue itself, to assist in coordinating with as much of the ecosystem as possible. As the fix affected the Zcash Foundation’s zebrad, we read in Pili, the head of ZEC engineering at the time, and ZF engineer Arya the following day. We would likewise read in Peacemonger and Neal from ZODL just prior to releasing the comms on the soft fork.
Over the previous few weeks, I spent time with the heads of a couple of the mining pools at various events, which was fortuitous. The relationships were fresh. We had planned to ship binaries, but even with fresh relationships, the pools and later exchanges wanted to review the code and compile for themselves.
After long chat threads and, in some cases, video meetings to prove I wasn’t compromised, we believed we were ready for the soft fork on Monday at 8:00 p.m. Eastern Time, for the pools to apply the patch. That proved too short.
It assumed everyone would be ready and that there would be no issues with deployment. It wasn’t until after 2:00 a.m. Eastern Time, and some anxiety on my part during a 25-block reorg, on the morning of the 2nd, that we confirmed the soft fork was the winning chain. We immediately posted the announcement as soon as we knew we had won.
Immediately following the soft fork, we agreed that the 2-hour window between code distribution and activation was far too short, even when mining pools were working directly with us in real time. We extended the hard fork window and followed up with node operators about what to expect. We subsequently agreed to extend it further at the request of one of the pools.
Sleep came in the form of two naps for most of us as we prepared for the hard fork. That rollout went more smoothly and activated at 00:05 Eastern Time on June 3rd, about 24 hours following the soft fork. And in the middle, as fate would have it, I got food poisoning. Lovely.
Yes, we are building a hardened protocol with a long-term view, but the solution to hardening and the subsequent proposal for a new shielded pool, Ironwood, were parochial. We resolved the issue, battle-tested our incident support processes, built stronger relationships with others who support the network, tested our own resilience, and unified as a community of builders to agree on a path forward. Zcash is stronger than it has ever been.
While many people and teams worked collaboratively through these challenges this week, I want to specifically thank the teams at ViaBTC and Foundry, who coordinated with us around the clock during the upgrade.
ZODL updates for the week.
Zodl (formerly Zashi)
- Coinholder Polling 3.5.0 went live on Android on Monday, completing the cross-platform launch of the first in-wallet shielded coinholder voting flow on a Zcash mobile wallet.
- Shipped a three-release Android stabilization wave (3.5.0, 3.5.2, 3.5.3) and a two-release iOS wave inside a single week. 3.5.2 carries the Auto and Manual server-selection redesign, the keep-screen-on fix during ballot submission, and the in-app polling-flow UI fixes. 3.5.3 removes the decommissioning lightclient endpoints ahead of the June 7 cutoff.
- Default lightclient server moved to zec.rocks on both platforms after a 50-country benchmark of nearly 9,000 RPC calls. jp.zec.stardust.rest and eu2.zec.stardust.rest are queued for removal.
- Memory Tagging on Android FOSS builds reached QA. The iOS Tor stream-timeout user-reported sync error and Automatic Server Switch also reached Ready for Testing.
Looking ahead at Mobile:
- Ship Zodl 3.5.2 on iOS and 3.5.3 on Android ahead of the June 7 endpoint cutoff.
- Continue on Zodl 3.6.0 scope: multi-server transaction submission, multi-currency conversion, SDK chain-tip tree-state optimization, with continued Valar Group collaboration.
Analytics (deltas reflect a two-week window; the prior published numbers were WC 18 May)
Zodl iOS:
- Unique Installs: 43.6k (+1.1k / 2 wks)
- Total Downloads: 52.1k (+1.5k / 2 wks)
- App Store Rating: 4.9★ (no change)
Zodl Android:
- Install Base: 16k (+0.2k / 2 wks)
- Total Installs (incl. Open Beta): 52k (+1.3k / 2 wks)
- Play Store Rating: 4.25★ (+0.01 / 2 wks)
Zcash Core
- Shipped the emergency NU6.2 network upgrade (zcashd v6.20.0) at mainnet block 3364600 (~03:00 UTC, Wednesday, June 3), remediating the privately disclosed Orchard circuit flaw in halo2_gadgets.
- Shipped a parallel time-critical zcashd v6.12.5 in coordination with a soft fork at mainnet block 3363426 (~02:00 UTC, Tuesday, June 2), fixing a coinbase value-balance desync that could node-crash on a single valid proof-of-work block and temporarily turning off Orchard actions until NU6.2 activated.
- Cleared the Zallet RPC review queue in a single Tuesday-evening pass: cookie-file JSON-RPC authorization, z_importviewkey, z_shieldcoinbase, and a lightwalletd darkside bugfix.
- Opened a scoping spike on a ZODL fork of Zebra, plus ZODL-operated lightwalletd and Zaino infrastructure. The upstream piece of the wider zcashd end-of-life arc.
- halo2_gadgets backports filed across the cryptography surface as part of the post NU6.2 cleanup arc.
Looking ahead on Core:
- Collaborate with other ecosystem organizations on the plan for the next accelerated network upgrade.
- Continue the ZODL fork of Zebra, plus own lightwalletd and Zaino infrastructure scoping toward a concrete deployment plan.
- Land the Zallet alpha.4 PRs in review and ship Zallet alpha.4.
- Continue on critical Zallet functionality at an accelerated pace.
Other / Company
- Coinholder and ZCAP polling for NU7 has been delayed due to our plans for Ironwood.
- ZODL Summit invitations and details have been sent out to those approved. We’ll follow up with others on the waitlist as soon as we receive confirmations.
- Blockchain Association membership submitted, with approval expected this week.
- Paul Brigner represented Zodl at the BA Law Enforcement fly-in. Kyle Schneps from DCG prominently mentioned Zcash during his remarks at the congressional briefing. Paul’s thread on the day.
- Zodl signed onto the Solana Policy Institute Founder/CEO Coalition Letter on Developer Protections.
- Outside counsel onboarding underway with Accrediv and Arktouros.
- We’re hiring. Open roles at jobs.ashbyhq.com/zodl.
- Layperson’s description of “formal verification” and what’s next for Zcash protocols. This was posted following Shielded Labs recommendation for a nool pool and our subsequent agreement on the scope the next day.
Market and Ecosystem
- Posts on Ironwood: Project Tachyon, Shielded Labs, ZODL
- Reactions from Mert and Iilya on the Rollup, Cameron Winklevoss, and Taylor Hornby
Subduing the world,
Onward.