Following conversations at Zcon4 in Barcelona, the Zcash Foundation (ZF) and Electric Coin Co. (ECC) are excited to announce that we’ve agreed to move core Zcash protocol work from zcashd to Zebra!
Zebra is a brand new Zcash full node implementation written from the ground up in Rust. Both the ECC and ZF teams believe the modern architecture and good engineering in Zebra will provide a safer and more efficient implementation of the Zcash core protocol.
Zebra was created by ZF with significant support and assistance from ECC developers, and relies on Zcash core libraries authored at ECC. Both organizations’ teams will collaborate to identify what functionality needs to be added to Zebra to address the necessary use cases, and to encourage and support key users to migrate to Zebra. Zcashd will continue to be supported during the transition, which will take an undetermined amount of time. ECC plans to continue independently supporting core Zcash protocol upgrades, such as its planned implementation of proof of stake, using Zebra.
ECC engineers built the original Zcash full node (“zcashd”) and have maintained and supported it through 69 releases and 6 network upgrades since 2016. This original implementation inherits the Bitcoin codebase and has grown challenging to maintain. Today’s announced transition will reduce the time it will take to deploy new features for Zcash, and allow more engineering time toward wallet SDKs and a transition away from PoW consensus. For ECC, this is part of its strategy of “Decentralize Zcash and Focus ECC”. For ZF, this marks a significant milestone for the Zebra project.
As part of this transition, Zooko at the Electric Coin Company has gifted a zebra suit previously seen on stage at Zcon1, to Jack Gavigan of ZF.
As a team that depends on zcashd for some of our infrastructure, it would be helpful to know if the full node wallet functionality will be ported to zebra as part of this.
Sure but what does this mean? Why? As you said zcashd was built using Bitcoin. That wasn’t safe? How much more efficient? What new features? Adding more fluff goes along way.
The zcashd codebase is written in C++ (a memory unsafe language) and is behind on a lot of the maintenance and refactoring PRs that have been made to bitcoin since zcash was forked from it. Moving to zebra should speed up development because its code is cleaner, and zebra is much safer due to it being written in Rust.
For some of us who still use the zcashd node and wallet.dat, what’s the upgrade path? I already got a zebrad node running but it appears to me zebra is just a node and does not handle wallet.
I don’t mind using command line wallet as far as it’s secure, i had no problem using zcash-cli wallet.
For zcashd internal wallet, there really isn’t one. The version will be bumped every so often for the forseeable future while people use it for things that zebra will not support (the internal wallet is not the only thing). Eventually, however, it is likely will need to move your funds to a new wallet.
I’d love to see zebra have a clean, minimal wallet to replace a handful or two of the RPC endpoints that zcashd has. Even better, or in addition, I’ve always wanted a sort of event-driven interface like a pubsub where code running with the node could subscribe to events like “transaction recieved.” Has anyone ever proposed something like this?
The kind of event-driven interface you’re talking about here is effectively how the zcashd internal wallet currently relates to the node, but using internal queues and process separation. The thing that makes this kind of approach tricky is that chain reorgs have to be handled gracefully, so a pub/sub protocol consumer has to be able to figure out which is the best fork and maintain the information needed to switch between forks.
My hope is that the effort to replace the zcashd wallet results in a wallet that is fully decoupled from any particular full-node implementation. I have suggested elsewhere that a potential way forward is to extend the lightwalletd protocol to support the full range of operations needed to support a wallet with equivalent capabilities to the zcashd wallet, and that then the initial phase of “full node wallet support” will likely involve running a local private lightwalletd atop a local zebrad node, with a zcashd-wallet-equivalent built atop the zcash_client_backend crate connecting to that local lightwalletd. In the future, it might make sense to integrate the lightwalletd functionality more directly into zebrad, but I don’t really have enough knowledge about the internals of zebrad yet to know whether this would be a sensible design. One advantage of a lightwalletd based approach is that we already have a pretty good story for how to handle reorgs; the lightwalletd server only serves the “best” chain, and then if a client detects a continuity error they re-request data until they can find the correct history.
In a limited way, the lightwalletd GRPC protocol already has some pub/sub characteristics; for example, all of the block retrieval is handled using streams, which can be held open and consumed incrementally. It’s not a full pub/sub architecture, but the implications for the code aren’t all that different.
The problem is that for this to work, zcashd has to be able to function as a full node with respect to consensus. So, for example, if NU6 were to include ZSA support, and the ecosystem hasn’t migrated off of zcashd by that point, then both zcashd and zebrad would need to be upgraded to provide ZSA support. That’s a non-trivial amount of work, and if it’s possible to conserve resources by only doing it once, I’d rather it just be done atop zebrad. The community has invested very heavily in getting zebrad into production, and that investment will only really be able to pay off once it relieves the maintenance burden that zcashd imposes.
The path forward that I think we should pursue here is to attempt to extend zcash_client_backend to support transparent functionality in more of a first-class fashion; right now, transparent functionality is relatively limited. While the lightwalletd infrastructure supports some of the functionality that would be needed, I think that there are currently some gaps; once those gaps are identified, then there may be additional lightwalletd work needed, and that will in turn may require additional data from the node, or it may be that the light wallet GRPC API already provides sufficient functionality (though additional caching behind those APIs may be required for performance reasons.) A server wrapping this wallet functionality in a zcashd compatible RPC interface would then be the host for this kind of “wallet”, which would connect to a local or private lightwalletd node for its data source.
The major challenges associated with this approach are:
The challenge of creating a safe and complete migration from the wallet.dat format, including the management of existing key data.
We would need a complete stack for interacting with Zcash’s flavor of transparent Bitcoin script implemented in Rust; this doesn’t exist yet.
Devising a testing regime that can give us confidence in RPC API compatibility and feature-completeness.
An alternative approach would be to attempt to factor out the zcashd C++ wallet from the zcashd codebase. However, this would almost certainly require the creation of new APIs to enable connecting to zebrad, and there are substantial challenges related to the extraction, as well as testing and build infrastructure if we chose to go with this route. At the end of such an extraction process, we’d still be faced with an ongoing maintenance burden, and would be essentially just pushing the migration costs described above into the future.
I can’t accurately estimate which of these options would take more time to implement in isolation; they both involve a substantial (and probably loosely comparable) amount of work. The relative merit of these options depends heavily upon what the future of transparent zcash is decided to be. If we expect transparent functionality to persist for several years into the future, then the former approach is certainly cheaper overall; if the community wants to instead sunset transparent functionality in the next couple of years, then the latter approach might be preferable.
What’s the time estimate here? Does it make sense to keep focusing on transparency? Is this for backwards compatibility? Is there a way to avoid having to do this work?
Seems a bit of a time-sink, when first-class support for shielded would be better for the platform long term.
For example, lightwalletd does not offer a way to retrieve a block header without fetching the whole block.
Support for transparent transactions in lightwalletd is pretty much limited to forwarding a few selected calls to zcashd. However, to provide more value, the wallet transparent transactions (not all t txs) should be included as part of the scan. Also, we need to have the resolution of inputs from TxIn to previous OutPoint. And finally, from txid → block header (or at least timestamp & height).
I’ve wondered about whether this is a good way to go, having the transparent addresses as part of the argument to compact block retrieval, or whether the separate GetTaddressTxids call is sufficient. Doing it all in the scan would certainly be less of a hassle, but I kind of worry about that approach if a wallet has a large number of taddrs; one advantage of having the transparent transaction retrieval be separate is that one can partition the set of taddrs you’re querying with without having to re-download all the shielded compact blocks for each subset.
For an exchange wallet that has tens or hundreds of thousands of taddrs (which is a use case we’re talking about here), you probably don’t want to use either of these APIs; instead, you want to just download essentially the full blocks. But maybe that could be served via having the extra argument to GetBlockRange be able to represent either a set of taddrs that is used to filter vin and vout, or just “give me everything”. Along those same lines, for privacy-conscious individuals who are willing to compromise on bandwidth and performance, we should modify the CompactTx format so that it can also represent non-compact data, so that you don’t have to download the memos separately (and therefore don’t have to reveal which transactions are yours to the lightwalletd server.)
You could have both. My point is that currently, the API is heavily biased towards shielded sync and very inefficient when it comes to handling transparent addresses.
