Project Zephyr Update Jan'23

[little.slingshot]

Part 1 - Intro

In the last few months we have been working on making progress on Zephyr Web Wallet (think “MetaMask for Zcash”) project. Here we present some of the highlights of our work and technical investigations.

Our strategy for creation of Web wallet core code was to adopt the codebase of Aditya’s wonderful zecwallet-lite-cli codebase and port it so that it is able to run as WASM application in the browser. This WASM-compiled module in turn would serve as the bridge to communicate with the Zcash blockchain and perform low-level blockchain operations. This way it is possible to do separation of responsibilities and the Web wallet UI will be just a simple web-app wrapped as Chrome/Firefox extension.

At the core of a webwallet there is a blockchain client implemented as WASM-module and hence executable within browsers. Luckily there are toolchains (eg. compilers) which are able to compile Rust code into WASM. However, as we set out on our journey we have encountered a number of hurdles which we had to overcome.

Part 2 - Challenges

Some of those hurdles were related to difference in architectures between the native Rust app (eg. running on Linux) and an app running within browser WASM-environment. Some of the first challenges were related to limited networking stack on WASM (eg. lack of access to TCP-sockets) and hence RPC communication couldn’t happen directly to LightwalletD daemon. Another set of challenges came from the fact that some library dependencies were not compatible with the WASM environment.

Most of the challenges we had to overcome were related to WASM being a different execution environment vs native OS apps. Whilst WASM is definitely the way for the future, its ecosystem is still in it’s “adolescent” stage. There’s many promising ways to solve any issue, but not always it’s clear which one of them is the most reliable path.

Running tasks concurrently is not as easy in WASM. This is because standard OS-threads do not exist in WASM environment and hence core Rust concurrency primitives like std::thread::spawn() cannot be used. To find a workaround we tried different approaches, but couldn’t yet find a stable way to do this withing crashing the browser

After hitting some WASM-concurrency stability issues we decided to focus on single-threaded approach for the moment (albeit it makes block syncing and scanning take annoyingly long time).

Another discovery was related to memory limitations in the browsers. Actually there is no hard limits on max memory per browser tab (eg. Chrome tab), but what we’ve discovered is that wallet-syncing may take up to 4Gb of RAM when syncing/scanning blocks and that can make browsers behave unstable. And browser can simply terminate the tab or crash the process. We have achieved stable runs by ensuring that OS had enough of RAM so that swap-file wasn’t used (or was very small). Seems that presence of decently sized swap file would increase chance of browser killing the WASM tab

Here’s some examples of RAM usage spiking in the browser tab which executes WASM-syncing

and another one

and then memory returns to normal

Part 3 - Living with Sapling (for the moment)

Another challenge was presented by the fact that Zcash ecosystem advanced and NU5 upgrade went live last year. This upgrade introduced protocol which is quite different from the Sapling codebase which we originally forked from Aditya’s wonderful zecwallet-lite-cli. To deal with a lot of moving parts (wasm-stability, concurrencty stability, networking, etc), we decided to test our WASM-codebase against Sapling version of the blockchain.

Eventually we set up the zcashd-v5.0.0 synced up to the pre-NU5 block 1,624,964 (where NU5 is 1,687,104) and then we run the node with -noconnect flag which still allows JSON-RPC protocol, but does not try to sync up with the rest of the network. This way we were able to set up a Sapling-sandbox for testing stability against some fixed target. This is a bit of a “hack”, but it allowed us to deal with uncertainty and allowed establishing a stable baseline.

Part 4 - Establishing baseline

So currently we have established a baseline, a PoC that the code can be ported to WASM and perform scan, and sync with the blockchain and find transactions.

Previously the only other WASM-code which worked with Zcash blockchain was a legacy ~5 year old version of ZecWalletLite).

Below one can find example screenshot of successful scanning of the blockchain for a test transaction (from the wallet initialized with out test-seed). You can see that for example transaction df2c041ca... was indeed detected and it matches with the info reported by the latest ZecWalletLite…

And matching transaction

Part 5 - Next Steps

Our next set of priorities looks like this:
a) introduce some* concurrency (can be related to block verification or to block scanning)
b) merge upstream changes from Aditya’s wallet (probably pre NU5)
c) test more on pre-NU5 blockchain (eg. on a small network of 3 own nodes)
d) upgrade to NU5 (which can be done by merging NU5 commits from Aditya’s upstream)

*there are some ways to introduce concurrency with Rust/WASM which we are looking at.

Overall we’re very positive about reaching this milestone (baseline), as we have managed to get past the most drudging and uncertain work of figuring out whether it can work at all. (despite the fact that it took us longer than we expected to get to this point). And now we have more specific goals to move things ahead.

Thank you!

Here are links to previous two updates:

Project Zephyr Update March'22

@zooko

Project Zephyr Update

Original grant thread for reference: Zephyr - A Metamask-style browser extension for Zcash Hello Zcash Community, I am Little Slingshot from the Zephyr Wallet team. Earlier this year we were awarded a ZOMG for the development of a browser wallet for Zcash. We are slightly overdue with the update, but despite the delay there are a number of exciting discoveries to share with the community. This is my first post, so I have included a short brief about me.

Brief about me

Within the last 10 yea…

[GGuy]

Congrats! Wonderful to hear some progress is being made !

[dismad]

Excited for this, hope all is well!

[skyl]

Thanks for working on this. IMO, this is probably the most important initiative in Zcash.

Did you consider using zcash-sync as the starting point? Do you have any insight into why zcash-sync would be harder to target WASM than the zecwallet-lite code? I’m not positive, but I might bet that zcash-sync would use less memory. Has anyone done a comparison?

[hanh]

Unfortunately, I don’t think this would work either. The main issue lies in the performance needed to synchronize the zcash blockchain especially with all the spam. We can see that the current implementation struggles pre NU-5, the spam started post NU-5.
We could port zcash-sync to WASM. It shouldn’t be hard. It’s 100% rust. The main challenge is the usage of async and multi-threading. But I think it could be done.
However, I suspect (and without a prototype this is guesswork) that it won’t be enough.
Metamask and the other browser wallet you mentioned can rely on a server to do all the synchronization and indexation work. Effectively, the browser app is only a UX when it comes to account management. It signs the transactions locally and that’s it. Computationally speaking, that’s a few thousand integer operations.
Synchronization and proof generation should be in the order of billions.
The WASM virtual machine is not optimized for this kind of workload. CPU have special instructions that allow to quickly implement multi-precision integer operations (adc, mad, etc.) but they are out of reach when you use WASM. There is a push to include them but it is not a priority and no one knows if/when they will be available.
OK, that’s the bad news.
The good news is that it could be possible if the user accepts some performance degradation vs a native wallet.

1. Reimplement the low-level crypto primitives with the WASM vm in mind. There was a prize for it. Accelerating Elliptic Curve Operations and Finite Field Arithmetic (WASM) | ZPrize
   Note that doing this had a price tag of 750 000 USD by itself. Clearly it was a difficult task and there are other areas to optimize before we get a fast zcash wallet. Halo2 Acceleration | ZPrize comes to mind.
2. Investigate the usage of the GPU through the WebGPU API. Most desktops have a GPU nowadays. It is far from trivial too. It is basically the same work, but using GSL instead of WASM.
3. Compromise and decide that synchronization will be done remotely. The browser wallet sends the account viewing key and offloads the work to a server. Not ideal because the server decodes the transaction history but easier, though still very hard to scale to 1000 of keys.

[decentralistdan]

Zephyr - A Metamask-style browser extension for Zcash

@fireice_uk, @little.slingshot, & Zcash Community, the @ZcashGrants Committee has voted to terminate this grant due to inactivity and/or a lack of responsiveness. This grant will be removed from the old grants platform and you will need to submit any future ZCG grant requests via Submittable.