Zecwallet's BlazeSync: Sync entire chain in < 60s!

[adityapk00]

tldr;

• Zecwallet Lite can now sync the entire blockchain - from the 1st sapling block - in < 60 seconds; but wait, there’s more
• Wallet funds are ready to be spent in seconds! No need to wait for sync to finish
• Requests the exact same data from LightwalletD, so no change in security/trust assumptions.

BlazeSync

Zecwallet is happy to introduce today our first Beta of BlazeSync - A new light client sync algorithm that is blazing fast. It can sync the entire zcash blockchain, from the first sapling block, in under a minute.

We’re making the beta release of the Zecwallet Lite CLI wallet and SDK today. Head over to our downloads page to try it out: Releases · adityapk00/zecwallet-light-cli · GitHub. Desktop and Mobile apps will be updated to use BlazeSync in the next 3 weeks.

As with all beta releases, if you run into bugs, please let us know

Why is it so fast?

• Does less work: BlazeSync scans the blockchain backwards - i.e., highest block first. This means the wallet sees spends before receives, which allows it to simply skip updating the witnesses for funds that have already been spent. This saves a lot of time!

• Out-of-order scanning: BlazeSync decouples the components of the sync from each other - Downloading blocks, doing trial decryptions, updating witnesses. Further, all of these pieces are done out-of-order, and synced together, which removes bottlenecks, allowing all available processing power to be used in parallel.

• Ready to spend funds instantly : By scanning blocks backwards (highest block first), Zecwallet can now update witnesses for sapling notes very fast, which means your wallet’s funds are ready to be spent almost instantly. You no longer have to wait for the sync to finish to start using your wallet and sending funds.

BlazeSync does all of this without changing any trust or security assumptions. It requests the exact same data from LightwalletD (albeit requests blocks in reverse order), it does all the checkpoint and block verification as before. Just much faster!

And because spending funds no longer depends on the sync finishing, the UX on the mobile and desktop apps improves significantly, allowing you to get on using your funds and sending shielded transactions just like you would with transparent addresses.

What’s the catch?

BlazeSync uses more RAM than the vanilla sync. The extra ram used is proportional to the number of blocks that are being synced in one go. It uses about ~0.7kB of RAM per block synced (i.e, 700MB of RAM to sync the entire blockchain).

When can I use it?

You can try out the beta release of our CLI wallet today. Over the next 3 weeks, we’ll work on ironing out the bugs and shipping it inside the desktop and mobile lite client apps.

What’s next?

BlazeSync was a big undertaking, and it cleans up a lot of the legacy infrastructure issues that Zecwallet Lite had, which was slowing down progress on new features. We have a lot more new and exciting features in the pipeline, and we’ll ship several of them in the next couple of months!

[covfefe]

Excellent update, can’t wait to try this out.

I remember thinking that you had rage quit over the initial ZOMG funding issues when you went quiet on the forums for a few days and was worried as you have brought so much to Zcash.

I’m glad you are around and building a team to carry out this important work, thanks for all you do.

[Shawn]

Wow, that’s pretty cool, making use of modern parallel CPU processing

[hanh]

@adityapk00

This means the wallet sees spends before receives

How do you recognize your spends before seeing the received note?

[holmesworcester]

This is awesome!

[Ziga]

Long syncing makes for a terrible UX. Really looking forward to seeing this in action. Great work!

[hanh]

Hi @adityapk00, you mention:

@adityapk00

BlazeSync - Improves sync speed by >5x, allowing the entire blockchain to be synced in <60 seconds

Could you specify how you achieved this result? So that we can understand how to reproduce it? for instance: machine configuration, wallet size, mainnet/testnet, number of received notes / height,…

I looked over the code and it seems to me that the sync has now a higher memory and cpu usage but is more parallelized. So I’m curious how it will work on older hardware.

Thanks

[adityapk00]

The benchmarks are running on my 2018 Mac Pro/ 8C / 16G ram. Mainnet wallet, has ~200 txns. Syncs the full blockchain in <60s.

Note that it is faster not just because of more parallelization - It also syncs backwards, allowing it to detect if notes have been spent as soon as it encounters them, which saves CPU on updating note/witnesses.

All the code (note the branch is blazesync) and binaries are available for you to run your own benchmarks. I’m also expecting to publish iPhone/modern Android benchmarks in ~2 weeks.

[hanh]

@adityapk00

All the code (note the branch is blazesync ) and binaries are available for you to run your own benchmarks. I’m also expecting to publish iPhone/modern Android benchmarks in ~2 weeks.

I tried but it panicked.

(main) Block:1287881 (type 'help') >> notes
{
  "unspent_notes": [],
  "pending_notes": [],
  "utxos": [],
  "pending_utxos": []
}
(main) Block:1287881 (type 'help') >> rescan
id: 1, in_progress: true
id: 1, blocks: 0%, decryptions: 0%, witnesses: 0%, tx_scan: 0%
id: 1, blocks: 0%, decryptions: 1%, witnesses: 0%, tx_scan: 0%
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', lib/src/blaze/block_witness_data.rs:781:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', lib/src/blaze/block_witness_data.rs:781:37
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', lib/src/blaze/block_witness_data.rs:781:37
thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'called `Option::unwrap()` on a `None` valuecalled `Option::unwrap()` on a `None` value', ', lib/src/blaze/block_witness_data.rslib/src/blaze/block_witness_data.rs::781781::3737

id: 1, blocks: 1%, decryptions: 2%, witnesses: 1%, tx_scan: 0%
id: 1, blocks: 1%, decryptions: 2%, witnesses: 1%, tx_scan: 1%
id: 1, blocks: 1%, decryptions: 3%, witnesses: 1%, tx_scan: 1%
...

@adityapk00

It also syncs backwards, allowing it to detect if notes have been spent as soon as it encounters them, which saves CPU on updating note/witnesses.

The code in is_nf_spent searches for the spend backwards from the latest height to the birth height. The advantages vs scanning forward are not obvious to me since the only difference is when the note is spent in a recent height vs earlier height. Not updating notes/witnesses is a good thing but it doesn’t seem related to the scan direction unless I am missing something here.

Also, does it mean that the entire blockchain scan range has to be kept in memory?
The blockchain will keep growing. Compact blocks are significantly smaller than real blocks but we really hope for more shielded transactions in the future. What are your plans when it is no longer feasible to hold the entire blockchain in memory?

Finally, I can’t see how you would detect a spend before receiving the note since the spend only has the nullifier (which is the output of a pseudo random function). Could you clarify your algorithm?

[adityapk00]

Thank you for trying it out. I think I found and fixed your issue earlier this week, so can you please try again and build from source?

git checkout blazesync && git pull
cargo build --release

If you run into further issues, can you please file a bug so I can keep track.

@hanh

the only difference is when the note is spent in a recent height vs earlier height

This is a big deal for performance. For most users, unspent notes are in recent blocks, and hitting those notes first and updating those witnesses first means they can be spent quickly, without waiting for all the trial decryptions (the slowest part of the sync) to finish.

@hanh

Not updating notes/witnesses is a good thing but it doesn’t seem related to the scan direction

Having the blocks downloaded backwards means that all the data needed to update the witnesses (The Nodes in the merkle tree, derived from the compact_output.cmu) is already available.So, as soon as a note is detected, it can be (1) Checked to see if it has been spent (2) If not spent, update its witness. In a forward sync neither (1) nor (2) are possible, since the blocks needed for them aren’t yet downloaded.

@hanh

Also, does it mean that the entire blockchain scan range has to be kept in memory?

It currently keeps it in memory, but note that it is not the whole blockchain - and not even the compact blocks. It only really needs the nullifier from the spends and the cmu from the outputs, so only ~64 bytes (plus some hashes)

Of course, this doesn’t have to be in memory. Storing it on a disk based key-value store will save not only memory, but will be faster too (The current search is O(n), disk-based key-value lookup will be closer to O(1)) There are lots of future optimizations here, so we won’t be limited by the growing blockchain.

@hanh

detect a spend before receiving the note

Perhaps I should say “detect spends as soon as we receive a note”. The original plan was to detect the spends by decrypting outputs using the wallet’s OVK, but I decided against it because there wasn’t a lot of performance gains there. So right now, it detects the spend as soon as it sees the note, so it can immediately decide whether to start updating the witnesses. I might go back to the ovk solution in the future, but that needs more work (and a LightwalletD API change)

[hanh]

@adityapk00

Thank you for trying it out. I think I found and fixed your issue earlier this week, so can you please try again and build from source?

git checkout blazesync && git pull
cargo build --release

This is exactly what I did on commit c80bb934bf9d1a1b7ffb5e0b4e2184578d81f873 which is the HEAD at this time. I didn’t use a pre-built binary.

@adityapk00

This is a big deal for performance. For most users, unspent notes are in recent blocks, and hitting those notes first and updating those witnesses first means they can be spent quickly, without waiting for all the trial decryptions (the slowest part of the sync) to finish.

I wrote a quick prototype to understand where the issues with sync are. GitHub - hhanh00/zcash-sync

It’s not optimized yet and is a straightforward implementation of a download and decrypt (parallel).
From sapling activation height to current height:

• download takes ~5.6s
• note decryption takes <6s

(amd 3900x)

It doesn’t update witnesses nor does it calculate the tree state. I think this is where we could win big.

To try it out, you need to create a .env file with IVK=zxview… then run the tests
cargo t --release -- --nocapture

Based on these preliminary numbers, download / decryption is so fast that it hardly matters which way you are going.

@adityapk00

So, as soon as a note is detected, it can be (1) Checked to see if it has been spent (2) If not spent, update its witness. In a forward sync neither (1) nor (2) are possible, since the blocks needed for them aren’t yet downloaded.

Right, and to “Check to see if it has been spent (2) If not spent, update its witness”, you need to go forward. It goes backwards to look for output notes then forward to look for spend and again goes back to update witnesses. All this stuff has to be holding the proper locks since it’s multi-threaded too.
In the end, I’m not sure it is faster…

@adityapk00

Of course, this doesn’t have to be in memory. Storing it on a disk based key-value store will save not only memory, but will be faster too (The current search is O(n), disk-based key-value lookup will be closer to O(1)) There are lots of future optimizations here, so we won’t be limited by the growing blockchain.

Not sure I agree with this assessment about disk being better than memory. Anyway, I feel like it shouldn’t need these optimizations in the first place. Your call.

@adityapk00

The original plan was to detect the spends by decrypting outputs using the wallet’s OVK, but I decided against it because there weren’t a lot of performance gains there.

Yeah, I think so too.

It’s great that we are in a position where we can experiment with different sync strategies!
Let’s see how low we can go

[adityapk00]

Can you please file a bug so I can try and get to the bottom of your problem?Are you connecting to a custom LightwalletD server? Is it a new wallet? existing wallet? Restored with seed?

[adityapk00]

I think you were using Viewkeys, right? I think I fixed your bug in Fix crash when there are no witnesses · adityapk00/zecwallet-light-cli@854ee7e · GitHub

Anyway, to your broader point: There are lots of optimizations possible here, and I’m sure we can easily get a 2x improvement from here as well.

[earthrise]

Awesome work! I’m glad we’re able to get these kinds of performance improvements with no loss in privacy. It sounds to me like this approach is even compatible with the “download all memos” defense for maximum privacy, at the cost of only having to download more data! Would you be interested in prototyping/benchmarking that, maybe as a setting for power users to turn on? My hypothesis is that the speed difference won’t really be noticeable, and the only downsides would be increased server bandwidth costs and costs for users on cellular data.

[adityapk00]

I think this is a good idea. I can add it as a setting, and we can then measure performance and usage. I will add it into the next release.

[earthrise]

Awesome I’m excited to try it out! Do you think the lightwalletd API will need any changes to support that better, like an API to download the memo ciphertext components? If the current API makes you query for each memo individually I can see that being a huge performance bottleneck.

[hanh]

I made a comparison of sync times between NW and ZW. Looking forward to updating it with BlazeSync.

Nighthawk vs ZecWallet Sync Battle

ZW: 55 mn
NH: 150 mn

[gmale]

@earthrise

Do you think the lightwalletd API will need any changes to support that better, like an API to download the memo ciphertext components?

Yes. We currently “get memos” by downloading the entire transaction.

When compact blocks are created, a bunch of things are stripped out. So that leaves two main options: 1) don’t strip the memo things (i.e. all “compact blocks” become a little less “compact”)
2) index those memo things that were stripped out and provide APIs to query them, directly

We’ll probably end up selecting 2 because I imagine that the bandwidth increase of always serving all memos to all clients will be prohibitive (on the order of gigabytes per user).

[gmale]

Like all good innovations, this has way more uses than the original problem.

Yesterday, I had a seed phrase where I forgot to write down its birthday height. I used BlazeSync to sync the entire chain and find my birthday height, immediately (the height of the oldest transaction).

Already, this saved me like 2-4h of work just yesterday. Soon, we won’t need birthdays anymore or even checkpoints.

I love it. @adityapk00 for president!!!

[adityapk00]

BlazeSync is now available across all Zecwallet Lite desktop and mobile apps. Download the desktop versions or update from the Google Play / iOS App stores or get the Android APK

BlazeSync is a new lightclient sync algorithm that syncs 10x faster. On recent CPUs and phones, the new Zecwallet Lite apps should be able to sync the entire blockchain in under a minute!

With this update, we now have the ability to build a lot of interesting new features and use cases into the Lightwallet, which should show up in Zecwallet Lite in the next few weeks.