Warp - the next engine for ywallet

Zcash Apps Ywallet
1

I’m proud to announce the early access of the wallet engine that is going to replace zcash-sync in Ywallet.

This has improved synchronization and transaction building components, and a totally new/updated code base.

36 Likes
2

Demonstrating Unprecedented Synchronization Speed with ZEC Pages

I’ve made remarkable progress and am eager to demonstrate this advancement using ZEC Pages as my benchmark.

Why ZEC Pages?

I chose ZEC Pages as the ideal candidate for my synchronization benchmarks due to several key factors:

  1. Transparency: Its publicly available viewing key allows anyone to replicate and verify my results.
  2. Extensive History: With several years of data, it provides a comprehensive test case.
  3. High Volume: The number of transactions and output notes far exceeds that of a typical wallet, truly pushing the limits of wallet applications.

My Objective

In this video, I will demonstrate the complete synchronization of the ZEC Pages wallet from the Sapling activation height to the current block. I aim to complete this process in approximately 5 minutes, a feat previously considered unfeasible.

The Synchronization Process

Let’s begin by creating a new database and importing the ZEC Pages account using its viewing key. It’s worth noting that I’m working with a Sapling viewing key so that Orchard note decryption won’t be necessary.

Now, I’ll initiate the synchronization process:

  • Starting Point: Block 419201 (Sapling activation height + 1)
  • Synchronization Method: Progressing in chunks of 100,000 blocks

Observing the Results

  • 1-minute mark: I’ve reached block 1 million, showcasing impressive initial speed.
  • At the 2-minute mark, we enter what I call the “spam range.” This section of the blockchain was subject to a spam attack that significantly increased the data load. Typically, this section requires hours to process.

However, observe how my newly implemented spam filter efficiently manages this data. The filter swiftly discards transactions containing more than 32 outputs, effectively neutralizing the spam without compromising legitimate data.

  • 3-minute mark: I’ve successfully navigated through the spam period and resumed the previous synchronization speed.

Completion and Verification

Remarkably, the entire synchronization process concludes in approximately 5 minutes. I tested this Mac Air M1 with 8GB of RAM. My Lightwallet server operates on the LAN, while the WARP server, responsible for blockchain data, runs locally.

To verify the synchronization’s success:

  1. I examine the message table, confirming that it accurately reflects the ZEC Pages message board.
  2. I’ve successfully captured all transaction data and witnesses for unspent notes.

Implications and Future Directions

This demonstration highlights the exceptional efficiency of the Zcash warp synchronization method I’ve developed. The ability to process such a large volume of data, including navigating through spam-heavy sections, in mere minutes represents a significant leap forward for Zcash wallet technology.

I encourage the Zcash community to conduct similar benchmarks using various wallet algorithms. By sharing our results, we can collectively contribute to the ongoing improvement and optimization of the Zcash ecosystem.

Thank you for your attention. I’m excited to see how this advancement will shape the future of Zcash and cryptocurrency wallets, and I look forward to your feedback and questions.

20 Likes
3

Thank you so much @hanh for your work! It’s great to see that Warp has such a high efficiency and I will be very eager to see the results of the Zcash technical community tests and of course the subsequent implementation of Warp synchronization in the wallet (and probably in the reference SDK-version).

I watched the full video and I want to make a Russian version. I ask you to clarify the only point that I didn’t really understand, but it’s not technical, but interesting for me in terms of context that I probably missed:
“The only users known to be affected by these spam filters are actually employees of the ECC.”

If possible, give me clarification about this.

4 Likes
4

Cool, great work!

Using zecpages is a great idea and it’s a good way to enable a common benchmark.

I’m wondering how it compares to you previous sync engine. Are there any particular performance-increasing tricks?

Has this been changed from what Ywallet previously did?

4 Likes
5

Without getting into details because of privacy, the ECC used transactions that have many outputs for testing and other things. These would look like spam and get discarded.

Note that with a spam filter on the server side, we can design more advanced heuristics and improve the filtering without releasing new versions. In particular, real spam has fairly unique patterns because every tx was created with the same tool.

  1. I re-implemented the trial decryption
  2. Previously, spam note commitments needed to be sent, and they had to be hashed to update the user note witnesses. This is no longer the case. This saves both bandwidth and computation (Pedersen hashes are resource-intensive).
5 Likes
6

Perfect use case! Now every wallet can, in this type of test, be compared.

Couldn’t agree more :+1:

5 Likes
7

I have a Mac Air M1 with 16 GB of RAM, and also run a lightwalletd on my LAN, so once I am done with the stack of work on my plate for this week, I’ll do a similar test with the current zcash_client_backend stack using zec-sqlite-cli for comparison.

8 Likes
8

For the sake of comparison, can you report the total number of transactions you observed, and the wallet balances? I have:

Final database size: 795.36m, 3468 transactions, 3371 received notes.

Final balances:

  Sapling Spendable:   2.11483572 ZEC
  Orchard Spendable:   0.00000000 ZEC
         Unshielded:   0.00000000 ZEC

Also, can you report a benchmark without the spam filtering in place?

5 Likes
9

My suspicion (both for Warp and for my upcoming similar test on zec-sqlite-cli) is that the sync will fail during sandblasting, due to scanning 100,000 blocks at a time. I’ve had zec-sqlite-cli configured to scan 10,000 blocks at a time as a balance between speed outside sandblasting and progress within it, but I don’t recall how large a range I tested before settling on that. But I’d be interested to see the benchmark anyway, and happy to be wrong!

10

@nuttycom
I have fewer txs at height 2641167.

db size: 27m
txs: 3349
notes: 3368
and a balance of 2.09760006 ZEC

If you could share your sqlite db, it’d be good to check why it’s different.

my db

That was without spam filter. It took 1h26m under the same setup.
7 txs were wrongfully filtered. However, with a small adjustment to the filter, they would have been picked up.

@str4d
warp breaks the 100000 block range into smaller sections when there are too many notes.
We can see in the video that it goes down to a few thousand blocks during the heavy spam.
It’s 100k blocks or 1m outputs+actions, whatever comes first.

4 Likes
11

That’s the same thing we’re doing in the mobile SDKs. The current split there is 1000-block ranges outside sandblasting and 100-block ranges inside sandblasting, to balance throughput against progress updates with the available performance budgets and the way our sync loop is split between the Kotlin/Swift and Rust code, but I’d like to move towards constant-bandwidth syncing once we’ve done the refactoring to enable it.

4 Likes
12

Takes 8 minutes on my cpu with powersave mode on :+1:

3 Likes
13

FYI, you are using the remote warp server in your setup. It is synchronizing over the internet (except for the last 200k blocks)

2 Likes
14

And its still fast! I’ll will continue my tests :+1:

2 Likes
15

with export RUST_LOG=info, you will get info messages. It shows the sync progress.

2 Likes
16

The counts I posted above were without filtering; I didn’t encounter any problems. I should have posted our benchmark time as well, on my Ryzen 9 5950x (64G of RAM) with no filtering the times were as I posted on Discord a couple of weeks ago: Discord

scan: 1h 7m 7s
enhance: 1m 49s

The “scan” phase is the trial decryption; “enhance” is retrieval of and decryption of the full transactions to extract the memos.

Something that I’ve been considering for spam filtering is to use a function of the ZIP 317 fee, such that only transactions that pay at least some user-configurable percentage of the fee are trial-decrypted.

@hanh here’s a link to the data db: Dropbox

As a note, this was using a public lightwalletd, not a local one.

3 Likes
17

Was that using zec-sqlite-cli as-is? That would be using 10,000-block ranges, which I expect to work. What I would be surprised to see work is 100,000-block ranges.

2 Likes
18

Yes, that’s using zec-sqlite-cli unmodified (except for the open PRs.)

1 Like
19

Is the tool zec-sqlite-cli available to the public?

20

When I first wrote it back in 2020, it was public in a branch I was maintaining as a cargo example on zcash/librustzcash. We then moved it to its own repo to turn it into a tool for testing the zcash_client_sqlite Rust crate, and I kept that private because I did not want a CLI wallet that was not written with security in mind to be public. But we’ve made some improvements over time (though it is still not suitable for non-developers to use), and one of the things I’ll be doing soon is altering how it stores the seed phrase. Once that’s done I’d be happy for it to be public.

4 Likes