All ECC teams focused on wallet performance

Ecosystem Updates
1

Hi everyone,

As many of you are aware, there have been multiple issues contributing to a diminished ZEC user experience since June 17th. An increased transaction load, on-chain, has resulted in increased sync times for many everyday ZEC users’ mobile wallets. Ecosystem partners are also affected. Our current focus at ECC is to ensure that these issues are dealt with as soon as possible, and that Zcash users have a user experience they expect and deserve.

The issues mentioned are currently affecting Zcash wallets, miners, payment systems, custody providers, ATMs and more. With regards to the users of mobile Zcash wallets, some have been unable to access their funds. As mentioned, this does not mean that funds are lost, it means that some mobile wallets are having syncing issues during this time.

Fixing these issues is priority number 1 for ECC, even taking precedence over our long-term roadmap. Performance improvements will ensure that users enjoy fast and reliable services when using their ZEC. We have a cross-functional team — including engineering, partner engagement and support, product marketing, and comms — focused on solving user experience. Our work centers around five key areas:

  • Improving the sync performance in our mobile wallet SDK to ensure that everyday ZEC users see an improved experience.
  • User research to understand ZEC users / holders experience, and highlight what solutions are available to them.
  • User experience research to understand how future protocol improvements will make ZEC more accessible to everyday people. This research will focus on proposing improvements to ZEC UX. This includes, but is not limited to, fee changes, mobile wallet improvements, and product strategy for the coming ECC wallet.
  • Outreach to ecosystem partners to communicate our work around performance issues.
  • Prioritizing changes that improvise long-term network resilience. This ensures the Zcash network will perform optimally in the event of sudden changes to network activity (i.e., transaction volume).

We recently released zcashd 5.1.0 and 5.2.0, which have already solved some, but not all, of the issues. The team is currently working on improving sync performance for our mobile SDKs which will benefit wallets such as NightHawk, Edge, and Unstoppable.

During this time, I will be communicating with the community and our partners regarding updates and the status of our work.

We’ll continue to monitor the forums for any questions from you all, and will hope to respond as quickly as possible.

If you have any questions or concerns, please let us know in this thread. If you would like to reach me directly, please DM me on Twitter.

Thank you. We will have another update for you all in the coming days.

Ian

22 Likes
2

Thanks Ian for the proactive updates :slight_smile:

What would be really awesome would be some sort of message from ECC that says something like, “We know it’s been a bit rough. It’s due to X situation, and here are all the ways our brilliant engineers are gonna kick this problem’s butt :crazy_face:.”

I know there has been some messaging along these lines, but my above suggestion is partly inspired by a conversation I had with @skyl about a conversation he had with @str4d haha. Basically Skylar seemed blown away by the potential solutions to the current wallet problem that the folks at ECC were thinking about (at least, that was my impression), and it may go a long way to putting some of the ideas out there to assuage the fears of the Zcash community.

Although Ywallet has been working great for me, I think additional public facing discussions about the current situation would be very helpful for morale and general confidence that this is a short term growing pain that has a good chance of being widely resolved in the near to mid term. This would also especially be helpful for those of us spreading the good word of Zcash to have more things to point to that the current situation is temporary.

Of course, it’s easy to suggest ways for people to do more work when I’m not the one doing it, and enough info may have been put out there already that covers my suggestion but I just haven’t read/seen. So take my suggestion with a grain of salt.

Keep up the good work Ian and all at ECC.

9 Likes
3

Thanks David! We appreciate the kind words! All the functional areas within ECC are focusing on this problem and working as hard as we can to continue to make improvements.

Our tactical next step to alleviate the impact of the increased shielded load we’re seeing is to introduce a faster sync option for our mobile SDKs. These are in use by NightHawk, Unstoppable, and Edge. A faster sync has always been on our roadmap, but we moved it up in priority to help mobile wallets scale better with the current load. This seems to be the number one pain point in the ecosystem right now and is our top priority in Engineering.

We made some pretty significant improvements with zcashd releases 5.1.0 and 5.2.0 and we’re currently surveying the partner and user ecosystem to identify other opportunities for tactical improvements. We’ve discussed a number of technical options beyond what we’ve already released and are combining that with user research and UX analysis to make sure we don’t create a bigger problem that we address. :slightly_smiling_face:

Thanks again for all your support!

5 Likes
4

Does this mean there will be three separate implementations of an accelerated batch sync? YWallet’s Warp Sync (@hanh) , Zec Wallet’s BlazeSync (@adityapk00) , and an now ECC implementation? If so, why? And what does this say for @hanh’s sync grant proposal?

FWIW NightHawk, Edge, and Unstoppable have always been virtually unusable due to tediously slow sync times.

5

Sounds great! I’m not a very technical person when it comes to this stuff, but how does the syncing algo ECC is working on compare to warp sync that is used by Ywallet?

And a follow up, would the warp sync integration proposed by @hanh in his recent Ywallet grant proposal (forgive me I don’t have the link here) aid ECC’s efforts at faster syncing?

Thanks I’m advance!

3 Likes
6

Why not hire more people?

2 Likes
7

We recognized SDK sync time was subpar even prior to the increased load which is why we had an improved sync in the roadmap. (as mentioned above we moved it up in priority)

You’re correct, there will be three different implementations of improved sync algorithms which is fine IMO. I’m not a first-hand user of Ywallet or ZecWallet Lite beyond limited testing but it is my understanding that each performs well in different circumstances depending on the shape of a given transaction load.

3 Likes
8

We looked at warp sync in details when we began working on an improved sync in earnest which was shortly after we packaged zcashd 5.2.0 for release. We determined then that the shortest path to an improved sync would be our own phased implementation to improve both the SDKs and the underlying librustzcash layer.

3 Likes
9

Thanks Steven, good luck!

1 Like
10

I’m definetly no expert but I think ECCs version is called DAGSync? Benchmarking is yet to be done for DAGSync AFAIK.

My understanding is at a high level DAGSync prioritises using your previously known spendable funds (i.e. notes) and trying to invalidate them (i.e. nullifiers). If no nullifier is found for a given note a few extra steps happen and those portion of funds are spendable.

Imagine that I’m in line at a coffee shop, I see they accept ZEC, and I pull out my wallet that I haven’t used in a month. I don’t actually care about being able to spend my entire balance. What matters to me, in that moment, is that I can spend enough funds to pay for a coffee.

WarpSync’s goal is to make all your funds (i.e. notes) spendable ASAP.

A typical mobile wallet user might benefit from DAGSync while an exchange or service or store might benefit more from WarpSync.

1 Like
11

We definitively need to do some profiling when DAGsync is ready. I think Warpsync will perform well in this scenario too.

Emphasis mine

1 Like
12

From the previous benchmarks you’ve provided it’s hard to disagree. I suspect both methods would be fast in the given example. We will soon find out I guess :slightly_smiling_face:.

13

Not solely that; what it does is find the shortest path from “current wallet state” to “spendable wallet state”. That may be by finding existing notes that are spent, and jumping ahead through the DAG, before updating any witnesses; or it may be by updating the witnesses for those existing notes, and then speculatively attempting to spend them (which could result in a double-spend). Both steps need to happen to establish for certain that a note is spendable, but they can happen independently, and give potentially tunable UX trade-offs and benefits.

Yeah, in the steady state mode (syncing a wallet that was last up-to-date in the last few days or weeks) I don’t think there will be too much difference (DAGSync includes improvements to the way witnesses are managed and updated that may be faster than what Warpsync currently does, but those can be pretty easily adapted). The biggest benefit that DAGSync provides (due to its fundamentally different design) is in recovery from backup, and the design should also help improve performance in the scenario where you have the same mnemonic phrase imported into two different wallets (so notes in a wallet can be spent while the wallet is offline).

3 Likes
14

Do you mean the case where you recover from backup seed and want your wallet to look for a recent note that you can spend right away?

1 Like
15

I had a question about how Warp Sync approaches the “make your notes spendable right away” issue. In order to spend a shielded note, you have to choose an anchor in the note commitment tree, and preferably your choice of anchor should be recent enough (say within the last 10 blocks) to reveal no information about the age of the note you’re spending.

The intention that I have is that, in conjunction with the DAG Sync changes, that the lightwalletd protocol will be modified to provide a cache of high-level nodes in the note commitment tree that can be used to produce current anchors for old notes, such that only ~2^16 notes in the vicinity of the note that you want to spend (along with the most recent 2^16 notes) will need to be added to subtrees of the note commitment tree in order to make it possible to spend your note. How does Warp Sync approach this today?

1 Like
16

Hi Chris, Ywallet current approach to “make your notes spendable right away” is to skip trial decryption for transactions that would have cost too much to spend under a progressive fee structure.
The idea is that I know people who send me zec would not pay more than X in fees.
I suggested pushing a fee filter to lightwalletd so that the wallet would not have to download epk & ciphertext but at the moment, the filter is only client-side.

Lightwallets must balance 3 resources: CPU, Memory, and Network bandwidth.
Of all three, IMO, Network is the biggest issue currently. In the last 2 months alone, the blockchain grew from ~1GB to ~5GB and 4GB is too expensive on most mobile phone data plans.
Next comes memory. Many phones have <3 GB of RAM. Therefore, caching is limited for them. Since I think redownloading and caching aren’t practical for many users, I chose to make WarpSync strictly streaming.

Finally, it’s CPU. Besides trial decryption, which we can skip for 99% of the notes, the heaviest calculation is witness maintenance. For this, roughly speaking, WarpSync combines all the note commitment updates in a pass. So that whether you have 100 notes or 0, the CPU cost is about the same (= the cost to update the tree_state). Because all the hashes needed for the updated witnesses were calculated at one point during the tree update. We just need to pick them up.

9 Likes
17

Hi everyone,

As mentioned previously, fixing performance issues is ECC’s number 1 priority at the moment, and we have a cross-departmental team — including engineering, partner engagement and support, product marketing, and comms — focused on it. Here’s an update.

  • 5.1.0 and 5.2.0 have largely addressed performance issues for full nodes
  • End user wallet sync remains an issue
  • Priority remains faster sync for mobile SDKs
  • Team is also researching possible fee change mechanisms
  • Team is also evaluating longer-term scalability options consistent with our scalability road map

As mentioned before, feel free to DM me, or comment below, with any questions.

Ian

7 Likes
18

Are you in communications with @hanh he has demonstrated himself as a solutions person.

YWallet has been a life-vest for hundreds of Zcash wallet end-users with sync trapped ZEC funds

2 Likes
19

Yes, we’ve both spoken with Hanh (async) and reviewed the faster sync algorithm he uses in Ywallet.

3 Likes
20

Hi @steven-ecc , you may have been talking with someone who impersonated me because the only communication I remember is in this thread.

2 Likes