At Zcon4 in Barcelona, @paullinator asked me what it would take to get detection keys to improve the user experience, that the UX benefit would far outweigh the small privacy tradeoff.
Detection keys would allow the light wallet infrastructure more quickly determine which transactions are relevant to a given user.
Paul reached back out last week and asked again. We were having some internal conversations about options and I’m dropping some of that information here so that its all open and all of us can engage directly.
A couple estimates from the team came back at “months” of work for DKs with a possible network upgrade, new receiver and possibly a new pool. Even then, scanning for old notes would take time.
Alternative approaches might be liberated payments (payments through an encrypted messaging channel) or a Detection Server. Some of @earthrise’s thoughts were captured here. Additionally, the team posed the question as to whether background sync will be sufficient.
In my opinion, I don’t think relying on background sync is sufficient and even the perception of long sync times is UX friction.
As Paul put it to me, “This is SOOOO valuable for usability… I can’t think of another protocol change of bigger importance.”
So the idea is that the wallet would have some way of receiving “notifications” of incoming payments without having to initiate (open) the wallet to see transactions ? Or would it just make syncing faster?
I think the idea is interesting, but I’m concerned about this part:
The whole pool experience/anon set is already confusing for users, I would try to avoid that if possible.
I would be interested in seeing what other UX ideas are under consideration to understand if it’s worth bumping other potential work for this item.
Instead of discussing possible features and tradeoffs here, I think that we should frame this discussion around the question of who are the users we’re attempting to serve. This has been the thing that’s been missing from our product roadmap basically as long as I’ve been involved with the Zcash project; we haven’t ever really articulated who we’re trying to serve or why those users are using Zcash in the first place. I don’t think that the discussion around which technical approach is best will resolve the current friction without properly identifying those users, because friction is contextual.
There’s no question that there’s a degree of friction associated with wallet sync at present, and we have several approaches to dealing with that friction. Detection keys are an interesting option; liberated payments are an interesting option, and there are several things that we can do just at the wallet and lightwalletd level to improve things as well. But in my personal opinion, the options that require more significant effort probably don’t provide all that much return on investment unless it’s clear to us that there’s a sizeable market that (a) we’re currently unable to serve because of this friction, and (b) will actually open up to Zcash if this friction is removed.
I think that while we’re in the process of answering the above questions, we should pick off some low-hanging fruit to improve user experience: we need to implement zcash_client_backend: Download treestates to reduce time to spendability by 75% · Issue #982 · zcash/librustzcash · GitHub in the ECC wallet SDKs (this takes inspiration from the Warp Sync approach) to make it possible to defer the scanning of note commitment subtree ranges prior to our received notes, while still making it possible to always use recent anchors when constructing transactions, in order to better preserve transaction indistinguishability. At the same time as we’re doing this work on the scanning and prioritization system, we should implement background sync at the application layer; with these two improvements in place, we should then reassess and see what makes the most sense as a next step. Detection keys will almost certainly make sense to implement at some point, but IMO Zcash needs to find its market, and we shouldn’t defer the small improvements that are available to us to implement while we’re doing so.
From my POV there are two user groups that stand out as particularly promising ground for Zcash adoption. The first group is people in developing countries with unstable currencies who want to both hedge against inflation and receive remittances, and that are also afraid of their government confiscating their savings. Classic examples are people in Venezuela, Argentina and Nigeria, but I think there also are other countries which fit this bill that are rarely brought up, like Uzbekistan.
The second group is people in developed countries who want to buy/sell illegal or taboo goods and services, like drugs or sex work. Because physical cash would be very difficult to beat in physical commerce, I think it makes sense to narrow this down to online services, so basically onlyfans and maybe some online gambling.
For the first group the friction of long sync times could be exacerbated by slow and unreliable internet, but on the other hand these transactions would probably occur regularly and not that often, which could reduce the need for fast sync.
For the second group I think that a majority of these transactions are impulse purchases, and long sync times could thus be a dealbreaker in some cases.
@Milton You are missing the largest group in developed economies. The people who want privacy in the same way that a bank account works when you write a check or use cash. Transactions to the outside world are all private; but there is some limited loss of privacy to remain compliant. I am willing to give up some very limited privacy if a court order were to force me to turn over a private transaction statement that may offer more details about my transactions. So, let’s call this group the privacy-lite group. I believe 90%-95% of the developed world and much of the developing world would be able to accept this level of privacy; especially if it means its easy to interconnect with the rest of the world.
I am convinced everything will be tokenized. All money, all stocks, all bonds, and everything else of value. So there is a massive benefit to keep the transaction histories private. No one wants the public to view their transactions, including corporations who want to conceal their strategic moves from competitors, and so many other legitimate reasons for privacy. When everything becomes digitized/ tokenized, it will be much more important than today when very little is done on the blockchain.
Now as it relates to your group 1 and group 2, as I have stated so often. ZEC wont work as it exists today, and I dont believe will ever work as its just simply too volatile, has no trading depth, and isn’t easily convertible back into fiat. The belief that ZEC is a fiat alternative has caused us to waste so much money, we need to put that behind us and move forward. Now maybe a private fully collateralized USDz stablecoin (or any collateral backed coin) could work for your use cases. But ZEC as a fiat alternative is DOA in my opinion (and its proven to be true so far). Just like our own orgs need to sell ZEC for fiat and for so called “hedges”, the people you think will be wanting ZEC wont be able to use it for the same reasons our orgs cant hold ZEC. I always open to changing based on empirical results and real life use cases; but, from everything I have seen, our best hope is applying the tech to ZSAs/Stablecoins as fast as possible, and working on an opt-in privacy-lite version for the developed world that does not affect the hard core ZEC maximalists.
I am hoping the vision can be more aligned with below.
Zcash is a blockchain technology designed for private peer to peer money transfers; including, collateral backed stablecoins, and cryptocurrencies.
ZEC is the cryptocurrency powering the Zcash blockchain.
I believe if the mission and funding is redirected in a major way to accomplishing the above, Zcash can carve out a very substantial niche in the global movement of money.
Detection key services are interesting. Also they could be source of revenue provided that they can’t be OFAC’d. Cloud services are expensive, energy costs will continue to rise, we need to figure out how to make LWD and other infrastructure profitable.
I feel kind of envious / sad (for myself) when I hear that a fellow developer says “I could sync my wallet in 2 nanoseconds with my 4TB broadband and my 300 core dedicated ZEC machine” Most people barely can affort 10 or 100Mbit at best with a home computer or mid-tier phone. So being able to outsource this would be good, but those who use it should pay something to the server.
I could think of a service that can be completely on-chain or mixed.
User with spendable ZEC
User sends ZEC w/ shielded memo to server and detection key, server can take the job and reply with a shielded memo with a session token or refuse and return funds minus fees.
user with no spendable ZEC
User needs to restore a wallet and does not have spendable ZEC. User can engage with a web app to pay for the sync service the same way proton mail can be payed with fiat or other crypto, service returns needed information, even wallet app of choice deeplink/universal link to start syncing through the detection service.
Offtopic,
I would add that this option is totally marginal. Purchasing prohibited things within a authoritarian or surveillance regimes with a private money application that would prove your felony if keys are obtained is not the most clever thing to do unless cash is banned in such places.
We should focus on private p2p transactions of any nature, make them super fast and easy and forget about this “illegal good barting” scenarios that are more proper of a bad motion picture than real use cases. People needing those scenarios will figure it out there’s no need for us to market these use cases.
My opinion regarding the value of detection keys comes from first hand experience dealing with privacy coins and user feedback and support tickets at Edge.
Edge was the first multi asset app to support Monero and we did so using view keys to allow instant sync. Yes this comes at a privacy compromise, and I even did a podcast with @NaomiBrockwell explaining that issue.
Nevertheless we received very positive feedback that Edge finally delivered a “usable” Monero experience. There’s no longer the tradeoff of using a privacy protocol but having to wait for sync times.
Then we integrate Zcash after years of working with ECC on the SDK. Even before the spam attack, we are inundated with complaints of Zcash syncing. Even when it took “just” 2 minutes to sync a days worth of blocks, people were confused as to why they couldn’t spend or see funds after opening their wallet. Waiting for sync is just not an acceptable UX to on board the next gen of crypto users. At least not a sync that is dependent on the time since last opening of your wallet.
Sure we can get the few privacy enthusiasts and dark market players, but that’s how you stay in the cross hairs of govt. You want to be resistant to govt, you need to be used by everyone, not just those that care ideologically. So you have to make the privacy invisible. Even within the privacy community, I know many people that have opted to NOT use Zcash because of the sync times. I don’t think I used Zcash even once at Zcon. I paid people back for dinner using BTC.
Now let’s talk Detection Keys. They are an AMAZING development that delivers both a better UX and without the privacy compromise of view keys. User wallets derive a Detection Key from the wallet’s private key and share it with a server. The server can then determine which transactions belong to the wallet without knowing the addresses or amounts in the transactions.
Contrary to what Josh said, Detection Keys should have NO privacy compromise over using a lightwalletd server. This comes directly from Zooko. While I don’t know the math behind the protocol, numerous talks with @zooko and @nathan-at-least have me convinced of its feasibility and value.
This is easily the holy grail of UX for a privacy coin. We’ve never had this before and offering such an option would be ground breaking and make privacy a no compromise option.
They are possibly a great revenue source. Operating a detection key server will likely cost around $0.10 per key per month based on the understood cost of a Monero LWS server. Charging users $0.25 per month would be incredibly reasonable and we’d even consider offering it for free to users in Edge via a service provider.
Detection keys drastically improve the UX for both groups. Even if you use your wallet frequently, realize that current syncing still requires quite a measurable amount of cpu, bandwidth, and storage which is a premium in developing countries. I couldn’t possibly recommend someone to use Zcash on a midgrade phone more than 3 years old.
Remember than WhatsApp gained HUGE popularity in LATAM because its data use is free to users.
I would disagree on burning more resources on “optimizing” sync. Any amount of sync that is dependent on blocks will always be too slow for regular usage. Even in the land of bitcoin, without the complexity, storage requirements, or cpu requirements of shielded transactions, we’ve largely stopped using wallets that require per block syncing.
A huge amount of resources were used for SDK 2.0 sync optimizations and it still not a mass adoptable solution. Bite the bullet and get it done right.
A huge amount of resources were used for SDK 2.0 sync optimizations and it still not a mass adoptable solution. Bite the bullet and get it done right.
My point is that the SDK 2.0 sync improvements are not yet complete. What has been released thus far is the MVP of out-of-order sync.
Nonetheless, I take your point that detection keys can provide a superior UX, and you’re totally correct that the current lightwalletd usage pattern exposes more user information to the lightwalletd server than detection servers would be privy to, so it’s a substantial improvement in multiple ways. However, that will take a lot of development time, so we need to be confident of the target market. We’ve tried “build it and they will come” an awful lot; it’s not at all clear that sync speed is the missing piece for larger adoption.
At current transaction rate, it will take 100+ years before we get the same workload than the spam.
In other words, the spam blocks contain 100s of outputs and the current blocks only a few.
Any available sync algo (including sdk before SbS) work with the current rate.
I would say that the main focus should be to increase usage (real not spam).
Another issue is that users rescan when there is any problem. And because by default the rescan starts from sapling activation, they have to process the spam.
Monero has encoded the birth height in their seed phrase. I think it was a great idea.
Oh, one more thing here: there are two slightly overlapping concerns, one is time to detection of one of the wallet’s notes, and the other is time to spendability of that note, assuming that we don’t want to force the user to use old anchors when constructing their proofs, because that reveals information about their wallet’s sync state not just to the lightwalletd server but on-chain to everyone.
Detection keys help with time-to-detection, and note commitment tree sharding as implemented in SDK 2.0 helps with time-to-spendability. Ultimately the ideal solution will probably involve both, unless the community decides to move wholesale to a liberated-payments model; in the liberated-payments scenario, detection time is also effectively zero but you still have to do note commitment tree maintenance, which sharding helps with.
This set of tradeoffs was part of what informed the design of SbS; sharding the note commitment tree and making subtree roots available from the lightwalletd server helps in every one of these futures. Right now we shard the note commitment tree into 2^16 note batches, but it is also possible to perform more caching on the lightwalletd server (say at 2^12 boundaries) and further reduce the amount of computation needed to make notes spendable with current anchors.
Also, with respect to the issue surrounding the choice of anchors, I don’t fault @hanh at all here for the choices that he made for YWallet with respect to using older anchors and spam filtering; both those choices are totally defensible for some set of users. But I think that the ecosystem also needs wallets that don’t make these same compromises.
Another issue is that users rescan when there is any problem. And because by default the rescan starts from sapling activation, they have to process the spam.
This is a really unfortunate situation, and basically we need for all wallets to be more robust so that there’s never a need to rescan. At present the Zashi wallet doesn’t even expose any rescan functionality, and we always use the wallet birthday when scanning.
How do the bridges work, and is there a spec for this API? The subtree sharding is effectively similar to using bridges; in fact, the bridgetree crate that I wrote was built with this kind of approach in mind, but we then later decided to prefer to use a fixed granularity so that the precomputed roots could serve everyone.
There is no public / spec API because this is not a released product. Essentially the idea is that there are 2 servers: LWD and a bridge server.
sync works normally, but also requests bridges from A->B when it sees that there are no new notes in that range. It avoids having to calculate hashes in most of the cases since <0.001 % of the blocks have a note from the wallet inside.
There is also another nifty optimization. The wallet uses the memo of the change output in order to store some state it can use for recovery. I have a video demo somewhere if you are interested.
Using memos to help record state information also makes sense; we’ve discussed that as a strategy, and also the strategy of intentionally choosing which notes to spend based upon how much of the wallet’s history can be discovered using the techniques described in DAGSync: Graph-aware Zcash wallets - HackMD
Ultimately I think that this is all a process of convergent evolution, and the more that we’re able to create modular and reusable pieces to make these strategies usable by all wallets, the better off the ecosystem will be.
Most people barely can affort 10 or 100Mbit at best with a home computer or mid-tier phone. So being able to outsource this would be good, but those who use it should pay something to the server.
there’s no need for us to market these use cases.