Zecwallet Lite, the light client mobile and desktop apps for Zcash have found some success within the Zcash community. The mobile and desktop apps are used by approximately ~7500 people today, but I think we can do much better. For background, Zecwallet has only 1 person working on it (me) part time, which is clearly not enough to grow usage and research protocol improvements. This grant proposes to expand the Zecwallet team to 3 people, including 1 full time researcher working for 6 months to ship improvements to Zecwallet and the Lite client protocol.
Motivation and overview
While Lite wallets are widely used by people already familiar with Zcash and its pros and cons, it can be quite confusing to new users or users who are familiar with Cryptocurrencies in general, but are new to Zcash. I outlined several of the issues in these two posts below. Please read these two posts first for a background on the overall motivation behind this grant.
The summary is that first time use of Zcash and Zecwallet in particular can be quite confusing, and is causing a significant drop-off in user acquisition. This is bad because these are users that are interested and motivated to try out Zecwallet/Zcash, but aren’t able to. This grant aims to research some of these problem areas, propose concrete improvements to the wallets and the Lite client protocol, and ship these as improvements in the Zecwallet Lite desktop and mobile wallets with a goal of improving usage, adoption and understandability of Zecwallet.
We will evaluate success/failure of this project based on improved usage of Zecwallet and shielded ZEC by new and existing users.
Do execute this work, I’m proposing to expand the Zecwallet team to 3 people (1 Devops/Testing/Infra and 2 Protocol Research/UX/Engineering)
This is a very large grant, so I’m splitting it up into two main parts.
1. Devops and Code improvements
While this is not directly related to improving Zecwallet UX, I think it is important to do this work concurrently. Zecwallet started as a side project, and grew from a small UI as it added features incrementally. Zecwallet was my first UI project, and consequently, at the beginning, I didn’t really know what I was doing from a devops and infra perspective. In hindsight, I made several sub-optimal decisions (and some outright bad ones). Zecwallet has accumulated a lot of technical debt which is making it (1) slow to build the next generation of features and (2) On board new developers. Doing this work will increase the ability of the Protocol Research team to quickly experiment with ideas and new UX, research usage improvements and deploy wallet code that is maintainable in the long run.
The goal of hiring one devops/engineering/testing person is to
Free up my time to work on protocol R&D and
Hire someone that knows what they are doing to fix my mistakes, pay down technical debt, and make it easier to build on the code in the future.
A detailed list of the work items in this area are listed below, but the highlights are:
Improve build quality and deployment ease to allow faster iteration
Improve testing and performance analysis, allowing easier code maintainability
Fix technical debt to reduce security risks
2. Lite client protocol and development.
Zecwallet has previously conducted a lot of UX research and tried to identify what it would take to improve adoption of Zecwallet and Zcash under a variety of scenarios. Please see the linked posts above, as they contain a lot of details and the motivation for these work items. While there are several teams in the Zcash ecosystem working on improving privacy, Zecwallet would like to focus on improving usability and adoption, while pulling in all the great work on privacy being done by other teams.
Several of the work items below are somewhat speculative in nature, and don’t have a 100% defined specification or even a definition of what exactly will be implemented. Several of these also are dependent on conducting UX research first. I list them here because these are promising directions for Zecwallet research to take, and I firmly believe that some of these will lead to significant improvement for Zecwallet and Zcash users if implemented.
However, I don’t expect all of the items in this list to be implemented successfully. Some of these might not even be feasible and have to be dropped after some initial research. I expect 50% of these to be implemented in some form, 25% of these to need significant rework before they are shipped, and another 25% to not be feasible and have to be dropped.
For the Devops portion of it, the execution risk is minimal. The work needed in this area is pretty well understood, ans should be doable without too much risk to timelines or budget. One issue might be my ability to hire someone for this role.
For the Lite client Protocol portion, there is considerable execution risk. As I outlined above, my goal is to target 50% of the items below to be researched, built and deployed at the end of this grant. Several of the work items need more investigation, understanding of tradeoffs, several prototypes and User research. Privacy implication of the work will need to be understood. Engineering feasibility is another area that will need work. A few of the items will need extensive research and reworking, but I think I’ve added enough flexibility in the schedule and work items to make it work. It is hard to outline exactly what will be built and shipped because of the very research-y nature of this work, but I ask the ZOMG and the broader Zcash community to evaluate the success and failure of this grant based on results (i.e., Better UX and adoption for Zcash and Zecwallet) instead of the amount/type/specifics or work being done.
This grant focuses extensively on usability and adoption of Zcash. There are obvious engineering, latency, bandwidth, privacy etc… implications to several of the work items here. Some of the improvements made to help adoption or usability might hurt privacy or bandwidth/network usage. Privacy improvements incorporated from other teams work might hinder adoption. One of the goals for the Research done as a part of this grant is to come up with a framework to evaluate these tradeoffs and make sure that users understand them. While individual research items might improve or hurt adoption/privacy/latency/bandwidth, I think at the end of the grant, when we look at them together, we should have a goal of overall improvement along all dimensions.
Zecwallet currently has ~7500 users, as estimated by the bandwidth used by the lightwallet server. Our goal should be do double this number by the end of this grant.
Now, I have no idea if this is actually possible, but since this grant is focused on improving UX and adoption, this seems like a reasonable goal. Another thing to note is that measuring usage accurately is not quite possible since Zecwallet doesn’t do any logging or collect any type of user data, so we’ll have to estimate usage via downloads, blocks served etc… This should be fine, since this is what we’re doing right now anyway.
Tasks and schedule
(Please see below for details)
Budget and justification
Zecwallet will hire 2 additional people for 6 months. A devops/test/infra person (part time) and another fulltime Protocol/Engineering researcher. I will also work on the Protocol Research piece part time.
| Fulltime / | Weeks | Calendar| Rate | Total |Part-time | worked | weeks |Hourly| Devops/Testing/Infra (TBH) Part time | 12 | 26 |150 | 72,000 Protocol Researcher 1 (me) Part time | 10 | 26 |187.5 | 75,000 Protocol Researcher 2 (TBH) Full time | 26 | 26 |187.5 | 195,000 Total | 342,000
The hourly rate includes all expenses, like software licenses, server costs, hardware/laptops etc…
This section of the work is to hire one additional devops and infra engineer, who can:
Clean up the code and pay down a lot of the technical debt
Improve build automation and automate deployments and monitoring
Add automated testing and benchmarking to the UI and the SDK.
Some of the important things in this area are listed below. For a few of these items, ECC already has some code (for eg., DarksideWallet for Lite testing), so we’ll also pull in and integrate these into our infrastructure. Some of the items below are somewhat uncertain and they need more investigation, so at the end of the project, some of these items might change or we might implement a different solution. We might also realise that some of the items are not possible to implement, in which case we might have to look for alternatives.
1. Switch from Flow to Typescript in the UI
2. Use Electron-builder for UI builds
The current process for building Zecwallet and Zecwallet Lite releases is very cumbersome, manual, and error prone. Using the electron-builder will allow us to at last properly use the Github Actions, to both build and sign the releases (including MacOS notarizing and Windows .msi signing), allowing us to release much more frequently.
3. Upgrade Neon to use the Electron <-> Native bridge
The current Neon infra is deprecated, because it uses context-free Nodejs modules, and Node has upgraded to use context-aware modules. The current Neon macros are also very brittle, passing everything as JSON strings. They allow multiple bugs and security issues while crossing the Electron → Rust boundary. Use the
serde library and proper types to make the SDK use more robust.
4. zcashd builds & automated tests.
Zcashd builds are very manual right now, and don’t come with automatic testing. Use Github Actions to create “official” zcashd builds for embedding into Zecwallet, allowing a much less manual and error prone process to be deprecated. This will also allow Zecwallet to publish zcashd builds for MacOS, Windows and Linux for users that want to run a full node.
5. zcashd integration tests for Zecwallet Fullnode
Automate integration tests when being used as embedded zcashd, catching RPC change-caused bugs automatically. Zecwallet Fullnode uses the RPCs extensively, so changes to the interface and behavior can cause breaking changes in the wallet UI. These are manually detected now which has caused several of them to slip past in previous releases.
6. zecwallet SDK benchmarks
Zecwallet has suffered multiple performance regressions in the past because of a lack of performance regressions testing. Especially for constructing transactions and parsing Compact Blocks. Write benchmark and performance tests, and run them as a part of the build.
7. Automate Zecwallet Lite SDK integration tests
Zecwallet SDK has unit tests, but no integration tests. Especially the API / RPC tests, that test the use of the SDK from inside the wallet, catching bugs automatically.
8. UI test automation
Zecwallet UI (for both Fullnode and Lightclient) doesn’t have any UI tests. A lot of previous bugs have been caused by a lack of UI testing, which should really be automated, to catch issues when making changes in the code.
9. Automate builds for Paper wallet
The Paper wallet is entirely manual build process today, which is super error prone, and causes new releases not be released very often. This has caused security issues in the past. Automate the build and release process for the ZecPaperWallet to bring it in line with the Zecwallet UIs, allowing for regular releases
10. Zecwallet Mobile Automated Tests
The Zecwallet mobile apps don’t have any unit tests. We need at least smoke tests and a simple end-to-end test to ensure that when we update dependencies etc…, we don’t end up with bugs, which causes App store rejections. This has caused multiple App store rejections in the past which adds a lot of delays and back-and-forth with Apple.
11. Automate zcashd <-> LightwalletD integration and benchmarks
Using zcashd and LightwalletD is very error prone today because of a lack of any integration or performance tests. ECC has many tests here, that we should really integrate that will allow smooth upgrades of zcashd, which have to be done every couple of months. This will reduce downtime and unnecessary errors with Light clients
12. Automate LightwalletD performance monitoring
Current LightwalletD logging and performance analysis is very manual. We should ingest the logs into Prometheus/Elastic Search and automate monitoring with downtime alerting. This has been lacking, which has caused several outages to be undetected for hours (until users complain, which is very bad)
13. Work through all open issues on Github
Github has multiple open items and upgrades spread out across multiple repositories that need to be fixed and purged out.
14. Sign Windows binaries with Microsoft certificate
Installing the .msi on Windows today produces a scary warning, because the installer is not signed. We need to obtain a Microsoft Certificate and sign with it (Similar to the MacOS notarization)
15. Unify LightwalletD API
The ECC’s LightwalletD and Zecwallet’s LightwalletD have different APIs (for various historical reasons). However, now that they both support T-addresses, we should have them be API compatible, so that wallets can talk to any LightwalletD server.
Lite client protocol research + Engineering
Zecwallet has previously conducted a lot of UX research and tried to identify what it would take to improve adoption of Zecwallet and Zcash under a variety of scenarios. Please see the linked posts, as they contain a lot of details and the motivation for these work items. While there are several teams in the Zcash ecosystem working on improving privacy, Zecwallet would like to focus on improving usability and adoption, while pulling in all the great work on privacy being done by other teams.
Several of the work items below are somewhat speculative in nature, and don’t have a 100% defined specification or even a definition of what exactly will be implemented. Several of these also are dependent on conducting UX research first. I list them here because these are promising directions for Zecwallet research to take, and I firmly believe that some of these will lead to significant improvement for Zecwallet and ZCash users if implemented.
However, I don’t expect all of the items in this list to be implemented as is. Some of these might not even be feasible and have to be dropped after some initial investigation. I expect 50% of these to be implemented in some form, 25% of these to need significant rework, and another 25% to not be feasible and have to be dropped.
The description of the proposals below only outlines the research needed, but the work item includes all the engineering and coding work needed to actually implement these work items (That is, these tasks include not just the research but also the actual implementation in the wallet - shipping the code into users wallets.)
1. Price feeds via LightwalletD
Currently, clients get the Zcash price from a API service (CoinMarketCap). This stands out because this is the only API call that is going out to an external service, which is less than ideal for users that want to minimize metadata leakage. The goal of this work item is to add a way to fetch current and historical prices in multiple currencies via the LightwalletD itself (and for LightwalletD to fetch and serve these prices), so that clients don’t connect to any external service. This will also allow multiple currencies to be supported, in addition to showing historical prices (i.e., show the price of Zcash for a TX when it was sent, which is a big source of confusion today)
2. Client upgrades and version check
When Zcash network (NU) upgrades, we need to ensure that people are on the latest version of the lite clients, otherwise they see annoying errors. There also was an instance where ZIP-212 caused a pretty serious bug (a perceived loss of funds in some cases) for users that had failed to upgrade to the latest version of Zecwallet after the Canopy network upgrade.
The main challenge here is to fetch the latest client version and do auto-upgrades. Electron allows for this, so we could show a prompt to allow users to upgrade. This needs some UX research though, as auto upgrades might be tricky from a wallet perspective, especially when upgrading pools or deprecating features.
3. Mempool access to show unconfirmed incoming transactions
A big drawback for users of Lite wallets today is that it has no access to the mempool, so users don’t see incoming or outgoing transactions immediately. This always causes substantial user confusion. We should allow the LightwalletD to serve mempool transactions, and the wallets should show these pending transactions. Also needs some UX research to handle dropped/expired transactions. It is not clear how to send notifications if the wallet is not being actively used (eg., phone is locked), so this part needs some investigation.
ANCHOR_OFFSET and spend unconfirmed funds (queue txns to be sent later)
UX research has shown that this is a very annoying problem for users, where users have to wait for several network confirmations to spend funds. We should allow users to spend just-confirmed or even unconfirmed funds. UX research is needed to understand how to handle failures or reorgs. There are some privacy issues here as well that need to be understood.
5. Server sync using ivk
A highly experimental idea where teh user could send their incoming view key to a server, and have the server do a lightning fast sync and return the wallet to the user. This will dramatically speed up sync, at the cost of leaking the user’s incoming transactions to the server. UX research to understand how users perceive this trade off, and if we could make a web wallet version of this.
6. Allow server to sign checkpoints or otherwise serve them
Users that just install the wallet see a very annoying first time sync, which depends on the last code checkpoint. We need to do some research to see if the LightwalletD checkpoint can be used as an “optimistic” checkpoint, and then verified in the background before sending an outgoing transaction. For example, fetch the latest blocks checkpoint so the user can start using the wallet instantly, but “verify” the checkpoint by fetching blocks in the background. We can also verify checkpoints when resyncing.
7. Large wallet support
The lightwallet clients today don’t support very large wallets (100k+ transactions). This is research task to see what it would take to handle a large wallet. Some large users (traders, miners, merchant processors) have transactions and addresses in the 10s of thousands, and the wallet is not very performant at this scale today.
8. UX improvements to privacy for fetching transactions, like fetching memos on demand.
Several ideas to explore here based on some research and proposals from the ECC. Fetch memo’s on demand, or even have LightwalletD send the first byte of memo for every transaction, allowing users to fetch memos on demand or in batches. Also needs UX research.
9. Low-end Android support
Zecwallet Mobile doesn’t work very well on low end Android phones today, likely because of memory requirements. Zecwallet originally traded off using more memory for faster sync, but I think there might be a way to get the best of both worlds. Investigate using less memory. For eg., the Sapling params are loaded into memory, which is not really needed until sending a transaction. Also offload the tx history from memory.
10. Handling Orchard/Halo 2
This is a big item. The upcoming Orchard/Halo 2 has several challenges for wallets. It features a new shielded pool, and a new way to “migrate” funds. Lots of new UX research and implementation is needed to handle this in the wallet. Zecwallet especially is affected because it will now support 3 types of addresses.
11. Unified Addresses
The ECC’s wallet team has a very interesting idea of using “Universal Addresses”, which will allow a wallet to automatically select what type of transaction to send, depending on capabilities and shielded pool restrictions. We could also use these to auto-migrate funds and make the management of shielded pools completely automated, allowing users to get max privacy without having to worry about shielded pools. Building this in Zecwallet would be very cool, and vastly improve the shielded user experience.
Edit 1: Added (15) to Devops section to unify LightwalletD API.