Zecwallet Lightclient Protocol Research and Development

[adityapk00]

Applicant background

Zecwallet Lite, the light client mobile and desktop apps for Zcash have found some success within the Zcash community. The mobile and desktop apps are used by approximately ~7500 people today, but I think we can do much better. For background, Zecwallet has only 1 person working on it (me) part time, which is clearly not enough to grow usage and research protocol improvements. This grant proposes to expand the Zecwallet team to 3 people, including 1 full time researcher working for 6 months to ship improvements to Zecwallet and the Lite client protocol.

Motivation and overview

While Lite wallets are widely used by people already familiar with Zcash and its pros and cons, it can be quite confusing to new users or users who are familiar with Cryptocurrencies in general, but are new to Zcash. I outlined several of the issues in these two posts below. Please read these two posts first for a background on the overall motivation behind this grant.

ZIP 314 - Privacy upgrades to the Zcash light client protocol

This is a well thought out proposal that will no doubt achieve the goal it has set out for (More private Lightwallet!). However, I think this focuses on the wrong goal. My experience with Zecwallet has been that Zcash’s current problem is adoption, not a lack of privacy. Zcash already is the most private protocol out there, and already features the most private way of using this protocol (With Zecwallet Fullnode/Zcashd/Tor). However, most potential users of Zcash aren’t converting into actual …

ZIP 314 - Privacy upgrades to the Zcash light client protocol

I’d like to share 3 interviews I conducted of users that deleted Zecwallet after trying it. Edited and reproduced here with consent. UserA Zecwallet: Why did you download Zecwallet? UserA: I downloaded Zecwallet because I wanted to donate to Signal. I liked how they were sticking it to WhatsApp. ZW: Why did you choose Zcash to donate? UserA: Because I don’t want them to know it was me donating. I wanted to keep it anonymous. ZW: Because the government could decide to harass you in the futu…

The summary is that first time use of Zcash and Zecwallet in particular can be quite confusing, and is causing a significant drop-off in user acquisition. This is bad because these are users that are interested and motivated to try out Zecwallet/Zcash, but aren’t able to. This grant aims to research some of these problem areas, propose concrete improvements to the wallets and the Lite client protocol, and ship these as improvements in the Zecwallet Lite desktop and mobile wallets with a goal of improving usage, adoption and understandability of Zecwallet.

We will evaluate success/failure of this project based on improved usage of Zecwallet and shielded ZEC by new and existing users.

Do execute this work, I’m proposing to expand the Zecwallet team to 3 people (1 Devops/Testing/Infra and 2 Protocol Research/UX/Engineering)

Technical approach

This is a very large grant, so I’m splitting it up into two main parts.

1. Devops and Code improvements

While this is not directly related to improving Zecwallet UX, I think it is important to do this work concurrently. Zecwallet started as a side project, and grew from a small UI as it added features incrementally. Zecwallet was my first UI project, and consequently, at the beginning, I didn’t really know what I was doing from a devops and infra perspective. In hindsight, I made several sub-optimal decisions (and some outright bad ones). Zecwallet has accumulated a lot of technical debt which is making it (1) slow to build the next generation of features and (2) On board new developers. Doing this work will increase the ability of the Protocol Research team to quickly experiment with ideas and new UX, research usage improvements and deploy wallet code that is maintainable in the long run.

The goal of hiring one devops/engineering/testing person is to

1. Free up my time to work on protocol R&D and

2. Hire someone that knows what they are doing to fix my mistakes, pay down technical debt, and make it easier to build on the code in the future.

A detailed list of the work items in this area are listed below, but the highlights are:

1. Improve build quality and deployment ease to allow faster iteration

2. Improve testing and performance analysis, allowing easier code maintainability

3. Fix technical debt to reduce security risks

2. Lite client protocol and development.

Zecwallet has previously conducted a lot of UX research and tried to identify what it would take to improve adoption of Zecwallet and Zcash under a variety of scenarios. Please see the linked posts above, as they contain a lot of details and the motivation for these work items. While there are several teams in the Zcash ecosystem working on improving privacy, Zecwallet would like to focus on improving usability and adoption, while pulling in all the great work on privacy being done by other teams.

Several of the work items below are somewhat speculative in nature, and don’t have a 100% defined specification or even a definition of what exactly will be implemented. Several of these also are dependent on conducting UX research first. I list them here because these are promising directions for Zecwallet research to take, and I firmly believe that some of these will lead to significant improvement for Zecwallet and Zcash users if implemented.

However, I don’t expect all of the items in this list to be implemented successfully. Some of these might not even be feasible and have to be dropped after some initial research. I expect 50% of these to be implemented in some form, 25% of these to need significant rework before they are shipped, and another 25% to not be feasible and have to be dropped.

Execution risks

For the Devops portion of it, the execution risk is minimal. The work needed in this area is pretty well understood, ans should be doable without too much risk to timelines or budget. One issue might be my ability to hire someone for this role.

For the Lite client Protocol portion, there is considerable execution risk. As I outlined above, my goal is to target 50% of the items below to be researched, built and deployed at the end of this grant. Several of the work items need more investigation, understanding of tradeoffs, several prototypes and User research. Privacy implication of the work will need to be understood. Engineering feasibility is another area that will need work. A few of the items will need extensive research and reworking, but I think I’ve added enough flexibility in the schedule and work items to make it work. It is hard to outline exactly what will be built and shipped because of the very research-y nature of this work, but I ask the ZOMG and the broader Zcash community to evaluate the success and failure of this grant based on results (i.e., Better UX and adoption for Zcash and Zecwallet) instead of the amount/type/specifics or work being done.

Downsides

This grant focuses extensively on usability and adoption of Zcash. There are obvious engineering, latency, bandwidth, privacy etc… implications to several of the work items here. Some of the improvements made to help adoption or usability might hurt privacy or bandwidth/network usage. Privacy improvements incorporated from other teams work might hinder adoption. One of the goals for the Research done as a part of this grant is to come up with a framework to evaluate these tradeoffs and make sure that users understand them. While individual research items might improve or hurt adoption/privacy/latency/bandwidth, I think at the end of the grant, when we look at them together, we should have a goal of overall improvement along all dimensions.

Evaluation plan

Zecwallet currently has ~7500 users, as estimated by the bandwidth used by the lightwallet server. Our goal should be do double this number by the end of this grant.

Now, I have no idea if this is actually possible, but since this grant is focused on improving UX and adoption, this seems like a reasonable goal. Another thing to note is that measuring usage accurately is not quite possible since Zecwallet doesn’t do any logging or collect any type of user data, so we’ll have to estimate usage via downloads, blocks served etc… This should be fine, since this is what we’re doing right now anyway.

Tasks and schedule

(Please see below for details)

Budget and justification

Zecwallet will hire 2 additional people for 6 months. A devops/test/infra person (part time) and another fulltime Protocol/Engineering researcher. I will also work on the Protocol Research piece part time.

                          | Fulltime / | Weeks  | Calendar| Rate | Total 
                          |Part-time   | worked |  weeks  |Hourly|

Devops/Testing/Infra (TBH)   Part time | 12    | 26       |150   | 72,000 

Protocol Researcher 1 (me)   Part time | 10    | 26       |187.5 | 75,000

Protocol Researcher 2 (TBH)  Full time | 26    | 26       |187.5 | 195,000

Total                                                            | 342,000

The hourly rate includes all expenses, like software licenses, server costs, hardware/laptops etc…

---

Devops

This section of the work is to hire one additional devops and infra engineer, who can:

1. Clean up the code and pay down a lot of the technical debt

2. Improve build automation and automate deployments and monitoring

3. Add automated testing and benchmarking to the UI and the SDK.

Some of the important things in this area are listed below. For a few of these items, ECC already has some code (for eg., DarksideWallet for Lite testing), so we’ll also pull in and integrate these into our infrastructure. Some of the items below are somewhat uncertain and they need more investigation, so at the end of the project, some of these items might change or we might implement a different solution. We might also realise that some of the items are not possible to implement, in which case we might have to look for alternatives.

1. Switch from Flow to Typescript in the UI

Flow for javascript typechecking is sub-optimal, because its optional, and doesn’t necessarily catch all the cross-file type issues. Using the Typescript compiler will catch many more bugs (Like the missing includes). This is mainly a build system change, since the code for Flow and Typescript is largely similar. Typescript may need some additional refactoring, but it should be limited since the existing code is already ES6 compatible.

2. Use Electron-builder for UI builds

The current process for building Zecwallet and Zecwallet Lite releases is very cumbersome, manual, and error prone. Using the electron-builder will allow us to at last properly use the Github Actions, to both build and sign the releases (including MacOS notarizing and Windows .msi signing), allowing us to release much more frequently.

3. Upgrade Neon to use the Electron <-> Native bridge

The current Neon infra is deprecated, because it uses context-free Nodejs modules, and Node has upgraded to use context-aware modules. The current Neon macros are also very brittle, passing everything as JSON strings. They allow multiple bugs and security issues while crossing the Electron → Rust boundary. Use the serde library and proper types to make the SDK use more robust.

4. zcashd builds & automated tests.

Zcashd builds are very manual right now, and don’t come with automatic testing. Use Github Actions to create “official” zcashd builds for embedding into Zecwallet, allowing a much less manual and error prone process to be deprecated. This will also allow Zecwallet to publish zcashd builds for MacOS, Windows and Linux for users that want to run a full node.

5. zcashd integration tests for Zecwallet Fullnode

Automate integration tests when being used as embedded zcashd, catching RPC change-caused bugs automatically. Zecwallet Fullnode uses the RPCs extensively, so changes to the interface and behavior can cause breaking changes in the wallet UI. These are manually detected now which has caused several of them to slip past in previous releases.

6. zecwallet SDK benchmarks

Zecwallet has suffered multiple performance regressions in the past because of a lack of performance regressions testing. Especially for constructing transactions and parsing Compact Blocks. Write benchmark and performance tests, and run them as a part of the build.

7. Automate Zecwallet Lite SDK integration tests

Zecwallet SDK has unit tests, but no integration tests. Especially the API / RPC tests, that test the use of the SDK from inside the wallet, catching bugs automatically.

8. UI test automation

Zecwallet UI (for both Fullnode and Lightclient) doesn’t have any UI tests. A lot of previous bugs have been caused by a lack of UI testing, which should really be automated, to catch issues when making changes in the code.

9. Automate builds for Paper wallet

The Paper wallet is entirely manual build process today, which is super error prone, and causes new releases not be released very often. This has caused security issues in the past. Automate the build and release process for the ZecPaperWallet to bring it in line with the Zecwallet UIs, allowing for regular releases

10. Zecwallet Mobile Automated Tests

The Zecwallet mobile apps don’t have any unit tests. We need at least smoke tests and a simple end-to-end test to ensure that when we update dependencies etc…, we don’t end up with bugs, which causes App store rejections. This has caused multiple App store rejections in the past which adds a lot of delays and back-and-forth with Apple.

11. Automate zcashd <-> LightwalletD integration and benchmarks

Using zcashd and LightwalletD is very error prone today because of a lack of any integration or performance tests. ECC has many tests here, that we should really integrate that will allow smooth upgrades of zcashd, which have to be done every couple of months. This will reduce downtime and unnecessary errors with Light clients

12. Automate LightwalletD performance monitoring

Current LightwalletD logging and performance analysis is very manual. We should ingest the logs into Prometheus/Elastic Search and automate monitoring with downtime alerting. This has been lacking, which has caused several outages to be undetected for hours (until users complain, which is very bad)

13. Work through all open issues on Github

Github has multiple open items and upgrades spread out across multiple repositories that need to be fixed and purged out.

14. Sign Windows binaries with Microsoft certificate

Installing the .msi on Windows today produces a scary warning, because the installer is not signed. We need to obtain a Microsoft Certificate and sign with it (Similar to the MacOS notarization)

15. Unify LightwalletD API

The ECC’s LightwalletD and Zecwallet’s LightwalletD have different APIs (for various historical reasons). However, now that they both support T-addresses, we should have them be API compatible, so that wallets can talk to any LightwalletD server.

---

Lite client protocol research + Engineering

Zecwallet has previously conducted a lot of UX research and tried to identify what it would take to improve adoption of Zecwallet and Zcash under a variety of scenarios. Please see the linked posts, as they contain a lot of details and the motivation for these work items. While there are several teams in the Zcash ecosystem working on improving privacy, Zecwallet would like to focus on improving usability and adoption, while pulling in all the great work on privacy being done by other teams.

Several of the work items below are somewhat speculative in nature, and don’t have a 100% defined specification or even a definition of what exactly will be implemented. Several of these also are dependent on conducting UX research first. I list them here because these are promising directions for Zecwallet research to take, and I firmly believe that some of these will lead to significant improvement for Zecwallet and ZCash users if implemented.

However, I don’t expect all of the items in this list to be implemented as is. Some of these might not even be feasible and have to be dropped after some initial investigation. I expect 50% of these to be implemented in some form, 25% of these to need significant rework, and another 25% to not be feasible and have to be dropped.

The description of the proposals below only outlines the research needed, but the work item includes all the engineering and coding work needed to actually implement these work items (That is, these tasks include not just the research but also the actual implementation in the wallet - shipping the code into users wallets.)

1. Price feeds via LightwalletD

Currently, clients get the Zcash price from a API service (CoinMarketCap). This stands out because this is the only API call that is going out to an external service, which is less than ideal for users that want to minimize metadata leakage. The goal of this work item is to add a way to fetch current and historical prices in multiple currencies via the LightwalletD itself (and for LightwalletD to fetch and serve these prices), so that clients don’t connect to any external service. This will also allow multiple currencies to be supported, in addition to showing historical prices (i.e., show the price of Zcash for a TX when it was sent, which is a big source of confusion today)

2. Client upgrades and version check

When Zcash network (NU) upgrades, we need to ensure that people are on the latest version of the lite clients, otherwise they see annoying errors. There also was an instance where ZIP-212 caused a pretty serious bug (a perceived loss of funds in some cases) for users that had failed to upgrade to the latest version of Zecwallet after the Canopy network upgrade.

The main challenge here is to fetch the latest client version and do auto-upgrades. Electron allows for this, so we could show a prompt to allow users to upgrade. This needs some UX research though, as auto upgrades might be tricky from a wallet perspective, especially when upgrading pools or deprecating features.

3. Mempool access to show unconfirmed incoming transactions

A big drawback for users of Lite wallets today is that it has no access to the mempool, so users don’t see incoming or outgoing transactions immediately. This always causes substantial user confusion. We should allow the LightwalletD to serve mempool transactions, and the wallets should show these pending transactions. Also needs some UX research to handle dropped/expired transactions. It is not clear how to send notifications if the wallet is not being actively used (eg., phone is locked), so this part needs some investigation.

4. Reduce ANCHOR_OFFSET and spend unconfirmed funds (queue txns to be sent later)

UX research has shown that this is a very annoying problem for users, where users have to wait for several network confirmations to spend funds. We should allow users to spend just-confirmed or even unconfirmed funds. UX research is needed to understand how to handle failures or reorgs. There are some privacy issues here as well that need to be understood.

5. Server sync using ivk

A highly experimental idea where teh user could send their incoming view key to a server, and have the server do a lightning fast sync and return the wallet to the user. This will dramatically speed up sync, at the cost of leaking the user’s incoming transactions to the server. UX research to understand how users perceive this trade off, and if we could make a web wallet version of this.

6. Allow server to sign checkpoints or otherwise serve them

Users that just install the wallet see a very annoying first time sync, which depends on the last code checkpoint. We need to do some research to see if the LightwalletD checkpoint can be used as an “optimistic” checkpoint, and then verified in the background before sending an outgoing transaction. For example, fetch the latest blocks checkpoint so the user can start using the wallet instantly, but “verify” the checkpoint by fetching blocks in the background. We can also verify checkpoints when resyncing.

7. Large wallet support

The lightwallet clients today don’t support very large wallets (100k+ transactions). This is research task to see what it would take to handle a large wallet. Some large users (traders, miners, merchant processors) have transactions and addresses in the 10s of thousands, and the wallet is not very performant at this scale today.

8. UX improvements to privacy for fetching transactions, like fetching memos on demand.

Several ideas to explore here based on some research and proposals from the ECC. Fetch memo’s on demand, or even have LightwalletD send the first byte of memo for every transaction, allowing users to fetch memos on demand or in batches. Also needs UX research.

9. Low-end Android support

Zecwallet Mobile doesn’t work very well on low end Android phones today, likely because of memory requirements. Zecwallet originally traded off using more memory for faster sync, but I think there might be a way to get the best of both worlds. Investigate using less memory. For eg., the Sapling params are loaded into memory, which is not really needed until sending a transaction. Also offload the tx history from memory.

10. Handling Orchard/Halo 2

This is a big item. The upcoming Orchard/Halo 2 has several challenges for wallets. It features a new shielded pool, and a new way to “migrate” funds. Lots of new UX research and implementation is needed to handle this in the wallet. Zecwallet especially is affected because it will now support 3 types of addresses.

11. Unified Addresses

The ECC’s wallet team has a very interesting idea of using “Universal Addresses”, which will allow a wallet to automatically select what type of transaction to send, depending on capabilities and shielded pool restrictions. We could also use these to auto-migrate funds and make the management of shielded pools completely automated, allowing users to get max privacy without having to worry about shielded pools. Building this in Zecwallet would be very cool, and vastly improve the shielded user experience.

---

Edit 1: Added (15) to Devops section to unify LightwalletD API.

[hanh]

Hi, thanks for the detailed grant explanation.

It’s not clear to me what constitutes improvements, technical debt fixes to ZecWallet vs contributions to the lightwallet ecosystem. For instance, it mentions switching away from electron, anchor_offset and scalability issues. These seems to be decisions that ZW makes and not necessarily applicable as a whole.
Then it also says “spend unconfirmed funds”. I’m not clear how this can be done since the note commitment is not yet part of the merkle tree. And then there are mempool changes to lightwalletd.
I understand there is a level of research and protocol improvements to be done but in my opinion, this should be part of another task towards standardization of the lightwallet libraries.
For example, when I was looking to implement a small feature I realized that my options were limited. Only open source wallet is ZW and therefore it is the obvious choice. But then there are also an ECC libraries and I learned ZW forked lightwalletd. To me, regardless of the technical/historical reasons, this gap makes it hard for a new developer to invest in ZW or even lightwallets in general.
As it stands, I’m concerned that this proposal will further separate ZW from zcash/lightwalletd. A new contributor would have to somehow pick a side.

[Shawn]

@hanh

To me, regardless of the technical/historical reasons,

I agree with your concerns about having two different approaches to lite wallets (ECC approach and ZecWallet approach) and I understand for a new developer it can be confusing why this is.

Basically ZecWallet started well before the ECC lite wallet code was released. Later ZecWallet released a lite wallet based on a forked version of lightwalletd which did something that I feel is critical for Zcash users (that the ECC wallet does have in limited capacity) which is T-addresses support. I don’t want to get into the philosophical reasons for maintaining T vs Z addresses, but with the current ecosystem still largely reliant on T-addresses I think we would be remiss to not have them fully supported (full sending and receiving) in lite clients and I’m not quite sure why the ECC team chose limit thier use in thier client.

Aditya has mentioned before that much of the codebase that he is working with is kinda “homebrew” to get things to work the best way he knew how in the time he had. This approach has worked remarkably well considering he was working on it part time and has resulted in the most used lite wallet in the Zcash ecosystem. I think this proposal aims to help address some of those legacy issues and hopefully make ZecWallet lite easier to work on as a result.

I too would prefer if there were some kind of unified option that ZecWallet team and ECC teams could come to support. One lite client to rule them all, but until the core ECC code addresses the current needs of the end user better with full T address support I don’t know how soon that could happen.

[adityapk00]

Then it also says “spend unconfirmed funds”. I’m not clear how this can be done since the note commitment is not yet part of the merkle tree

This is actually a great question, and something I’ve been thinking about too. Here’s the situation:

Users want to spend unconfirmed funds. But from a technical perspective, it seems impossible, since the note commitment is not yet in the blockchain. So, what can we do?

I think there’s quite a bit of design space in between “User wants something” and “Technically Impossible” that I would like to explore in this grant. In this particular case, can we come up with some creative technical or cryptographic solution that can address this? Here’s an idea:Both Android and iOS allow running some sort of background tasks now, so we could:

• User receives an unconfirmed incoming z-address transaction
• User goes to the spend screen and types in the address, amount etc… to send.
• User presses “SEND”, and the wallet sets a background task that will wake up in 1 minute.
• After 1 minute, if the incoming note is not confirmed, go to sleep for 1 more minute
• If it is confirmed, then gather the commitment, the anchor offset, construct and sign the transaction and then broadcast it.

Now obviously, there are many challenges here (reorgs, reliability of the background task, privacy etc…), but I would like to recursively explore creative solutions to those as well and see if we can find some combination that will work for the user. I can think of such “creative” solutions for every one of the 11 items listed above. Maybe some of them work, and others don’t work. But I would like to try and explore them.

I think this outlines the philosophy I want to pursue pretty well. I’m not sure 100% that it will work, but I would like to explore this design space between “User wants it” and “Technically Impossible” and see if we can think of some creative technical or cryptographic solution to the problems and make it work. Zecwallet has done this in the past, and I’m confident we can do this kind of work again.

Now, it is entirely possible that it can’t be solved, and we have to give up and move on, I want to admit that. But I’m asking for the Zcash Community (and ZOMG in particular) to take this risk with me and see if we can make progress towards building things that will delight users and make Zcash a pleasure to use!

[adityapk00]

To me, regardless of the technical/historical reasons, this gap makes it hard for a new developer to invest in ZW or even lightwallets in general.

This is a fair criticism of the situation we are in now. As Shawn has alluded to above, I had to fork librustzcash and lightwalletD to launch Zecwallet Lite. However, since then I’ve tried to work backwards as well, trying to get back to a consistent state. I would particularly like to thank the whole ECC SDK team, who I feel like have sometimes bent over backwards to accommodate my requirements.

Thanks to our joint work, I’ve managed to submit all changes I made to librustzcash upstream, and now Zecwallet’s fork of librustzcash is nearly eliminated.

LightwalletD is also in a similar state. There’s some work needed on the Zecwallet side (listed above) that will once again make the two LightwalletD’s come back together, at least from an API perspective. That is, even if the implementations differ, the LightwalletD’s and Wallets will be two-way compatible with each other.

These two examples (there are many more divergences between Zecwallet and the ECC Wallet) illustrate what has worked in this ecosystem. Zecwallet goes off and explores a bunch of things. Some things don’t work out, and some things do. For things that do work out, we work together to integrate the changes back into one piece again. And then we go off and explore ideas again. Rinse and repeat.

I think this model is very healthy, and results in much more overall progress than if we would all have to wait for consensus to make changes. That’s not to say your criticism is unwarranted. You are absolutely correct, and we should strive to integrate things back together where we can, and I want the community to hold us accountable for it.

Your feedback is well taken. I will add a 15th item to the Devops section above to specifically address integrating the LightwalletD changes.

[hanh]

I clearly appreciate Aditya’s contributions to the LW. Without his work, we wouldn’t be at this stage. I think the issue is actually the lack of a reference rust library. librustzcash is improving but is not “official” yet.
Nowadays, rust libs can be called directly from most languages. For example, I could write a mobile watch-only wallet app in a few hours. Before, it wasn’t the case and more code had to be written to interface with rust.
Before going down that route, I reviewed the entire code base of ZecWallet. I came to the conclusion that it would be too difficult to use it because of the tech stack used and the design choices that were made. This is not a criticism at all. It’s just that the code is opinionated. For instance, t-addr are automatically migrated, storage is in memory, etc. It’s perfectly fine for an app. For a library, it limits its reusability.
It seems that part of the grant proposal is about funding ZW development and another part is about research.
In my opinion, this is unclear how this would be reusable outside of ZW.

@Shawn

One lite client to rule them all, but until the core ECC code addresses the current needs of the end user better with full T address support I don’t know how soon that could happen.

I think having more coordination with ECC is paramount. Because, Zcash encrypts the spending notes. It makes it impossible to check their validity (the zksnark doesn’t cover this part). As such, if a wallet app produces an invalid note, it will end up being unspendable. There was a close call with the issue related to ZIP212. Maybe you could look into funding this - or ZW with ECC.

Finally, regarding the list of work that the funding would be for, IMO, they should be reviewed for benefit/cost.

My take would be:

1. price feed via lightwalletd. I don’t think it’s the job of a node to get market value. Though it’s more of philosophical thing, lwd shouldn’t do more than what zcashd does. Plus it would open the door for a debate on what makes a reliable price feed.
2. it should be fixed and I think it would have been avoided if ZW could rely on a library for “consensus” critical parts.
3. mempool. Agree, it is a big problem.
4. reduce anchor_offset. This is a wallet decision. The code change is trivial but its implications are big.
5. queue tx, it’s a workaround that is not quite the same as spending a unconfirmed tx. This doesn’t seem as useful as for bitcoin since the 2nd tx is not actually linked to the 1st one (not like lightning)
6. server side ivk. If that is pushed to lwd, then it should be done in the main branch. not convinced it should be though.
7. this “verification” is not needed. If ZW trusts lwd for compact blocks, it should trust checkpoints too
8. I don’t think many people have 100k+ transactions on their mobile wallets. I would consider this a low hanging fruit.
9. sounds good
10. this memory requirement is caused by having the app load everything in memory? In any case, low end android support may not be very important if this means switching the storage layer to db
11. orchard/halo 2. Yes it is big. this works should be done in a common library. Every wallet would need to go through this.

It’s a big funding proposal. I’d rather have the money spent on the development of an official wallet or if not possible, an official library upon which to build wallets.

[hanh]

Sorry, it seems that our replies got crossed. Please allow me to reply to your points.

IMO, the proposal bundles several items together and it is difficult to see where the grant funding will actually be spent on.

@adityapk00

These two examples (there are many more divergences between Zecwallet and the ECC Wallet) illustrate what has worked in this ecosystem. Zecwallet goes off and explores a bunch of things. Some things don’t work out, and some things do. For things that do work out, we work together to integrate the changes back into one piece again.

This is good and it’d be great to see it continue.

@adityapk00

I think this model is very healthy, and results in much more overall progress than if we would all have to wait for consensus to make changes.

If I can play the devil’s advocate, I’d say that while it’s certainly healthy for ZW, it has driven the development of the light client ecosystem towards a specific set of needs and tech choices. Basically, we have a library that works well with ZW because it originated from ZW. More on this later.

Personally, I’d appreciate if the proposal was split into several grants.

ZW Tech Refresh & Engineering Best Practices

• Refactorization, Code clean up,
• Tech debt: native wrapper instead of electron, react native → json → node.js, javascript, rust
• Test automation,
• Continuous Integration,
• etc.

UX Experiments

• ANCHOR_OFFSET,
• spend unconfirmed funds (queue txns to be sent later),
• Server sync using ivk,
• Scale to 140k tx

I feel these tasks should be prioritized by an external party (like a product manager job). Product features that are driven by the engineering team tend to end up disconnected from user benefits. I think the ZOMG could decide individually on which feature and improvement to fund, rather than the ZEC team getting a lump sum.

Core functions

• Orchard/Halo 2
• UA

This is where I think we disagree the most. IMO, this is consensus related and therefore should not be under the responsibility of a client code. Any deviation from the protocol (regardless of who is at fault, protocol doc or implementation) could lead to loss of funds. We have experienced it recently and the Bitcoin team has had their share of this too.
Anyway, a wallet implementation should not have to understand the inner working of UA or Orchard/Halo2. If it needs to write zksnark or even ECC primitives, something is wrong.
From this perspective, orchard and halo 2 integration should not be massive. Obviously, it will have an impact on the data model and the UI, but I expect it to be of the same scope as T-address / Z-address.

Ecosystem

Expanding from the point earlier about the development model, I think we should also have a grant for tasks that are not interesting or relevant to ZW.
For example:

• Transparent Notes management. ZW chose to not manage t-addresses and auto-migrate immediately. That’s not always what the user wants.
• Storage options. ZW keeps data in memory and files, librustzcash stores in sqlite. The library shouldn’t force the dev into any tech. The refactoring work with the data api in librustzcash was a great move into this direction.
• Handling reorg in the core library.
• Developer documentation and tutorial.
• Ideally, the library should provide enough guidance in its API that it is hard to do the “wrong thing”. Today you can do the “right thing” but easily mess up too.
• Finally, More language choices. Today it’s all Rust. If someone wants to build a transaction in Java, Javascript, Python, Go, C++ or C#, he has to use t-addr (and that’s only because he can use a bitcoin library). As a recent comer, this was the first thing that stroke out to me. These are the most popular languages out there and by not offering practical bindings (rust ffi doesn’t qualify ) we are missing out on developer talent and cool apps that leverage zcash shielded pool.
  For instance, I think web3j is very helpful for the Eth ecosystem.

[holmesworcester]

@hanh I think you’re right that it would be desirable for there to be one-stop shopping for developers coming into this space, with a clear library to choose that is built for a wide range of purposes.

And I totally agree on the “ecosystem” points.

But I think I’d echo Aditya’s point that there are tradeoffs here. Right now, Aditya is able to make improvements up and down the stack to address issues he’s hearing about directly from users. That’s let him move a lot faster on usability than wallets that depend on the ECC SDK, which moves slower. Zbay has been using zecwallet-light-cli as our library and it’s been a good choice for us so far. We chose it because at the time, the ECC library did not support t-addresses at all.

If Zecwallet succeeds at addressing half of the UX-related lightwallet protocol issues above, it will continue to be the best choice—at least for projects whose use-case and tech stack are similar to Zecwallet’s (i.e. cross platform wallets with Electron and React Native, which are common stacks!) For example, if Aditya can address the issue of long syncing times before returning users can access funds, that would be huge leap ahead of the ECC library, for Zecwallet, Zbay, and any projects using the Zecwallet libraries.

I think we can also take your comment as advice for ZOMG about what to fund.

On that front, Nighthawk, which is building on the ECC SDK, is in a position where in order to achieve their goals they will need to make a lot of upstream fixes to the ECC SDK. That’s part of the grant they just applied for. If we fund them, we’ll be funding work on the official library, and will hopefully increase its feature set, velocity, and responsiveness to user needs. We’re also funding Aditya as part of this grant to bring the lightwalletd’s closer together, so there’s that too.

We’d be interested in funding other groups to work on the official SDK too, as well as on the other items in your ecosystem list. If this is work that you’d be interested in undertaking or in managing a team around, we should discuss this!

The one anxiety I have is that we are in a situation common to open source where a project has forked an official library to make it more responsive to their own needs, but where it would be beneficial (both to the official project and the forking project, which is now carrying a growing maintenance burden) to get improvements from that library merged upstream—and I’m wondering whether we are approaching that in the best possible way. Like, @adityapk00, do you think that divergence from the official library is a kind of technical debt, since it could become more and more burdensome over time for you to maintain your fork on your own? And if it is, are we giving you the support you need to address that technical debt? I know there’s a plan for this on the lightwalletd side, but is there a plan for it on the zecwallet-light-cli side?

[holmesworcester]

Also, Aditya I genuinely admire how seriously you’ve taken the work of understanding your users, and how hard you’re pushing for protocol changes that will unblock, un-confuse, or—ideally—delight them.

[hanh]

@holmesworcester

For example, if Aditya can address the issue of long syncing times before returning users can access funds, that would be huge leap ahead of the ECC library, for Zecwallet, Zbay, and any projects using the Zecwallet libraries.

This is pretty hard to address technically since it’s part of the design of zcash to hide the data. Of course, you can improve the UX by doing background download, scanning while idle, etc.

But I think we need to be very clear on what improvement will be made in ZecWallet vs offered to the community.

People tend to underestimate the difficulty of extracting functionalities from apps - especially when it was not designed as such in the first place.

@holmesworcester

The one anxiety I have is that we are in a situation common to open source where a project has forked an official library to make it more responsive to their own needs, but where it would be beneficial (both to the official project and the forking project, which is now carrying a growing maintenance burden) to get improvements from that library merged upstream—and I’m wondering whether we are approaching that in the best possible way.

This is exactly what I’m trying to say. Both grants for client wallet developments are rather large. From an economical point of view, it would be easier to see the cost/benefit if it was more directed towards the ecosystem. Otherwise, I’m afraid we would have two teams forking the main code and making changes in parallel. At the end, there would be ECC, ZEC, NW versions. The grants mention making contributions back upstream but it doesn’t seem to be their priority. IMO, it needs more safeguard and commitment.

Just to be clear, I’m not against funding teams in order to help them cover their technical debt. I would like to have transparency on where the funding goes.

One of the main challenges with lightwallet development is how much they are tied to the protocol.
As such, we are constrained by what can be done. For instance, “multi signature” had to be done using threshold signatures and not multi signatures per say. Funding research is a little bit tricky here because it may end up being “it cannot be done”, or “i could only do ”, and the ZOMG may end up feeling disappointed by the results.

@holmesworcester

We’d be interested in funding other groups to work on the official SDK too, as well as on the other items in your ecosystem list. If this is work that you’d be interested in undertaking or in managing a team around, we should discuss this!

I’d like to discuss it at least informally. Seeing people coming in the forum asking for help to create transactions in go, python and even javascript and get no answer is a little disheartening. We are way behind other crypto projects on this aspect. For me, this is the low hanging fruit.

Another topic that could be worth discussing is whether ZOMG would like to fund Layer 2. For instance, would it be possible to have smart contracts? (just a wild thought)

[holmesworcester]

@adityapk00 Great news! ZOMG has accepted your proposal.

We’re really excited to see Zecwallet scale up in the ways you’ve proposed, and I think defining the goal in terms of growth is a great way to move forward, to focus attention on the UX issues that are holding back Zcash adoption.

We’re still figuring out how best to support grantees re: security audits, but that’s on us for now, and we can talk more about it when that work on our end is complete, or in the next application cycle.

@hanh I really appreciate the quantity and quality of feedback you’ve provided to this application, and I don’t want it to seem that we aren’t listening.

For most of the needs and issues you’ve identified, I’m persuaded that these are real, and that ZOMG should try to address them in its funding decisions.

That said, in this case we have an applicant who has a vision for what they want to do, and is applying for that. And it’s a good vision! If Zecwallet focuses on security, stability, and user growth, and optimizes for that, and succeeds, that’s a success for Zcash, independent of how reusable the tools they’re making are for other projects.

(And again, so far my experience building an Electron-based Zcash app for Mac, Windows, and Linux has been that Zecwallet’s underlying tools are actually very reusable.)

If another applicant comes along saying, “look, this is the wrong way; we can move faster and build a better foundation by focusing on building a very modular SDK (or unifying projects around the official ECC SDK)” and they convince us that they can do it, I think it’s likely we’d fund them too.

@hanh

Seeing people coming in the forum asking for help to create transactions in go, python and even javascript and get no answer is a little disheartening. We are way behind other crypto projects on this aspect. For me, this is the low hanging fruit.

I’m curious to learn more about this. My worry would be that if we fund these libraries absent projects that are going to use and maintain them, they’ll get stale as the protocol evolves. Do you think that’s a risk? (Also, perhaps you’d like to start another thread for this discussion?)

[adityapk00]

This is fantastic!

Thank you @ZcashGrants ! I’m really excited to get started working on this, and to bring Zcash to more people and more usecases!

[hanh]

@holmesworcester

(And again, so far my experience building an Electron-based Zcash app for Mac, Windows, and Linux has been that Zecwallet’s underlying tools are actually very reusable.)

Yes, if you are doing what ZW does, it helps a lot. It’s when you want to do something different that it’s harder. Btw, IMO electron based app aren’t the best way to do portable mobile apps anymore.
(I have made a few videos about integrating librustzcash into flutter. I think it’s much easier. https://www.youtube.com/playlist?list=PLePMF1PHQMvE1vwBp0yI-8ZwQUIZbuHUR)

@holmesworcester

I’m curious to learn more about this. My worry would be that if we fund these libraries absent projects that are going to use and maintain them, they’ll get stale as the protocol evolves. Do you think that’s a risk? (Also, perhaps you’d like to start another thread for this discussion?)

Sure, it’s a chicken and an egg problem. Currently, it’s organic. Libraries are created as needed which ensures that they are used. However, I think it’s an investment worth doing in the long term. At the beginning, they probably won’t have much usage.

[aristarchus]

At the risk of being off topic I just want to second @hanh on the need for good python, go, and js libraries for creating transactions. I’d love to see the @ZcashGrants fund this.

Ok, back on topic, congratulations to @adityapk00!

[pacu]

First: Congratulations @adityapk00 !

Yes, if you are doing what ZW does, it helps a lot. It’s when you want to do something different that it’s harder. Btw, IMO electron based app aren’t the best way to do portable mobile apps anymore.
(I have made a few videos about integrating librustzcash into flutter. I think it’s much easier. zcash - testnet wallet - YouTube )

This is very interesting. My personal point of view is that Multiplatform UI frameworks come and go, and Cryptocurrencies are here to stay. So, we should aim to create tools that support Base platforms in their native languages when available, because the Flutters or ReactNative of the world will have to provide a way to interface with the underlying system anyway.

This design pattern consisting of UI Framework with general purpose language that abstracts multiple platforms and specifies via Plug-ins has been verified many times since the advent of mobile smartphones (see cordova/phonegap, react native and now flutter) and nothing tells me that there’s a trend going against it

We should not worry about interoperability with UI frameworks per se, but provide reliable tools for each relevant platform, secondly if one of these platforms pick up a huge developer base, the plugin necessity will manifest itself and projects will emerge because of that, which ZOMG could fund if somene requests it or maybe companies developing these wallets will.

Tooling dispersion is a thing that happens in other privacy coin projects too and that we are in a good place to avoid it completely.

Monero’s main mobile wallets wrap a C++ api called ‘libwallet’ from the monero full node repo. If you search the Header filename in Github ( wallet2_api.h) you will find more than 5300 search results across hundreds of projects. That supposes a lot of effort carried on by developers wrapping their heads around complicated stuff on C++ that wouldn’t happen if there were proper tooling for the most relevant platforms at the time.

It’s not accurate to pretend that mobile developers should also learn rust or C++ to be able to integrate with a Crypto.

So I believe that we must assume our condition of “rara avis” of being developers that have the time and will of going full Js+Rust+WebAssembly, Swift+rust+C or Java+Kotlin+Rust and build a good toolbox for Mobile developers so they don’t have to take side quests to fulfill a task that might be a only one requirement of their software and not the main goal of it.

We all love Zcash, and it’s one of the main things in our lives, that would indicate that when Zcash finally becomes massively adopted, we will be “core developers” and that’s totally fine, there will be numerous developers that just want to integrate with Zcash and carry on with their lives, and that’s great too, because otherwise it would mean that Zcash is too difficult to be wide-spread.

[hanh]

Well, ideally there should be packages for react native, flutter, etc. Having only native libraries doesn’t reflect the way most mobile apps are developed. The cost of two code bases (Android ios) isn’t worth the potential speed up and UI look and feel.
At this moment, the situation is similar to https://www.zfnd.org/zcon/0/workshop-notes/developer-libraries/

The Android library is a wrapper around librustzcash with some implementation for storage and networking.
I don’t see much value in using it over librustzcash directly since storage and networking are the things that the app probably wants to pick anyway.

[aiyadt]

@hanh

Well, ideally there should be packages for react native, flutter, etc. Having only native libraries doesn’t reflect the way most mobile apps are developed. The cost of two code bases (Android ios) isn’t worth the potential speed up and UI look and feel.

Native Android & iOS libraries are provided by majority of the top software providers(from analytics, billing to KYC), and it will stay that way as long as Apple & Google spearhead their development in different directions. Flutter/React are wonderful tools and work “okay” for specific use cases like a simple Banking App or Car Key app. But for a full-fledged app that utilizes platform specific APIs for secure storage, biometrics, dependencies on other 3rd party libs, it requires having 2 separate native codebases for Android & iOS.

Also note that Google has a history of dropping support for entire platforms, so I wouldn’t be surprised if Apple changes their architecture that could render Flutter for iOS useless, and soon Google itself could drop Flutter.

@hanh

At this moment, the situation is similar to Developer Libraries – The Zcash Foundation

Check out Halo Arc for Zcash proposed for release later this year - Electric Coin Company

@hanh

The Android library is a wrapper around librustzcash with some implementation for storage and networking.
I don’t see much value in using it over librustzcash directly since storage and networking are the things that the app probably wants to pick anyway.

There is a real use case for the Android & iOS libraries that provide secure networking & storage in the optimal manner which a client app should not need to worry about. The Zcash SDKs allow easy drop in integration for apps that want to enable Zcash Shielded & soon Transparent support, directly accessible via an Android lib repository. Additionally the documentation of SDKs would help developers who are adding non-custodial ZEC support to their apps. Those devs don’t need to tinker with librustzcash and create possible blunders for their users.

@NighthawkApps entirely relies on the GRPC networking for everything in the iOS app except Android app fetching the ZEC price via https(which we plan to migrate to use GRPC too).

[hanh]

@aiyadt

But for a full-fledged app that utilizes platform specific APIs for secure storage, biometrics, dependencies on other 3rd party libs, it requires having 2 separate native codebases for Android & iOS.

Hmm… not really. Here are some packages for what you mentioned.

barcode_scanner | Flutter Package

biometric_storage | Flutter Package

@aiyadt

Also note that Google has a history of dropping support for entire platforms, so I wouldn’t be surprised if Apple changes their architecture that could render Flutter for iOS useless, and soon Google itself could drop Flutter.

Yes, their track record is pretty bad.

@aiyadt

Check out Halo Arc for Zcash proposed for release later this year - Electric Coin Company

I don’t see anything planned regarding (from the ZF link)

• Discussion on what do we need?
  • In-browser Signing / Creation of transactions
  • cross-language tools
  • Need for a curated list of dev tools
  • APIs for humans and machines
• Back to where dev tools are required: – Low friction - Convenience: — Browser — Docker — Packages with every language (npm, pip)
• Operations required for such libraries to do: – Address creation — ZIP32 – Address (key) Management – Signing — Multi-signature Architecture – Proving – Payment detection – UTXO Management – Serialization

@aiyadt

There is a real use case for the Android & iOS libraries that provide secure networking & storage in the optimal manner which a client app should not need to worry about. The Zcash SDKs allow easy drop in integration for apps that want to enable Zcash Shielded & soon Transparent support, directly accessible via an Android lib repository. Additionally the documentation of SDKs would help developers who are adding non-custodial ZEC support to their apps. Those devs don’t need to tinker with librustzcash and create possible blunders for their users.

I’m glad it worked out for you though I still don’t see the benefit of having native android and native ios for the features you mentioned.

To be clear, I’d rather use the react native/flutter way

React Native/Flutter

than the native way

Native Only

but today I have to do

React Native FFI

Not my best choice either. I’d rather have flutter/reactive packages.

[aiyadt]

@hanh

Here are some packages for what you mentioned.

Sure there are packages created by the community for Flutter, but they are not reliable as the moment there are changes in the underlying Android/iOS platform, the flutter libs need to be updated to match up. Same goes for vulnerabilities that might remain unpatched via the Flutter route.

I’d only use Flutter for a basic app, but for any feature-rich app I’d stick to native platform specific development.

I agree on the point of having packages in every language golang, java, python… Maybe ECC folks can look in to that following the initial Halo Arc release. For now, the aim to provide drop-in libs for popular platforms is the right approach.

@hanh

I’m glad it worked out for you though I still don’t see the benefit of having native android and native ios for the features you mentioned.

The feature set never ends for a native app, the features can be extended to several integrations. I’ve worked with clients that have 50+ SDK integrations in a single app and the only sane way to manage it is to rely on up-to-date SDKs with documentation. No team has bandwidth to debug issues in one particular Rust/C++ bindings affecting one library.

[hanh]

@aiyadt

I’ve worked with clients that have 50+ SDK integrations in a single app and the only sane way to manage it is to rely on up-to-date SDKs with documentation. No team has bandwidth to debug issues in one particular Rust/C++ bindings affecting one library.

I guess I didn’t make myself clear. Rust/C++ bindings is not the way to go for me either. Proper native packages wrapped to work with flutter/react native is my preferred way.
But if they don’t exist, I will still not opt for two code bases.