Zecwallet Code, Maintenance and Infra

Community Grants Applications
1

Applicant background

Zecwallet’s family of products have seen regular maintenance, bug-fix and security releases for over two years, thanks to previously funded grants from the Zcash Foundation. This is a continuation grant intended to cover ongoing maintenance (bugfixes, security releases and cloud computing costs) for Zecwallet’s family of products for 6 months, from Jan 2021 to Jun 2021.

Funding requested is the same as last year - USD 7500/month

Motivation and overview

This grant supports the maintenance and continued releases of Zecwallet’s family of products, which consist of:

  • Zecwallet FullNode
  • Zecwallet Lightclient and Zecwallet Lightclient Library/CLI
  • Zecwallet Android Companion App
  • Zecwallet Lite iOS
  • Zecwallet Lite Android
  • ZecPaperWallet
  • Wormhole Service for Zecwallet Companion Apps
  • LightwalletD service for Zecwallet lightclients

It covers all updates and maintenance, including:

  • Bugfixes and other minor feature requests
  • Security fixes and patches
  • App store fees
  • Keeping up with upstream Zcash protocol changes and zcashd releases
  • AWS server hosting costs for Wormhole service for companion apps
  • AWS server hosting costs for LightwalletD service for Companion Apps.

Technical approach

Monthly Maintenance tasks are:

Zecwallet Fullnode

Maintenance and new releases of Zecwallet Fullnode with embedded zcashd. The main tasks here are operational, compiling/testing new releases of zcashd with Zecwallet Fullnode

  • 8 hrs/month
  • Release/Compile zcashd to windows, macOS and linux. zcashd is officially only supported on Linux, so Zecwallet needs to build and test its own binaries for Windows and MacOS
  • Full test suite (RPC, gtests and bitcoin-tests) * 3 platforms
  • Full sync from scratch * 3 platforms
  • Upgrade from previous version’s .zcash directory * 3 platforms

Zecwallet Liteclient SDKs

This is the lightclient SDK that is used by all Zecwallet Lite UIs (MacOS, Windows, Linux, iOS and Android).

  • 8 hrs/month
  • Keep up with librustzcash changes, network upgrades, checkpoints
  • Desktop app releases
  • Bug fixes and security updates

Mobile (iOS / Android) clients

Google Play store is usually straightforward, but Apple Store updates are always painful. New MacOS/XCode, new iOS versions and app store policies almost always cause new app versions to be painful.

  • 8 hrs/month
  • Updates to SDKs, bugfixes, security updates

LightwalletD infra and devops

This is the main devops task, keeping the Zecwallet LightwalletD servers up and running for the lightclients.

  • 8 hrs / month
  • 6 servers, updates, patches, zcashd version updates
  • LightwalletD code fixes and updates
  • Uptime, load and server monitoring

AWS Server costs

This section describes funds that are spent directly on AWS, which hosts the Lightwallet infrastructure. Note that Zecwallet’s implementation of LightwalletD is more resource heavy than the stock LightwalletD, and so needs bigger machines to deploy.

  • USD 1,500 / month
  • 4x AWS large instances (across 2 regions) for zcashd and LightwalletD (primary + backup)
  • 1x AWS t2.small instance for load balancing, website and other static hosting
  • AWS bandwidth costs (approx 35GB / month), including zcashd serving blocks
  • Devops machines used for testing (zcashd syncing, upgrades), Build machines (3 instances - Mac, Win, Linux)
  • SSL certs, App store developer fees and other miscellaneous one-time costs

Execution risks

Zecwallet has maintained this infra for over a year now, so there are few unknowns. The main risks are around downtime of LightwalletD. We had a couple of incidents last year, but we’ve invested in upgrading our server monitoring and backup/failover policies, that are already incorporated in the infra.

Downsides

Zecwallet Lightclient continues to have a dependency on the (centralized) Zecwallet LightwalletD server. Specifically, this LightwalletD is incompatible with ECC’s LightwalletD implementation for various reasons (See other grant proposal), and needs to be addressed. This proposal only requests funding for 6 months, as by then we hope to transition to a more decentralized model for Lightwallet infrastructure.

Evaluation plan

Success of this project is mainly measured around:

  • Lightclient infrastructure uptime >99.5%
  • Up-to-date releases and version of Zecwallet Fullnode and Lite apps, keeping up with Zcash Network upgrades and upstream bugfixes.

Tasks and schedule

Please see the Technical Approach section.

Budget and justification

Code and Devops :

  • USD 6,000 / month (4 tasks * 8 hrs/month * USD 187.5/hr)
  • USD 1,500 Cloud infrastructure.
6 Likes
2

Perhaps after launching and debugging, it’s time to reduce tariffs, I’m not criticizing just 90,000 per year as for me it’s not a cheap option, I understand that you were the first, but at the moment, personally, my opinion is that the costs should be average for the market.

3

Hi @adityapk00

The @ZcashGrants discussed this proposal today and opted to approve a prorated version of this grant for 2 months pending a discussion around how the ZOMG will fund continued maintenance of projects like Zecwallet and other ecosystem projects. You should receive an email from ZF regarding this portion of the grant.

We want to ensure that projects like Zecwallet are adequately funded into the future, while ensuring that work isn’t duplicated or unnecessarily repeated across the space. For example, you mention that " zcashd is officially only supported on Linux, so Zecwallet needs to build and test its own binaries for Windows and MacOS" - this is obviously work that doesn’t only apply to Zecwallet, and as such the ZOMG would like to understand what it would take to make these builds official so that all ecosystem projects can benefit from this work.

For infrastructure costs, it would be helpful if you could provide receipts/invoices and server utilization data, so we can understand the exact breakdown of costs. For an application that is effectively bandwidth constrained, $1500/month seems particularly high (given that the average cost for a terabyte of bandwidth overage on many providers is in the range of $10). Have you investigated other cloud providers or dedicated hosting?

As far as features go, we understand that the quoted rate essentially captures the cost of “staying in the same place” i.e. the Red Queen’s race - no new major features or work is captured here - is that correct? Do you foresee any way to reduce these costs through automation or other efficiencies?

We would also like to ask about future plans for new features and other initiatives e.g. localization, accessibility support etc. and how those might impact ongoing maintenance costs.

Thank you for your proposal, and we look forward to hearing from you and working with you to ensure the continued success of zecwallet.

4 Likes
4

Respectfully, I will decline the 2 months of funding.

It’s hard to convince engineers and devops folks without long term visibility into a project’s roadmap and funding viability. I work on Zecwallet part time, and I was hoping these funds would allow me to hire and expand the project into a more professional one, but without long term support, it’s not really feasible.

I don’t mean to be flippant - I’m sure that open source projects are viable with this model of funding, just that I don’t know how to build products that way.

Thank you for your time and consideration of this proposal.

6 Likes
5

Hi @adityapk00, it’s your choice of course to decline, but I think you might be misunderstanding what we’re saying here :slight_smile:

To quote Sarah and add my emphasis:

We think that getting on the same page and being able to provide long term funding is very possible.

We wanted to push this 2 months of funding through right away, in this week’s meeting, since we know you have ongoing costs and didn’t want to leave you hanging for those while discussions continue.

Your project is really important to the space, so how we fund it merits a lot of attention. We’re 5 people who meet bi-weekly, so that means we need a little time to figure this out, and also we need to hear responses from you to questions we have.

But we’re very eager to give you long-term funding.

10 Likes
6

Perhaps you could comment a bit more on what you mean by “risk of duplication?”

I can see someone reading that thinking 1) i don’t know of any duplicate efforts right now 2) if ZOMG is worried about it, they must know of non public ones and the only way thats possible is they are supporting it 3) this means they very well might pull my support in 2 months to support something else.

I don’t think this is the case, but I could see someone seeing it that way.

Is this just " you want a plan to make sure you don’t do duplicate funding in general", or you are specifically worried you may have duplicate funding plans for wallets.

1 Like
7

Just after that term I provided an explicit example of work that was listed as part of the grant request which I would term at risk of unnecessary duplication across the ecosystem:

For example, you mention that " zcashd is officially only supported on Linux, so Zecwallet needs to build and test its own binaries for Windows and MacOS" - this is obviously work that doesn’t only apply to Zecwallet, and as such the ZOMG would like to understand what it would take to make these builds official so that all ecosystem projects can benefit from this work."

i.e. It makes very little sense for every single wallet team and other zcash integration to be building their own zcashd from source for common platforms like Windows and MaxOS every single month. If ZOMG is going to be funding things like that then we want to make sure that everyone can benefit from the work, and that the work is being done by the appropriate teams - so that the funding that we do dedicated to specific projects is focused on new features and improvements to that specific project and not on masking holes in other parts of the ecosystem.

3 Likes
8

Thats for the clarification, that does make it clear you are only specifically concerned with those issues.

It seems your objection isn’t to funding in general, just that some of it might be duplicated.
So maybe we can avoid this whole conflict by giving zec-wallet funding on a longer term basis for the non duplicate expenses (devs, some amount of hosting) and then ongoing funding for the other things until the time when they are no longer duplicated?

4 Likes
9

Also, @aditya I just wanted to say I feel really bad about this, and I’m sorry if you’re upset.

The “2 months” thing was my idea, since I thought you were feeling some urgency to get funded and get to work. I completely understand that it would be super disappointing and upsetting, given all the work you’ve done, to think ZOMG wasn’t going to support your work long term.

I’m really sorry it gave you this impression! That wasn’t at all my intent in suggesting the 2 months.

Again, and I know you know this, but partly for everyone else here, my project Zbay uses zecwallet’s SDK, so like all your many users here I’m a stakeholder in your work and want it to go really well.

8 Likes
10

Yes, this! The 2 months thing was my idea for buying us a little time to figure all of this out, without aditya having to shoulder these bills in the meantime for infrastructure he’s providing to everyone, or pause his ongoing work on Zecwallet.

This proposal has way more moving parts than the others we’ve discussed so far, and it combines a lot of different things into one proposal, and it’s right at the center of many users’ (and developers’) use of Zcash so all these details matter a lot.

But whether we do this bridge/short-term funding or not, we should be able to figure this out in the course of 1-2 meetings.

4 Likes
11

To chime in: I too think that Zecwallet is of paramount importance, to both Zcash users and developers. @adityapk00 wrote and maintains two out of the three usable GUI shielded wallets, and his is one of the very few projects that truly exercise and stress-test the Zcash protocol and codebase.

I’m glad that @adityapk00’s work is recognized by @ZcashGrants as crucial enough for immediate funding. I think it’s reasonable for the committee to sync up to learn more about the specifics before committing to long-term funding, in order for example to identify opportunities for pooling of efforts. That said, I hope the committee will recognize Aditya’s excellent discretion in the past in managing collaborations and facilitating code reuse, and thus will provide the requisite leeway, promptness and long-term stability.

17 Likes
12

I just want to echo what @tromer said. But also to point out that even an entire year’s worth of the funding requested in this grant constitutes a small percentage of a single month’s worth of the ZOMG share of the dev fund.

Having been through various grant proposal processes, I’m sympathetic to the frustration that comes with putting in a lot of work to demonstrate a PoC, then having a grant committee withhold funding. Normally this is understandable because the number of fundable proposals is much greater than the funds available. Here we are kind of in the opposite situation. It would be a catastrophe to lose one of the best wallets in the ecosystem over what is basically a rounding error in ZOMG’s budget.

I am not urging ZOMG to spend profligately. But I would love to see better security achieved more through overspending (eg more security reviews, redundant competing projects) rather than withholding funds from projects with a track record.

11 Likes
13

@Matthewdgreen — I agree, but think there’s a misunderstanding here and I just want to make sure we don’t solidify the misunderstanding.

ZOMG’s intent was to approve funding immediately for Zecwallet, at whatever level made sense, while we asked the applicant questions about two interconnected proposals that each have a lot of moving parts.

Just to get a sense of this, just 4 bullets in this proposal have the same related issues as an entire other proposal.

Our meetings are short, we want to give responses fast, and questions were coming up, so we needed more time to discuss, but wanted to ensure ongoing work was funded.

2 months seemed like more than enough time to sort out any questions and arrive at a longer term proposal, so I proposed 2 months, we decided on that, and we asked our questions.

I think Aditya read this as us rejecting the proposal or not wanting to support his work with long-term grants, but I’m not sure how this happened, since Sarah explicitly said we did want to make sure projects like Zecwallet were funded long-term.

2 Likes
15

Also, I think we all need to recognize that there are some big challenges inherent in communicating about sensitive topics (like beloved projects, funding, and livelihoods) via an asynchronous written medium, as an ad hoc committee of 5 people.

It’s really easy in written online communication to perceive conflicts or assume the worst of each other.

I think if these things were communicated in a phone call with the group, we would have caught this misunderstanding in the flow of conversation before it became anything. But it’s important that communication with grantees is clear and auditable by the community. And the forum seems like the best way to do that for now.

So I think we all need to take a deep breath and really try to understand each other.

4 Likes
16

Also, @adityapk00 I totally would have reached out to you personally about this, but we want to avoid private backchannels with applicants, so I’m putting what I’d say to you here!

Again, I’m really sorry that my proposal for 2 months of funding upset you and I hope you get that we want to support your work on Zecwallet long term. :heart:

1 Like
17

@ZcashGrants This is a very important point that needs to be considered going forward. Zcash development interests are still small at this stage compared with other chains with dev funding that have expanded programs for community efforts, meetups, Dapp ideas and a wide range of active funding hand outs with an understandably low success rate for each project.

With Zcash, the Dev fund allowed us to have a solid dev fund to attract new developers and businesses to apply for grants. I like that we are being extra careful with spending the future of private money by being very questioning of every grant, but we need to be able to identify and do what is required to build a solid base.

Aditya made a very important point of the challenge+work required for attracting developers to contribute to work on the bleeding edge of cryptographic applications. I second that and I’ve found it very challenging to set up a roadmap and attract capable devs and paying them a fair share for their work. Additionally, once these projects are out there, they need funding to be maintained with the ever-changing supported platforms, additional QA testing, responding to issues on GitHub and social media accounts. I’ve had my fair share of dealing with this, and I believe there needs to be enough incentives provided to enable the growth of the ecosystem at this stage.

Going forward, please consider funding these large maintenance of widely used projects as Zcash is still small. We might be in a different place in a few years, but a lot of groundwork needs to be laid out till then.

6 Likes
18
1 Like
19

Again, I’m really sorry that my proposal for 2 months of funding upset you and I hope you get that we want to support your work on Zecwallet long term.

My reading on this was different. Four minutes before posting in this topic, Aditya posted the following in the related proposal, in response to that proposal being put on hold:

Of course, without the Zecwallet Lite SDK, Zecwallet also cannot exist, so this decision also makes Zecwallet’s other proposal moot.

In comparison to the Zecwallet Lite SDK being put on hold, the 2 month thing seemed to be less of an issue. Of course, that was just my interpretation after having read both threads (which, side-note, is a considerable chunk of time for all of us–perhaps there is a better way??)

I agree wholeheartedly with @holmesworcester that, fundamentally, this just sounds like a breakdown in communication that could probably be mitigated in the future as the proposal process matures.

Let’s keep in mind ZOMG just came on-line in December.

It’s January. Growing pains are to be expected.

5 Likes
20

I think you’re totally right to point to phrase “put on hold” as a partial cause of the misunderstanding. On its own, it reads negative or like a rejection.

I think in terms of the selection process documentation technically the phase we were in is “proceeding with diligence” and “requesting improvements to the application.”

That language would have been more clear.

That said, what we said was “put this proposal on hold, pending the questions below,” after which follows two clear questions.

So in that sense it was clear that we were not rejecting, but rather asking questions.

Yes! All of us are—by the wisdom of the ZIP and the democratic nature of the process—completely new to our roles, and all applicants are the unfortunate guinea pigs / early adopters! :slight_smile:

We can discuss what the official language we use is for describing stages in the process at the next ZOMG meeting, so that we’re more standardized on some battle-tested phrasing next time.

2 Likes
21

And @holmesworcester after re-reading my note I want to emphasize that I’m not pointing fingers. Rather I want to empower the ZOMG. More concretely, I want to make it clear that, from my perspective as a community member, you should feel free to put the spending pedal to the metal on things that are even arguably worth it. If spending gets super out of control, in a few months the community will let you know to rein it in. But right now just go, go, go.

4 Likes