2 years of Lightwalletd Infra hosting & maintenance

Community Grants Applications
1

Zcash needs stable, reliable & well maintained test-net and main-net infrastructure for use by developers and apps.

Following @ZcashGrants initiative to fund projects to help Zcash ecosystem, please provide your feedback and ask questions for a long term test-net and main-net infrastructure setup with 3 high capacity, highly available light wallet servers for use by developers & apps in the Zcash ecosystem.

Nighthawk Wallet Team has 20+ years of combined experience, with dev-ops led by Vamsi who has 10+ years experience in building & deploying scalable software with experience in handling peak demands, load balancing and triaging real time issues. Vamsi has helped setup, maintain and recently migrate Nighthawk’s lightwalletd server securely, and with minimum downtime to an enterprise grade cloud service provider.

With this long term grant, we want to use our experience of deploying stable versions of lightwalletd to make Zcash services available to developers & applications in the Zcash ecosystem and maintain the services with security patches right from the platform to the endpoints.

Motivation and overview

The goal of this grant is to make available Zcash infrastructure as a test-net sandbox for new developers and stable main-net infrastructure for existing applications like Nighthawk Wallet and Zbay. This goal is in line with ZF’s mission of enabling R&D and community involvement while opening up access to Zcash blockchain to other cryptocurrencies and oracles. This operation will serve the Zcash light wallet and services users.

Technical approach

  • Set-up 3 large 32GB firewall configured instances running on Ubuntu, Zcashd and Lightwalletd tagged releases.
  • 1 instance configured for test-net and 2 load balanced instances configured for main-net use.
  • Buy a domain with pre-payment for 2 years for accessing lightwalletd infra.
  • Set-up a Status page to reflect the uptime of the light walled services.
  • Set-up a GitHub page with instructions to connect to lightwalletd.
  • Create documentation for community members to automate setting up similar instances via Ansible Playbooks.
  • Create an email to receive support requests and communications and aim for a 24 to 48hr turn-around by being available in Zcash Community Discord & Telegram groups.

Execution risks

  • Faults in the lightwalletd software require frequent patching to restore to working versions.
  • Excess server loads resulting in frequent peaks and requiring spinning up of new server instances and/or setup a proxy based load balancer.
  • Availability and Support from ECC Engineers to help debug and and fix issues real-time.

By having a status page and a connection to the community via social media and Zcash Form channels, we plan to keep up to date with the ongoing changes, planned & un-planned software upgrades.

Downsides

While Zcashd is a decentralized service, the Lightwalletd endpoint is centralized and may be subject to attacks and takedowns. We will have a contingency plan to switch to new-endpoints to triage issues that arise along the way. On the Availability front, issues in chain syncing in lightwalletd and Zcash upgrades need extra attention during specific times where the dev-ops may not be responsive even when Nighthawk Team members are present across North America & India time zones.

Doubts around integrity and privacy may arise and we will do our best to be transparent with any government/state requests that comes our way. Otherwise, we will adhere to no permanent logging policy (while having logging enabled for debugging purposes only).

On the stability front, we can choose the best SLA enterprise contracts, but even the best cloud providers have suffered massive unexpected downtimes which are out of our control. And continuous increased usage will require additional funding for servers.

Evaluation plan

Having a well funded infrastructure for Zcash will attract developers to test drive and create applications year round and deploy them knowing they can connect to stable services.

While we don’t have year long uptime stats for Zcash services, we will be aiming for high 9s for the uptime that might be affected by software issues, maintenance turn-around time and data center issues.

  • 99.8% 17.52 hours per year of downtime
  • 99.9% 8.76 hours per year of downtime
  • 99.99% 52.56 minutes per year of downtime

Tasks and schedule

First month:

  • Sign long term server usage contracts.
  • Spin up instances and start configurations, syncing and backup security certs.
  • Buy and configure domain name to resolve to lightwalletd servers.
  • Verify end-points, test response chain info data to match with community servers.

Second month:

  • Work on load-balancing strategy for GRPC/HTTP services.
  • Set up documentation, landing pages and status page before making the lightwalled services available for public use.

Budget and justification

The operational costs are: $1,300/month, broken down as:

  • $900: $300 x 3 Large instances + Large Storage for indexed Zcash blockchain.
  • $300: 6 x $50/hr Dev-ops costs to support upkeep and monitor lighthttpd service, certs.
  • $100: Setting up domains, landing & status pages and community outreach.

Other-one time costs: Setting up a status page, buying a domain with 2 year pre-payment

Total: $1,300 * 24 months = $31,200

12 Likes
2

Sorry, but this work includes all the same (and a little more) that in the other offer for 13140 for 3 months but will cost 31200 for 24 months?

3

The primary costs are related to the cloud server rates that are generally valid for 12 months out. Rest of the operational costs are distributed around the 24 month time frame.

Note that these servers will be open for any application to use and Nighthawk Wallet will use the capacity from these servers too as the prior funding runs till March https://grants.zfnd.org/proposals/68882602

1 Like
4

I am expressing my support for this grant proposal. I think having a reliable public lightwalletd instance is a must if we expect more adoptions. Most people will not run their own full node so they will have to depend on public lightwalletd.

However, I have some reservations for the test-net. Is it that crucial to have a public lightwalletd instance for test-net? If so, can a smaller instance (say, medium instances instead of large) satisfy that need?

Cheers!

4 Likes
5

Thank you @tokidoki for your Question.

You’re right that test-net will not see medium to heavy usage, which gives us the capacity to run the main-net status page, diagnostics and other testing related services on the test-net server. And running test-net requires running a full zcashd and lightwalletd.

A reliable public test-net server is required for development purposes.
And there is no harm in having excess capacity in the test-net server as it can serve as good service for load-testing and for development during hackathons. Also, there isn’t much price difference from the medium instances.

3 Likes
6

A no brainer proposal.

5 Likes
7

This looks like a beneficial proposal to me as well. I’m feeling happy to see our original proposal sparking additional interest in providing similar and complementary services.

If an expanded version of our proposal is approved, it’s my hope we can collaborate to operate an infrastructure diverse, highly available and secure set of services to benefit the Zcash community.

2 Likes
8

Some initial thoughts, prior to committee discussion:

I’ll put it out there that my main concern with the current batch of lightwallet infrastructure grant requests is that I think the future needs to ere towards enabling hundreds of low cost, low trusted options rather than tens of well maintained options with expensive maintenance costs.

That being said, I’m definitely willing to consider short term proposals to jump start that ecosystem and lay the foundation for others to follow - but - I would argue that if ZOMG is to consider funding longer term maintenance of nodes, then requests should ensure adequate funding to assure the privacy and security of those nodes. In particular around this proposal, I’d definitely like to see some more rigorous thought around the following point:

Lightwallet servers are privy to a lot of interesting metadata and I would venture that even “debug” logging verges on irresponsible in a production environment (depending on the jurisdictions involved - relatedly, have you put thought into where these servers would be physically located?)

While privacy-audits on services that consider things like logging aren’t all that particular assuring from a user perspective, it might be something to consider from a service-provider perspective.

It stands to reason that any set of servers funded by ZOMG will be defacto-more-trusted than others, and my main priority in assessing any proposal on that would be ensuring that such a default assumption would be justified.

5 Likes
9

Thank you for your review and concerns.
Nighthawk team has taken a strong stance against logging, even disabling the base debian system level logging on the cloud servers. And the only instance where we had to turn on logging in prod was to debug a chain state sync issue following a bug in lightwalletd code which was preventing accurate headers from syncing, we applied the patch to lightwalletd and disabled logging again immediately.

The server location is Toronto, Canada.

I couldn’t agree more and am willing to coordinate on audits and walkthrough of the implementation once ready and take feedback to immediately fix any concerns that arise from the audits.

2 Likes
10

Hi @aiyadt

Thank you for your answers to my questions.

This proposal was discussed by the @ZcashGrants during today’s meeting, and we are opting to provisionally move forward with this proposal pending a few additional questions and due diligence, with a definite recommendation to be voted on during the next committee meeting on the 20th of January.

Feedback and Additional Questions

On the future of light wallet infrastructure

One of the main concerns discussed centered around the strategy for operating light wallet infrastructure in the long term. ZOMG wants to make clear that any funding we provide for such infrastructure is, by its very nature, temporary. It is likely that we will either see light wallet infrastructure trend towards an entirely p2p operated (decentralized and distributed) or concentrated in a handful of well trusted, highly scrutinized entities.

We note that this proposal falls, somewhat awkwardly, in the middle of these two extremes - as it must based on the current state of light wallets - and the ongoing discussions about possible alternative designs.

  • What are your teams views on the longer term vision for light wallet infrastructure?
  • Have you thought about how you would use the funding provided if changes were made to the protocol, during your 2 year term, that fundamentally changed the nature of light wallet infrastructure (e.g. moving towards p2p or an ecosystem push for high-assurance, centralized nodes)?
  • Given the current state of discussions around the future of the light wallet protocol how did you settle on the 2 year horizon for hosting?

On branding, security and privacy

Related to above, approval of this proposal would result in a set of ZOMG funded, light wallet infrastructure. This in particular raises a concern around branding as any potential security or privacy issue would not only reflect on your team but also on the wider Zcash ecosystem.

Your proposal identified some of these risks, but was light on details around handling them.

  • Does your team have experience handling responding to requests for information from 3rd parties in a service environment?
  • Why did you settle on Toronto, Canada for the physical hosting of the servers?
  • Do you think that the amount of funding that you have requested is enough to assure the privacy and security of the infrastructure (against reasonable risks associated with light wallet infrastructure?).
  • Please also detail the measures you would take to provide that assurance.

Thank you again, and we look forward to your response,

Sarah

3 Likes
11

:+1: It would be a true victory for Zcash to run lightwalletd Infra on a P2P/SPV/Neutrino like decentralized service in the long run.

Definitely leading towards a decentralized lightwalletd framework for wallet software, but ZOMG funding for maintaining endpoints that host entry/exits for Oracles for use by other blockchains’ smart contracts.

Since the funding is primarily securing long term server contracts, the maintenance hrs will be used to cater to switching/migrating/repurposing the infra for community needs with priority to keep the wallet/services that depend on it running.

The 2yr contract is the minimum for a discounted long term contract on cloud providers and it also helps with assurance to the apps that connect to the lightwatted infra.

No, we plan to rely on support from ZF/Zcash community.

Proximity to the North American audience while not being in the U.S.

Following the maintenance requirements for Nighthawk Wallet Servers in the past 3 months, the funding should be adequate. But if there are spikes or requirement changes, we will apply for further funding to handle operations. As for assuring privacy and security of the infra, we plan to follow the industry wide best practices for security and privacy of data (although there is no user data stored with use of lightwalletd). I will request @vamsi the lead dev-ops to provide input on this part.

1 Like
12

@aiyadt to clarify this one, I think “3rd party requests for information” means things like law enforcement or court systems in different countries requesting information.

These requests will come to whoever’s name is on the server, and the issues you’ll be dealing with will be in the jurisdiction you live in, in addition to the jurisdiction where the server is in, I think, though I’m not a lawyer. The important thing is to have clear internal policies, and an existing relationship with a lawyer to turn to in case something happens and you have questions. It’s probably a best practice to implement a warrant canary as well, in case a request for information comes with a gag order.

One thing that small projects at the cutting edge of tech / privacy / autonomy do is reach out to a civil society organization like the EFF. They’re often able to give some orientation or connect you with a pro bono lawyer in some cases too. I could make an introduction if that’d be helpful.

The Berkman Klein Center Cyberlaw Clinic is another potential resource, though I know they’re really busy and more project-based so it’d be more helpful for setup than dealing with specific incidents I think. I could ask there as well.

One interesting question here is whether ZOMG can fund legal support for grantees, or if we might want to do that in a similar fashion to how we’re considering providing security support.

4 Likes
13

I don’t think its a good use of funds to pay this much for infrastructure that harms user privacy.

I would much rather see a proposal for all existing lightwallets to add an option (prominent in the UI) to connect to a remote full node controlled by the user and add warnings of the consequences of using a node controlled by someone else

14

The infra costs are actually bare minimal that we can go for this long term contract. We might need to update the numbers considering the security protocol we need to put in place.

FYI @NighthawkApps was the first to create a custom settings page to allow the user to enter their lightwalletd server of choice in android. I believe, having an experienced team, supportive of the Zcash mission, security audits and follow security patching protocols run the base infra for sandboxing and allowing free apps in the Zcash ecosystem to use the ZOMG funded infra is a better option than random lightwalletd’s showing up for users to connect to.

We do have work around adding tor support pending which we will explore in another grant.

3 Likes
15

Thanks for the explanation. I will be happy to test any and all mobile wallets that are

  1. Open source (F-Droid support and apk options would increase adoption for those too lazy to compile from source)
  2. Ability to connect to my own full node (either remote full node or one I run on my phone itself)
  3. Shielded support (exclusively)

I think T address support is the thing that is hurting Zcash shielded adoption the most, but I dislike lightwallets too. If 3rd party developers want to use them it should be at their own expense IMHO.

There are both pros and cons to more decentralized but “random lightwalletd” infrastructure. I dont think funding centralized lightwalletd infrastructure is a good use of funds In my opinion we can help users stay more by encouraging them to run and connect to their own full nodes.

2 Likes
16

@aiyadt Good news! We’re approving your application, with these stipulations:

  1. That you publish a document (e.g. a blog post) describing your policy in cases where you receive a 3rd party information request, such as a subpoena.
  2. That you implement a warrant canary.
  3. That you add these to the deliverables for an early milestone. (You may also want to add other details from the proposal to the milestones, or add more milestones, for clarity.)

The next step in the process is to proceed through the KYC process with Zcash Foundation.

4 Likes
17

Also, I wanted to respond to the comment above, just to not leave it hanging…

This is true, but given that the working lightwallets and SDKs we have right now use centralized infrastructure like this, and given that ensuring privacy in a decentralized lightwallet scenario will take much more time and resources, we felt it was important to fund this now.

We also welcome proposals to make lightwallets more decentralized, in privacy-preserving ways, and to make centralized lightwallet setups more private.

Since part of ZOMG’s goal is to drive Zcash adoption and make it more accessible, we want to remove barriers to using Zcash.

The very large disk space requirements and long syncing time for running a full node is definitely a barrier to adoption. That said, apps like Zecwallet let you connect to your own full node if you want that. So it is an option for users who want it.

1 Like
18

Thank You. I have updated the Grant to include 3rd party information request sharing policy on the landing page & plan to start maintaining a warranty canary monthly. Additionally, I have added a budget of $150 monthly to make time for security audits(per discussion in this thread) and to maintain the warranty canary. Aldo added the same to the Milestone 2, following which the service goes live.

19

Would you be willing to run the blockchains on HSM’s?

would you like a hand security testing?

2 Likes
20

HSM solutions are not part of the setup we have planned. We can review and evaluate the benefits of HSM and follow up if we plan to undertake it. I would appreciate if you can point out to resources and pros/cons of using HSN for blockchain security.

As for security testing, I will DM you within a month of our initial setup to get your inputs. cc @vamsi
Following which we will be open for the security audits as discussed in this thread.