RFP - Zcash Lightwalletd Infrastructure Development and Maintenance

Request for Proposal (RFP) for Zcash Lightwalletd Infrastructure Development and Maintenance

Project Overview

The Zcash Community Grants (ZCG) committee is inviting submissions for an essential initiative focused on the development, deployment, and ongoing maintenance of Zcash lightwalletd infrastructure. ZCG is interested in decentralizing and distributing infrastructure amongst multiple providers and is open to approving more than one proposal. This project is crucial for improving the performance, scalability, and security of lightweight Zcash wallets, facilitating their interaction with the Zcash blockchain for shielded transactions without necessitating a full node. The aim is to provide a seamless, reliable, and secure service that meets the expanding requirements of the Zcash ecosystem.

Background Information

Zcash is a privacy-centric digital currency, founded on strong scientific research. The lightwalletd service plays a fundamental role in the Zcash infrastructure, enabling lightweight wallets to execute shielded transactions efficiently. This service is key to Zcash’s scalability and accessibility, allowing users to benefit from privacy without the need to operate full nodes.

Scope of Work

The selected vendor(s) will be responsible for:

  • Infrastructure Architecture Design: The vendor will design an architecture that is scalable, secure, and resilient, tailored for the Zcash lightwalletd service. This includes configuring servers, databases, load balancers, etc., to ensure high availability, fault tolerance, and geographical redundancy. The design must support growing transaction volumes and user connections, with strategies for scaling to meet demand and deploying across regions to maintain uninterrupted service.

  • Implementation of Security Measures: Robust security measures will be integrated to protect against DDoS attacks, unauthorized data access, and vulnerabilities. This includes data encryption, secure authentication, comprehensive security policies, active monitoring, and rapid response protocols to mitigate security incidents.

  • Security Measures: Implementing comprehensive security measures to defend against DDoS attacks, unauthorized data access, and other cyber threats.

  • Testing and Quality Assurance: The infrastructure will undergo testing for scalability, security, and reliability, including load testing, security penetration testing, and disaster recovery testing. Quality assurance processes will ensure continuous performance and security monitoring, with regular updates and patches for new threats and Zcash network compatibility.

  • Deployment Strategy: Establishing infrastructure within a cloud-based environment, it is crucial to ensure geographical redundancy and high availability across a minimum set of global regions—including North America, South America, Europe, EMEA (Europe, the Middle East, and Africa), and Asia—to provide uninterrupted service.

  • Maintenance and Upgrades: Offering ongoing maintenance, active monitoring, and prompt upgrades to the lightwalletd service to ensure it remains compatible with Zcash network updates.

  • Incident Response Plan: Developing and implementing an incident response plan that includes procedures for identifying, responding to, and recovering from cybersecurity incidents.

Proposal Requirements

Prospective vendors should demonstrate:

  • Expertise in cloud-based infrastructure design and management, emphasizing security, scalability, and reliability.

  • Experience in the deployment and maintenance of critical network services, with a preference for those in the cryptocurrency sector.

  • A solid grasp of cybersecurity measures and best practices for countering modern threats.

  • Previous work with blockchain technologies, with experience in Zcash being an advantage.

Submission Guidelines

Submissions must include:

  • Proposal Document: A detailed proposal that outlines the approach for infrastructure development, deployment, maintenance, and security, consistent with the Scope of Work.

  • Budget & Timeline: An itemized budget and a project timeline, adhering to a maximum budget of $50k for the year per applicant and considering the possibility of selecting multiple applicants.

  • Professional Qualifications: Documentation of the vendor’s professional history, highlighting experience pertinent to this project.

Evaluation Criteria

Proposals will be evaluated based on:

  • The vendor’s technical capacity and track record with similar projects.

  • The proposed solution’s fit with the Zcash ecosystem’s goals and needs.

  • The cost-effectiveness and efficiency of the proposed budget and timeline.

  • Prior experience with blockchain and cryptographic technologies, especially within the Zcash ecosystem.

Additional Requirements

Applicants are encouraged to detail:

  • The global distribution of infrastructure to ensure robustness and fault tolerance.

  • Uptime guarantees and the penalties for failing to meet these guarantees or in the event of excessive latency.

  • A detailed breakdown of costs, including setup, maintenance, and any variable costs, should be clearly outlined.

  • Applicants are expected to list the third-party providers they plan to use, including cloud services, networking, and any other critical infrastructure services.

  • The proposal should clearly define its limits and liabilities, highlighting what is not covered within the scope of the project.

  • Information about the team composition is crucial, including the expertise and roles of team members contributing to the project.

  • Detailed procedures for implementation, monitoring, and maintenance must be included, along with any other information critical for a comprehensive evaluation of the proposal’s scope and execution plan.

7 Likes

Hello! Our proposal is below.


Proposal for Zcash Lightwalletd Infrastructure Development and Maintenance

Executive Summary

This proposal outlines our approach to developing, deploying, and maintaining Zcash lightwalletd infrastructure in a reproducible, open architecture.

Our plan leverages our existing server hardware in Texas and NYC datacenters and suggests potential sites in Latin America, Europe, and Asia to ensure a secure, scalable, and resilient service for the Zcash ecosystem. We aim to deliver this project within a budget of $45,000 for three regions, with an option to expand to a fourth region (Asia) for an additional $15,000.

Team Background

Our team has extensive experience with architecting, deploying and maintaining scalable application infrastructure. Our lead engineer has experience working as a senior site reliability engineer and currently maintains self-hosted nodes for eleven different blockchains, all self-hosted on owned hardware and orchestrated with Kubernetes. They wish to keep their identity private in this initial proposal but will share their identity with Zcash Foundation.

Zancas is co-applicant and is a known contributor to the Zcash ecosystem.

Approach and Scope of Work

Architecture

  • Our design avoids major public clouds and instead uses Kubernetes on dedicated hardware.
  • The infrastructure will be distributed across three to four regions: USA, Europe, Latin America, and optionally Asia, each with at least two physical dedicated servers operating in a cluster.
  • We will use hardware in our existing datacenter for the USA region. In non-USA regions we will lease dedicated servers from major datacenters.
  • Each region will have sufficient excess capacity to enable upgrades and maintenance without downtime.
  • We will employ a global load balancer, likely Fly.io, to announce anycast IP addresses which distribute traffic across regions efficiently.
  • A public cloud to be determined will be used for DNS resolution only, without enabling any man-in-the-middle proxying. Public clouds have impressive DNS infrastructure. Use of centralized DNS does not impose a significant threat to user privacy in accordance with our threat model, and increases user security to use a known provider with a proven security track record.
  • A public status page will be maintained.

Security Measures

  • The threat model of hosting open source software and public blockchain data does not involve hosting personally-identifiable or otherwise sensitive information, and as such is relatively low-risk.
  • We will follow best practices to keep all servers up-to-date with the latest security patches and will disallow password authentication on all servers.
  • Privacy is paramount; no logs will be taken of any user IP addresses, public keys, or wallet addresses.
  • Bandwidth and connection counts will be monitored for observability purposes, but with no logging of user-identifiable metadata whatsoever.
  • We will develop and implement an incident response plan to rapidly address and mitigate any incidents.

Testing and Quality Assurance

  • We will use open source software wherever possible to observe the reliability of the global infrastructure.
  • We will use continuous integration to ensure that our build and deployment processes are repeatable, predictable, and result in infrastructure that operates as expected.

Deployment Strategy

  • Our deployment strategy emphasizes geographical redundancy and high availability, with a minimum bandwidth of 1 gigabit for each region, and relying heavily on Kubernetes best practices.
  • We avoid centralized public clouds wherever possible to increase user privacy and project independence.
  • A global load balancing endpoint will be maintained that directs traffic to a user’s closest region.
  • A Tor Hidden Service (.onion) will be maintained that also distributes traffic to the least-burdened region
  • The team does not yet have experience with Nym but will spend up to 10 hours towards launching a custom service within the scope of this grant, documenting the progress and coordinating with Nym’s team to launch a proof-of-concept endpoint.

Maintenance and Upgrades

  • We will apply security upgrades in a timely manner and coordinate with the development teams of Zcashd and Lightwalletd to ensure that we build and deploy new versions quickly when they become available.
  • We will deploy excess capacity in each region with a goal of zero-downtime upgrades.

Accessible and Reproducible Infrastructure

  • Our goal is to launch reproducible infrastructure that anyone can host.
  • We will build openly and release documented open source software to simplify launching Lightwalletd infrastructure. For example, we will open source Kubernetes helm charts to deploy Zcashd and Lightwalletd to any Kubernetes cloud, including self-hosted infrastructure like ours.
  • We will use existing official containers wherever possible but may maintain our own container builds if the maintainers of our dependencies are not releasing updates quickly enough for our security standards.

Budget & Timeline

  • We propose a budget of $45,000 evenly distributed in quarterly installments each paid up-front. This would enable us to host three regions with two dedicated servers hosted in each of them for one year, and a global load balancer.
  • We estimate that each additional region would cost $15,000/year to deploy. Some regions such as Australia may cost more due to bandwidth pricing realities in different geographic areas.
  • The USA region will launch within 3 weeks of receiving the first installment.
  • Global regions would launch within 6 weeks of receiving the first installment.
  • Infrastructure-as-code, including Kubernetes helm charts and container builds, will be built in the open on public Github repositories and released on an ongoing basis.

Circular Economy

  • We will use third-party vendors that accept cryptocurrency wherever possible, including datacenters.
  • When a provider does not yet accept Zcash we will advocate that they consider it, educating them if there is interest.

Risks

  • Using Fly.io as a global load balancer may incur bandwidth fees beyond the scope of this grant. We may switch to another provider or need to iterate on our load balancing approach, or request additional funding if egress bandwidth fees surpass the budget allotted.
  • Large changes to storage requirements may change the expected costs and require physical hardware upgrades, especially during major upcoming upgrades such as the migration away from Zcashd. We consider this to be a low risk and are targeting at least 2TB of server storage per cluster.
  • Network-level DDoS attacks may affect our costs. We are architecting for DDoS resilience wherever possible and are selecting datacenters that have existing DDoS mitigation hardware that could be employed in the event of an attack.
  • A downside to physical hardware is that it does not automatically scale. Over-provisioning is necessary. Scaling physical hardware requires a few days of lead time and could impact our budget. To mitigate this risk we will ensure that our Kubernetes infrastructure-as-code is compatible with a major public cloud’s Kubernetes offering. In a worst-case we could sacrifice our self-hosting ethos and flip the load balancer over to a public cloud. This would incur costs out of the scope of this grant but allow the infrastructure to scale far beyond this initial deployment for a positive user experience.

Conclusion

Our proposal represents a comprehensive approach to enhancing the Zcash lightwalletd infrastructure, emphasizing self-sovereignty, open repeatability, security, and scalability. By leveraging our team’s existing hardware, we offer a cost-effective solution that aligns with the Zcash ecosystem’s goals and needs.

5 Likes

Welcome @emersonian !! I am very excited about your proposal. I hope it gets funded!

4 Likes

The proposal says you are a co-applicant @zancas…Your reply makes is sound like you are not associated with the proposal. Are you a co-applicant?

From the proposal

Seems to me that ECC should take over all wallet related funding given Zashi and the new roadmap, and ZCG should focus more on Qedit funding for ZSAs, and Stablecoins. No more third party single asset wallet funding by ZCG until we get ZSAs, deprecated ZcashD, Stablcoins, POS, programmability.
(and I dont think we ever should be funding any single asset wallets outside Zashi)…ECC should lead the charge on wallets. Does ECC need this RFP for Zashi?

The proposal looks very suspicious.

  1. A wallet RFP from ZCG when ECC is working on Zashi who should be the ones determining what is needed for a wallet and the timing.
  2. Less than 1 day after the RFP is released; Emersonian has a very detailed proposal.
  3. Emersonian created his profile 1/29/24 or about 6 weeks ago.
  4. Emersonian wants to keep his identity private; yet the speed and detail implies Emersonian had a heads up this RFP was coming.
  5. The creator of the proposal refers to themselves as “they” inferring the team doing the work didn’t write this proposal. So is Emersonian acting as an agent for the team doing the work?
  6. Zancas is a co-applicant per the proposal; but Zancas acts like he has never seen the proposal before.
  7. The team does not have experience with Nym; but they are wanting to do work related to Nym.
  8. The proposal is including ZcashD work when we are trying to deprecate ZcashD.
  9. Many risks costs will be higher.
5 Likes

Hi Jgx7! Thank you for reviewing the proposal. Some clarifications are in order, inlined below.

No more third party single asset wallet funding by ZCG until we get ZSAs, deprecated ZcashD, Stablcoins, POS, programmability.

This is not a proposal for a wallet. Our proposed Lightwalletd Kubernetes infrastructure as code makes it easier for the entire ecosystem to host their own Lightwalletd infrastructure, this is a win for everyone. We will use this code to host a minimum of six physical dedicated servers in three regions, “dogfooding” the Kubernetes approach to make sure it is production-tested for community use during and long after the scope of this grant expires.

  1. A wallet RFP from ZCG when ECC is working on Zashi who should be the ones determining what is needed for a wallet and the timing.

This proposal is for lightwalletd infrastructure that all wallets in the ecosystem will benefit from.

  1. Less than 1 day after the RFP is released; Emersonian has a very detailed proposal.

I have been considering applying for a grant for over a year and this RFP is the closest match to my expertise.

  1. Emersonian created his profile 1/29/24 or about 6 weeks ago.

I am a long time lurker and Zcash user. I recently decided to take a more active role in contributing as an experienced user.

  1. Emersonian wants to keep his identity private; yet the speed and detail implies Emersonian had a heads up this RFP was coming.

I highly value my privacy and believe that a community will respect that. I am a new contributor to this project. I am no insider, and I am open to contributing openly under my name in the future. Right now I just want to focus and improve the server infrastructure for this project.

  1. The creator of the proposal refers to themselves as “they” inferring the team doing the work didn’t write this proposal. So is Emersonian acting as an agent for the team doing the work?

I am the primary contributor for this proposal. I may loop in a few other engineer friends here and there to troubleshoot or unblock me and wanted to keep the tone neutral and professional without gendering the team.

  1. Zancas is a co-applicant per the proposal; but Zancas acts like he has never seen the proposal before.

I know Zancas and he respects my desire for privacy. I think it’s pretty awesome that someone can contribute here pseudonymously in a pure meritocracy, he’s been encouraging me for a long time to contribute. I thought it would be unlikely that an anon could land a grant so I asked if he would be a co-applicant, though perhaps anons land grants here all the time.

  1. The team does not have experience with Nym; but they are wanting to do work related to Nym.

I don’t know anyone who has experience with Nym. I thought it would be cool to learn within the scope of this project. I have never seen or used a Nym custom service which I think is a real problem for Nym to ever get significant adoption. So, let’s experiment with it and try to push both communities forward with one project.

  1. The proposal is including Zcashd work when we are trying to deprecate Zcashd.

Can lightwalletd work today without Zcashd? If so I’m happy to adjust the proposal.

  1. Many risks costs will be higher.

To be clear the foundation will incur no additional costs within the scope of this grant. It is possible that a spike in traffic usage slows down the new server infrastructure.

What would happen is that the lightwalled infrastructure deployed as part of this project would slow down or go offline due to hardware-limited constraints. In this scenario I could not afford to eat the costs of scaling, and would potentially ask the community and/or the foundation, for help. This is an edge case that I do not anticipate happening.

5 Likes

Hi emersonian,

thanks for your interest and putting your ideas out in the open to be debated and more importantly actionable.

9) Many Risks costs will be higher
The infrastructure in your plan, how many light clients can it service? Is there more costs incurred to your infrastructure based on the type of light client connecting to your service (ie. ios app, android app, desktop app, cloud machine) ? What type of activities from the connecting clients cost your infrastructure more money? Observing the other two lightwalletd service providers in the ecosystem what methods will you deploy to make your solution more cost competitive?

3 Likes

Thanks for your detailed and prompt review on this proposal.

TL;DR:

  • I endorse the trustworthiness of @emersonian
  • this is not funding for a wallet
  • I am not a potential grant recipient
  • I am the insider. I 100% was aware of the draft RFP, and pushed @emersonian (and every other expert that I knew) to apply for it, this in no-way inhibits other applicants. The RFP aspires to fund multiple proposals.
  • If there’s an appearance of favoritism, or anti-competitiveness, that’s on me. I believe that this work is urgent, and should be funded as quickly as possible. Please tell your hardened-infrastructure Zcash enthusiast friends to apply.

Slower Recap

I believe that this work is essential to the health of the lightclient network, and therefore Zcash.

I am not an author on this proposal, nor would I receive any funds if the proposal were approved.

I know @emersonian and endorse them as a trustworthy contributor who can support a critical need for the whole community.

I am unclear on where you perceive this “act”. I absolutely have seen the proposal before and certainly did push @emersonian to apply with it. I am not a “co-applicant” in any financial sense.

Are you concerned about potential applicants being unfairly scooped?

I was aware that this RFP would be published because I was asked to review it. I consulted with every expert I knew in the space, including @emersonian . What’s the issue?

Finally, I really appreciate the skepticism and diligence that @Jgx7 brings to the forum. I’d appreciate their feedback on Zingo’s latest release which I believe has bearing on some concerns about optimal resource allocation.

4 Likes

I think it’s great you want to build a third party app on top of zcash. with that, I have several concerns because it’s funded by the dev fund.

  1. you are a grant recipient. so it appears you collaborated with ZCG to create the RFP. So my concern is grant recipients may have captured the ZCG. It appears you need this for Zingo when we/ECC are already funding Zashi. We only need 1 wallet in my opinion.

  2. In my view, ZEC is fiat for day to day spending vision will drive ZEC to single digits. We must change the vision. This vision has caused a misallocation of resources to single asset third party wallets. The cost to support Zingo, Nighthawk, Ywallet and Zashi is too great. So we need to focus on Zashi only when it comes to single asset third party wallets.

  3. I think if you want a wallet for other countries, it will be cheaper to localize zashi when it’s ready than create a wallet like zingo. I think it’s been proven now that more wallets doesn’t equate to more users or more zec transactions.

  4. i think 100% all resources should be focused on deprecating ZcashD, rolling out ZSAs, and stablecoins, then integrating them into zashi. Then move
    to POS. after that is done, then see where we are at. Until then, no more wallet funding because it doesn’t add value in my view. We can risk the entire project when so much critical blockchain work is in front of us. ECC should be leading the charge on wallets now. We shouldn’t have third party wallets out in front of Zashi.

  5. i’d love to see a gas/fee structure so you could charge users to use Zingo. then you can make as much money as you want and not be dependent on begging for money from the dev fund. Until then, ECC should be deciding what gets done for wallets–zashi.

1 Like

There seems to be some ongoing confusion here:

Zashi will NOT work, without lightwalletd infrastructure.

Without the infrastructure that this RFP aims to support, there would not be 1 lightwallet… there would be 0.

Are you advocating for a 0-lightwallet system?

2 Likes

I fundamentally agree with the sentiment in these bullet points:

As you will note in ZingoLabs’s most recent release announcement we are actively (and extremely cheaply/efficiently) contributing to:

  • zcashd deprecation (via adding regtest mode to zebrad)
  • librustzcash sync performance (which has implications for Zashi
  • indirectly by supporting test requirements for YWallet, freeing up @hanh to focus on DEX’s which addresses your concern about single-asset wallets

In short, I think your reasonable concerns and priorities might be more aligned with efforts than you realize.

I love this idea.

3 Likes

i’m advocating for ECC to take the lead because they are developing zashi

1 Like

ECC is over-loaded with responsibilities… the last thing they need is to run a bunch of infrastructure.

Just to reiterate, that’s what this RFP is for:

To run infrastructure for ALL lightwallets.

(Including Zashi, which needs infrastructure to function.)

Also I am strongly in favor of more diverse providers. So I hope more folks offer proposals for this RFP.

9 Likes

Agreed [10 char]

3 Likes

Thanks for making the “unconfortable questions”!

This RFP addresses one of the Light Client Working Group initiatives.

@Zancas was indeed appointed to steer this initiative on December 14th 2023 called “decentralizing core dev”. Which aims to as its name says, remove ECC as a central actor of core development and opening the game for other teams to jump in.

LCWG is composed of wallet developers and we work with the members we have. Every wallet developer within the ecosystem is welcome to participate and join in.

Ultimately the initiatives have been losing priority to the (more important and critical) TEX address Outreach and deployment initiative. But that does not mean that we don’t keep working on other stuff.

@gguy consulted me about this RFP as Zcash Wallet Community Developer. I reviewed it, suggested changes which he did include and appreciate. This task was tracked here but I did not mention any specifics because it wouldn’t be appropriate before it went public.

This RFP is for ALL wallets. One problem of Zcash’s ecosystem is its tendency to fragmentate which is not precisely a synonym of decentralize. It has been a problem that Zec Wallet had its fork of LightWalletD, then Ywallet had one and ECC would have its own private instances and then ZCG would host public ones. That’s all cool, but it’s a problem when one goes down and users want to switch servers and they are not idempotent or worse incompatible with each other.

This RFP attempts to establish a global public infrastructure, maintained by many teams both in terms fo the “clouds” and the code they run.

I hope this helps clarifying a bit.

6 Likes

I am not sure I understand how the proposal fits here.
It actually does not mention the software version and its update policy.
It’d be great to also understand the budget allocation to pay for the cloud services vs the compensation for the team.

4 Likes

FYI, there are no “Ywallet” fork.

It is ECC + ZWL. Zcash-infra has both interfaces.

2 Likes

Based on the funding constraints, doesn’t this push costs to ZCG who are supposed to also be funding Qedit, ZSAs and stablecoins. Does this transfer (along with the current and expected comittments) affect ZCGs ability to fund the new Qedit proposal?

Side note - I personally think we need to think about ECC and Foundation jointly funding ZCG and maybe its even voluntary for both orgs Each one funds 2.5% to ZCG and they can fund more if they want. Seems like ZCG is more of a way to help the foundation and ECC offload some of the things they cant do themselves or dont want to do; but it should not be anything other than a way to help the existing orgs much as they are here by helping ECC. This could help stem some of the apparent infighting over money (its mostly ZCG trying to take money from the foundation, miners and adding more inflation).

2 Likes

I’m not the author of the RFP, so this questions should be asked to @gguy.

Well, if you look in the paragraphs below there are some bullet points about what teams are encouraged to detail and cost breakdown is one of them.

1 Like

I wasn’t talking about the rfp…

1 Like

Kindly notice the past tense of the sentence.

1 Like