Grant Idea - Zcash Blockchain Infrastructure (zBI)

Community Grants
1

Zcash Blockchain Infrastructure (zBI)

A platform for automating and orchestrating containerized blockchain infrastructure

Background

Alphega Solutions is a consulting firm established by John Akinyele, an AWS Certified Solutions Architect Professional. The firm provides clients with solutions for migrating and implementing cloud-based production workloads. Alphega Solutions began working with Bolt Labs, Inc in 2018 to implement a highly available Zcash full node and lightwalletd endpoint leveraging cloud-based containerized technologies.

Motivation

This project aims to create a platform (zBI) for operating and maintaining a high-availability architecture that allows for seamless scaling and a failure resilient infrastructure for operating Zcash instances with 99.9% uptime out of the box. zBI will provide mechanisms for deploying Zcash instances (comprising of Zcash full node and lightwalletd server) that are fault-tolerant with redundancies to handle high traffic. zBI promises to provide the Zcash community with an automated and repeatable containerized solution for operating Zcash instances. The platform will provide a framework for creating a secure multi-user shared environment based on cloud infrastructure.

zBI Platform

Cluster

The platform represents an environment/framework that can support one or more highly available Zcash nodes/instances. The framework will provide facilities for monitoring the health and state of the instances and auto-scaling to ensure the desired instances are maintained.

Node/Instance

Each instance in the cluster represents a full-node Zcash and/or lightwalletd provisioned and configured with storage, certificates, and a set of authorized users.

zBI Users

Platform Administrator
This represents the administrators for the underlying infrastructure with responsibility for purchasing and provisioning physical/cloud infrastructure. These users have the responsibility for creating instances, provisioning required storage and assigning instance administrators.

Node/Instance Administrator

This represents the administrator of a specific instance in the cluster. These users have the responsibility for managing the instance and determining which APIs to expose to end-users through the JSON-RPC or gRPC endpoints.

Node/Instance End-users

This represents the developers that can access available APIs through the endpoints exposed by the instance administrators. Instance administrators will authorize and provide end-users with appropriate credentials for secure access.

zBI Use Cases

The goal of the zBI platform is to support use cases that further the ZOMG mission to make Zcash ubiquitous. Specifically, we will focus on two essential use cases that will support the ecosystem:

Quick access to Blockchain data

Efficient and scalable access to comprehensive historical Zcash blockchain data. This is useful to exchanges and data analytics providers.

Node Management Platform

Leasing private nodes to builders and end-users for Zcash application development.

Technical Approach

zBI is a containerized solution for managing Zcash blockchain infrastructure in a portable and extensible containerized-orchestration system that allows for automated deployment, scaling, and management. The platform will leverage Kubernetes to orchestrate clusters of virtual machines and schedule containers to run on those virtual machines based on their available compute resources and the resource requirements of each container. Kubernetes is designed to group containers into pods so that they can be scaled to the desired state. It automatically manages service discovery, incorporates load balancing, tracks resource allocation, and scales based on compute utilization. In addition, it checks the health of individual resources and enables applications to self-heal by automatically restarting or replicating containers.

Software Components

zBI will use a declarative approach to define a desired state for the software components and corresponding configuration required to create a Zcash instance. These components (Zcashd and lightwalletd server) already have containerized solutions and it is also possible to create alternate containerized solutions from forked repositories in other to allow for customizations within these components. Other entities such as zcash.conf, security assets, and TLS certificates will be similarly represented as configurable assets. This approach allows for flexibility in configuring environments that meet the desired need. The need could be for a single-node instance (full node and/or lightwalletd) connecting to either testnet or mainnet; or an isolated peer network (with two or more nodes) fitted for a variety of testing or simulation needs.

Control Plane

zBI will provide a control plane with functionality to manage and operate the infrastructure. The control plane will provide mechanisms for defining and continuously managing the components of the platform to make sure the desired state is maintained.

High-Availability

High availability will be achieved through the combination of a continuous deployment pipeline to maintain the environment and deploy the necessary software components and leveraging auto-scaling and self-healing features of Kubernetes.

ZBI Architecture
ZBI Architecture751Ă—556 41.1 KB

Execution Risks

The execution risk is minimal for this project. Kubernetes ecosystem is designed with features such as service discovery, load balancing, storage orchestration, automated rollouts/rollbacks, and self-healing. DevOps/GitOps principles will be leveraged to ensure the environment keeps up with changes to Zcashd and to overcome any inherent instabilities.

Downsides

This platform will be hosted in US East 1 region of AWS and thus subject to the availability and downtimes of its availability zones. Kubernetes ecosystem has a proven rich set of design patterns for implementing robust and fault-tolerant systems. In addition, disaster recovery plans will be put in place to replicate to an alternate region in case of catastrophic failures.

Evaluation Plan

Alphega Solutions will demonstrate the viability of this platform by operating and maintaining a set of Zcash full node and lightwalletd endpoints to the testnet sandbox environment using a Kubernetes cluster over 5 availability zones in US East region of AWS. The platform will also provide a framework for individuals to run their own Zcash infrastructure on personal devices or in a cloud environment. Ultimately, the goal of this project is to promote the deployment of containerized environments for R&D purposes and to increase community involvement.

Tasks & Schedule

Initialization Phase (1 month)

Setup AWS account
Purchase 1-year reserved EC2 instances
Configure AWS services
Setup Git Repository

Development Phase (3 months)

zBI Control Plane
DevOps Pipeline configuration

Deployment Phase (2 months)

Implement DevOps pipeline
Test and approve environment
Release zBI to Zcash community
Bug Fixes

Maintenance Phase (1 year)

Operate environment
Release Git repos and documentation to Zcash community

Budget

Infrastructure Cost $ 40,000

Compute – EKS, ECR, EC2
Storage – EBS, S3
Network – Load Balancer, NAT & Internet Gateway
DevOps – Code pipeline, Code build

Development Cost $ 20,000

1 Software Engineer
1 DevOps Engineer

Maintenance Cost $ 20,000

Infrastructure Monitoring
AWS Support
DevOps Cloud/Kubernetes Admin

Overall Cost $ 80,000

7 Likes
2

Hi Welcome to the forum!

Can you elaborate on your business relationship with BOLT labs since 2018? What was the outcome?

Is the similar username to @jakinyele (founder of BOLT labs) just a coincidence?

How would this differ from existing services like GetBlock that are free for up to 40k requests per day?

And lightwalletd.com which is an existing ZOMG funded lightwalletd endpoint?

4 Likes
3

I don’t think so. I interpreted the proposal as creating the necessary scripts, automations and containers to operate a lot of nodes easily. This would be infrastructure that Block Explorers, Exchanges and others services might adopt since you wouldn’t need to take care of a lot of “boilerplate”.

Basically is building GetBlock software open-source for Zcash and let people run and store it on their own.

Am I correct @john_akinyele ?

8 Likes
4

For full disclosure, we are relatives so that explains the last name. Coincidentally, our first initials are the same. We have continued on a collaborative effort (along with Electronic Coin Company which I failed to mention in my proposal) to define a boiler-plate environment for orchestrating containerized zcash software components. This proposal is based on that collaboration. The post below by tloriato is an appropriate depiction of what is intended.

8 Likes
5

Yes. your description is correct

6

Ok, thanks for the clarifications @tloriato and @john_akinyele .

I’m wondering how often a tool like this would be used?

The ability to easily spin up a bunch of nodes is cool in principal but other than Exchanges (who likely already have something like this working, but proprietary) and blockexplorers (?) who are the primary target users? ECC core devs? Zcash Foundation devs?

I ask because I think ZOMG will need to consider thier perspective on how this tool would fit into thier workflow/pipeline.

1 Like
7

So, I see some network-effects here.

Having this project might help stress test and foolproof the core implementations of the protocol. I think is a little bit of a battle-testing in real-life.

If I’m in a company that wants to start doing anything with Zcash, you can be sure that I would build a similar thing to what is proposed. Sure, right now those companies are Exchanges/BlockExplorers, but if we start with ZSA/etc/etc you don’t really know what it could become.

Maybe this will eventually be the infrastructure that supports a Private Opensea?

Maybe it help us launch more explorers, more on-chain analysis, onboard on more exchanges, etc…

6 Likes
8

As you pointed out, there are various use-cases for such a platform. The focus is to build a framework that will allow users to spin up nodes based on their needs.

9

@Shawn see the post by @tloriato and my response below. The primary target is the Zcash development community.

2 Likes
10

As an engineer, the first time I was interested in Zcash, I looked for something similar to this. A script to get me started running zcash on local machines. All deps and configs taken care of, so that I can immediately experiment with available apis.

2 Likes
11

Zcashd can be run from the command line with an empty config file. Not sure I understand what script someone needs to run a node locally.

12

Zcashd with all the experimental features + lightwalletd and all the connection setup?

13

That’s still command line arguments, isn’t it?
The main pain is having to wait for blockchain download and sync. This proposal would help in that aspect AFAICT.

2 Likes
14

Thats how I read the proposal and agree it’s worthy of developing. The proposed solution is modern/up-to-date as far as I can tell.

Security

I suggest given the security implications and given we’d only have 1 software eng and 1 devops eng on the project it might be nice to have another set of eyes over the code/solution. Not sure if someone at ECC or ZF can be lined up to assist but I’m also aware finding a security expert in both software and devops internally might be tricky. But maybe we’re not worried and the repo can just be flagged as “not-production-ready: no security audit/review”?

ECC/ZF feedback

I’d like to hear from developers at ECC and ZF first to confirm nothing is/has been developed internally. Also I’d hope ECC/ZF are end users of this so we should hear from them to confirm they would indeed deploy/use this.

Conclusion

no idea about price but I’d be leaning positively

3 Likes
15

everything but this :slight_smile:

16

Hi @john_akinyele it has come to our attention that your proposal is still in “draft” mode on the Grants platform.

Please fully submit it on the platform as soon as possible.

4 Likes
17

ZOMG member here:

I love the idea of zBI. I worked for years as the devops co-lead at the Ethereum Foundation where I saw the progression from people having to run their own node to robust automation of nodes. I also saw in the Ethereum ecosystem the need, that happened gradually, then all at once, for many nodes for users and businesses, to be automated with failovers, etc. for things like OpenSea and DeFi dapps.

I like that the approach isn’t overly engineered and follows what I consider best practices. If it’s open source that will be very helpful on people iterating on this.

I have some questions for the team:

  1. I would love for the git repos and documentation to be released as it is happening and not at the end (unless there is a reason I am missing why it should be hidden). Helps ZOMG track progress and encourages community participation. In your outline it appears to only be released after the whole thing is done.
  2. I’d like more information on the Control Plane (I wasn’t sure if Control Plane was an AWS thing or just a way to say control panel). Is it GUI? How granular to zcashd does it go as far as selecting commands?
  3. Has the zBI team talked to ECC/ZF about this to see if they already have deployment code?
  4. To what extent will the documentation include things relating to the safety of running certain node commands/features (RPC related security, DDoS protection, etc.)

In general something like this is very necessary for Zcash when it starts to gain traction with the possibility of ZSAs and programmability and in general I am in favor personally of this grant.

9 Likes
18

I approve the grant! (Although I have no power over ZOMG).

What’s next is offering Zcash full node as a service. Very few companies offer Zcash node services.

Even if that’s not the case, ZBI will help wallets, exchanges, hedge funds, custody providers easily run their own Zcash nodes as more investors/users onboard to Zcash.

Bonus points: As Zcash moves towards PoS in next few years, ZBI will help individuals spin up their own validators.

Please make it open-source as this is community funded.

6 Likes
19

It’d be interesting to know how much capacity we are getting for the price. How many zcashd nodes, how many lightwalletd.
Is it going to be like Infura? With them you get your own instance for your project.

5 Likes
20

+1 to Zifura

2 Likes