Block explorer + Testnet infrastructure

Community Grants Applications
1

Title:

Block explorer + Testnet infrastructure

Applicant name:

Cacho

Pitch: A one-liner elevator pitch version of your proposal

To provide developers community with a testnet infrastructure

Total Request (USD):

$15000.00 USD

Have you previously received a grant from Zcash Community Grants (formerly called ZOMG) or ZF?

No

Are you seeking or have you received funding from other sources for this proposed project?

No

Applicant background:

With more than 15 year of experience in technical and managerial positions, explored diverse industries and their challenges.

In my academy facet, I got a Phd and perform research in software engineering. I’m teaching Cryptocurrency Introduction Lecture at the National University of La Plata where one of the main topics is privacy and Zcash as the main platform to transact tokens. I’m presenting this grant based on a need identified during Zcash teaching where basic resources for browsing block information and tokens for assets are require.

In the industry side, I’m am leading some development projects and product implementations.

Description of Problem or Opportunity:

Zcash community actively works on providing the technology for a blockchain focused on privacy. These privacy properties are not present in other blockchains making Zcash unique on its kind, therefore both developers and end-users may need a different learning path to familiarize themselves with the concept of a privacy preserving blockchain and cryptocurrency protocol.

Testnet plays a fundamental role in the development cycle of a blockchain ecosystem. And most importantly during the first steps of any end-user and developer with intentions of joining the community. During the Zconf 24 it was claimed many times by key players that the community should focus on providing tools to ease the technology adoption. Unfortunately the official testnet node is not working, limiting the approach of any person interested in the blockchain using a safe method.

On the other hand, the block explorer application has shown to be a useful tool in prod environment however there is no block explorer for test environment and production’s one is going to stop working by end of May 30th. Although the Block Explorer One is] public for testnet and production and Blockchair for prod [bc], they are proprietary and source code is not open-sourced so the community is not able to audit them or run a similar instance by themselves.

[beo] https://blockexplorer.one/

[bc]https://blockchair.com/zcash/

Proposed Solution: Describe the solution at a high level.

For this grant I propose to work on providing a minimum infrastructure to let a Zcash adopter to test the blockchain.

For such a thing, I will provide a Kubernete project based/inspired on Testnet in a Box (TNB) [tnb] project to use the latest Zcash node (Zebra) and Block Explorer [be] version. Then I’ll deploy it in the cloud with public access.

Because the Block Explorer has been running on top of Zcashd, the first deliverable will be based on the latest version of Zcashd node. The goal is to have a version live as soon as possible so that anyone can test. Secondly, I will work on updating the project to use the Zebra version of Zcash.

Then, I will include a faucet so that we are able to fund (deposit to) addresses, transact on the test network and browse transaction details in the Block Explorer. At this point, the infrastructure will provide anything a user needs to test the blockchain and see the results.

The fourth milestone consists of providing monitoring tools like Kibana to monitor and share operation data to the community.

The grant includes the expenses for hosting all the resources mentioned above.

Optionally, the node+block explorer components can be deployed on production doubling the grant budget. This could be a good idea because the project hosting the Block Explorer[be] is finishing by the end of May. Another option to consider is to deploy another Zebra node (even with a different version) on a different location ( EU) to exercise node synchronization. This is super useful for adopters who can track their transactions across the nodes but also for devs to run in an isolated context two Zcash nodes with different versions ( for example, main branch vs latest stable). Compatibility issues can be detected early.

Finally in order to speed up the testnet node initialization, a snapshot of testnet blocks will be published to get downloaded from a CDN. [tnb] GitHub - zcash-hackworks/zcash-testnet-in-a-box: A method to deploy a quarantined zcashd testnet with monitoring.

[be]GitHub - nighthawk-apps/zcash-explorer: Zcash Block Explorer

Solution Format: What is the exact form of the final deliverable you’re creating?

The goal of the project is to get a testnet live and thus I plan to provide a Kubernete project that orchestrates the required components deployment. Therefore, the deliverables are incremental version of such project:

  • Deliverable1: A kubernetes project deployed in the cloud for the latest version of Zcashd in open-source. Similar to Testnet in a Box project that packages many testnet components.

  • Deliverable 2 A block explorer deployed in the cloud using Zcashd.

  • Deliverable 3 Production Block Explorer

  • Deliverable 4 An updated version of the new version of Testnet in a Box for the Zebra node.A Block Explorer including any compatibility change required for Zebra Node.

  • Deliverable 5 Monitoring tools for the testnet.

  • Deliverable 6 Second testnet instance in a different location and using Zcashd.

  • Deliverable 7 A donation feature to support the block explorer and avoid recurrent grant application to support these critical features.

  • Deliverable 8 Testnet snapshot download from CDN.

Technical Approach: Dive into the how of your project. Describe your approaches, components, workflows, methodology, etc. Bullet points and diagrams are appreciated!

The components mentioned above will be packaged in a Kubernetes or similar container based solution to easy its deploy and maintenance.

Minor compatibility issues will be developed for supporting zebra in Zcash block explorer.

Dependencies: What external entities is your project dependent on? What involvement is required from ZF, ECC, and/or other external organizations? Who would have to incorporate your work in order for it to be usable?

In order to expose the new Testnet server under the zcash domain (explorer.testnet.z.cash), Zcash domain’s owner should update DNS records with the new CNAME record. Otherwise I will provide a new domain.

Execution risks: What obstacles do you expect? What is most likely to go wrong? Which unknown factors could jeopardize success? Who would have to incorporate your work in order for it to be usable?

One of the major risks is the improper estimation of hosting requirement .This could require to increase the server capacity to serve all the demand properly.

During the block explorer integration with Zebra, there could arise unexpected issues that will require support from both apps. I expect to rely on the community for assistance.

Unintended Consequences: What are the negative ramifications if your project is successful? Consider usability, stability, privacy, integrity, availability, decentralization, interoperability, maintainability, technical debt, requisite education, etc.

I don’t see any negative impact.

Evaluation plan: What metrics for success will you share with the community once you’re done? In addition to quantitative metrics, what qualitative metrics will you commit to report?

I will share quarterly reports manually or automatically showing many metrics like;

  • Node latency

  • Node transactions

  • Block explorer activity by different criteria like Country, timezone, and feature

  • Storage and CPU usage for future references

Hardware/Software total budget:

$0.00 USD

Please provide justification for the total hardware/software budget:

NA

Services total budget (cloud, hosting, etc.):

$5400.00 USD

Please provide justification for the total services budget:

Monthly cost of dedicated cloud server (or similar) per instance. Some reference price from Amazon:

Amazon Elastic Block Store for 500GB: usd 40.00

Amazon EC2 On-Demand instances for t3a.xlarge 4vCpu 16GB RAM: 109.79

Total: 149.79

The estimation is based on the official Zebra documentation System Requirements - The Zebra Book.

Three nodes are required:

  • Testnet US: Running the latest stable version of Zebra (testnet) and Testnet block explorer in US.

  • Testnet US: Running the main underdevelopment branch Zebra (testnet) and its Testnet block explorer in EU.

  • Production Block Explorer: Running Block Explorer in a US location.

Compensation total budget:

$9000.00 USD

Please provide justification for the total compensation budget:

1 - Server provisioning, and node deployment 5 hrs

2 - Kubernetes project development and testing 20hs. (Milestones 1-4)

3 - Block Explorer changes to work on top of Zebra and testing 20hs.

4 - Monitoring tools 15hs.

Total 60 hs x 150 usd/hr = 9000

Cloud servers maintenance: usd 500/mo

Do you require startup funding?

Yes

Start up funding - detailed explanation:

The node and block explorer hosting is the core of this grant. Without server’s provisioning the milestone can’t be accomplished.

The server hosting will be paid upfront to the cloud provider.

Start up funding - USD value of payout upon approval of grant:

$450.00

Milestone 1 - estimated completion date:

05/31/2024

Milestone 1 - USD value of payout upon completion of deliverables:

$4500.00

Deliverable 1.1

1 weeks : Deliverable 1 - Zcashd node in testnet.

Deliverable 1.2

2 weeks: Deliverable 2 - Block Explorer in testnet.

Deliverable 1.3

4 weeks: Deliverable 3 - Production Block explorer

Milestone 2 - estimated completion date:

06/30/2024

Milestone 2 - USD value of payout upon completion of deliverables:

$4950.00

Deliverable 2.1

6 weeks: Deliverable 4 - Block Explorer on top of Zebra.

Deliverable 2.2

7 weeks: Deliverable 5 - Monitoring tools

Deliverable 2.3

8 weeks: Deliverable 6 - Second testnet Instance

Milestone 3 - USD value of payout upon completion of deliverables:

$4500.00

Milestone 3 - estimated completion date:

07/31/2024

Deliverable 3.1

10 weeks: Deliverable 7 - Donation feature

Deliverable 3.2

11 weeks: Deliverable 8 - Testnet snapshot download

Deliverable 3.3

10 month hosting. 3 servers - 10 month

Total proposed USD value of grant:

$14400.00 USD

How was the project timeline determined?

It was done based on components dependency and bandwidth.

Application submission date:

05/10/2024

7 Likes
2

Hi @Cacho - Welcome to the forum, and thank you for submitting your grant proposal! We will review it in the upcoming weeks and reach out if we have any questions.

In the meantime, if you have any questions for us, you can post them to this thread or DM us at @ZcashGrants.

Zcash Community - We want to hear your feedback on this grant! You can post your comments to this thread or DM us at @ZcashGrants if you’d like to provide feedback in private.

2 Likes
3

“Unfortunately the official testnet node is not working”

What do you mean?
To my understanding, zebra currently lacks RPC functionality to support a full-fledged Explorer as we know it to be, hosted on a website, particularly with getrawtransaction.

4

It was down for a few days falling because the following error when using ZingoLib:

Error with get_lightd_info response at https://testnet.lightwalletd.com:9067/: Status { code: Unknown, message: “invalid http POST response (nil), method: getinfo, id: 104, last error=Post "http://127.0.0.1:18232": dial tcp 127.0.0.1:18232: connect: connection refused”, metadata: MetadataMap { headers: {“content-type”: “application/grpc”} }, source: None }

thread ‘main’ panicked at zingocli/src/lib.rs:469:13:

Now it is up again

1 Like
5

To elaborate on my second point, zebrads getrawtransaction only serves the hex field, which is fine if you are a light wallet asking for transactions that might belong to you, but an Explorer has nothing to work with there. You cannot display fees, note info or anything. I think a test net Explorer and free public testnet node api would be great but currently can only really be done w/ zcashd (the explorer, amyways).

6

One of the things that I had always wished that getblock io would implement was a public test net node to go along with their mainnet node API service. And again, zebra could do that, but it doesn’t quite have parity with call returns, so it is something to consider.
I would prefer to see zebra, you know, as in principle, right? But functionality of a full functional Zcashd node would be superior and the fact that the test net is only 7 GB (with txindex, maybe 14?) means that it would likely not be much work to swap them out at some point, maybe.

7

After running for some days a server with 2 cores, 4 gb Ram and 30 GB HD. It has been running smoothly <10% CPU and 20 GB usage (zebras, cache and OS). This is much less than documented requirements for prod. This server costs usd25/mo on GC.

Regarding the block exploration, yes, Zebrad-rpc module should be improved with additional verbosity levels to return JSON objects and/or decoderawtransaction method to decode the HEX data and return a JSON representation like in Zcashd:
decoderawtransaction - Zcash 4.5.1-1 RPC Docs

8

@Cacho at the most recent meeting, the @ZcashGrants Committee voted to approve this proposal and has requested that you provide monthly updates via the forum in this thread.

1 Like
9

Hi all, a quick update on the progress:

Deliverable 1.1

1 weeks : Deliverable 1 - Zcashd node in testnet. :white_check_mark:
It is running on http://185.62.57.21:18233.

Notes:
Zcashd is crashing unexpectedly on testnet. Still trying to figure out the issue. The service is rebooted.

Deliverable 1.2

2 weeks: Deliverable 2 - Block Explorer in testnet. :white_check_mark:
It is running on http://185.62.57.21:4000.

Deliverable 1.3

4 weeks: Deliverable 3 - Production Block explorer :white_check_mark:

Zcashd servier is running at: http://185.62.57.21:8233
BlockExplorer is running at: http://185.62.57.21:4001

The block explorer is: GitHub - nighthawk-apps/zcash-explorer: Zcash Block Explorer
Next steps:

  • Setup a proper domain for all. Do you prefer to use one domain under z.cash domain (i.e. https://explorer.z.cash/)? Please confirm. Otherwise, I’ll get a new one.
  • work on deliverable 2 package.
4 Likes
10

I think the human readable address version would be better. Thank you for this!

1 Like
11

The Zcash explorer landing page is available at:
https://zcashexplorer.app/

Feel free to update the dead link ZEC Block Explorer - Z.Cash to the new blockexplorer.

8 Likes
12

I missed this, super glad it’s running, thanks @Cacho and ZCG!

6 Likes
13

Hi,

A quick update on pending milestones and deliverables.

Deliverable 2.1

6 weeks: Deliverable 4 - Block Explorer on top of Zebra. :warning:

Deliverable 2.2

7 weeks: Deliverable 5 - Monitoring tools :warning:

  • An elastic search+kibana instance was deployed on block explorer server to use as block explorer datasource. Unfortunately critical data is missing like block id, transaction id, and tx confirmation count.

Deliverable 2.3

8 weeks: Deliverable 6 - Second testnet Instance

Deliverable 3.1

10 weeks: Deliverable 7 - Donation feature - Pending

Deliverable 3.2

11 weeks: Deliverable 8 - Testnet snapshot download - Pending

2 Likes
14

Could you fix the calculation of the fee? It has been wrong for years.

4 Likes
15

FYI https://mainnet.zcashexplorer.app/ is down with “Internal Server Error” (testnet is up though)

2 Likes
16

Working on it. I´ll update this thread asap.

2 Likes
18

hey could you upgrade the explorer to 6.0.0
cause atm it seems to not be in sync with network :thinking:

19

Hi,

Yes. working on it. It will take some time to split a new server to avoid OOS during indexing.
Do you mean testnet or mainet?

Best

20

Mainnet.

Testnet seems to work atm.

21

The explorer is down since 3/4 days.

1 Like