Thank you all for the transparency with how you are operating this infrastructure, the tooling you’re creating, the challenges that the ecosystem can collaborate with you on, and for asking the vendors that you are working with to accept ZEC!
6 Likes
latest ywallet includes it!
6 Likes
Hi all - here are the configurations currently used for the zec.rocks edge containers on Fly and the regional Kubernetes cluster Helm charts.
I will keep the repository updated as changes are made.
Next up will be an updated README with instructions on how to deploy these in your own Fly account.
4 Likes
Milestone 2 is delivered: 3 months of uptime for the zec.rocks Lightwalletd infrastructure.
Completed:
- Four global regions are now running on dedicated hardware using k3s, diversifying our backend and giving us more control over the physical hardware layer. Our hosts are HiVelocity for the dedicated hardware and Vultr Kubernetes Engine for overflow. Each region has 2 dedicated servers.
- We open sourced our HAProxy and Fly configurations to bring transparency to our load balancing approach and to help other teams host resilient Zcash infrastructure.
Opportunities:
- We currently have excess compute and are happy to provide CI runners or host additional services for the community within reason. Our main constraint is engineering time.
Works in progress:
- Deploying a block explorer
- Building a web application to monitor the uptime of Lightwalletd nodes across the ecosystem
- Improving the documentation for our open infrastructure-as-code based on user feedback (if you’re interested in deploying a lightwalletd server using our Helm charts, we’d love to collaborate!)
A heartfelt thank you to all who have supported this work. We will continue to post updates here.
8 Likes
Just a quick reminder that running a seeder would be an excellent use of some of those excess compute resources!
4 Likes
Hello, Emerson!
ap.zec.rocks is down.
Thank you for reporting this. The issue should be resolved.
2 Likes
Did I hear correctly that there are Dockerfiles somewhere for these nodes?
Edit: I guess it’s Kubernetes.
There is a Zebra-LWD docker config that will spin up a local instance.
https://zebra.zfnd.org/user/docker.html#running-zebra-with-lightwalletd
Quick updates from the Zec.rocks team:
- We delivered Milestone 3 of the grant, continuing to provide global lightwalletd infrastructure for the Zcash network.
- A new endpoint “zcashd.zec.rocks” was launched to assist with troubleshooting differences between Zebra and Zcashd.
- “testnet.zec.rocks” is maintained as well.
- We continue to maintain the Helm charts used to deploy our infrastructure on Kubernetes
- We are working on hosting a block explorer and will announce it here when it is ready.
- A project is in process to track the uptime of lightwalletd nodes, and eventually Zcash nodes. It is a web application that will be open sourced, stay tuned here for further updates on this as well.
Thank you, Zcash users and community! We are honored to be contributors here.
4 Likes
Zec.rocks infrastructure is now running Zebra v2.0.1 globally, with a few backup Zcashd v6.0.0 nodes running as well.
The zcash-stack helm charts are updated, and are now published on ArtifactHub.
Current maintained lightwalletd endpoints:
- zec.rocks:443
- na.zec.rocks:443
- sa.zec.rocks:443
- eu.zec.rocks:443
- ap.zec.rocks:443
- testnet.zec.rocks:443
- zcashd.zec.rocks:443
8 Likes
Our block explorers are live: (shoutout to @dismad for the help!)
We forked zcash-explorer to fix its CI build and make a few small tweaks. We will try to continue maintaining it time-permitting.
The zcash-stack Helm charts are updated to support launching zcash-explorer, and a snapshot of zcashd’s data directory indexed for block explorers is available for quick bootstrapping.
We also launched a new lightwalletd region in Dubai, on dedicated hardware running k3s:
me.zec.rocks:443
We plan to host a hands-on workshop on how to run your own lightwalletd nodes in December, stay tuned for more information on the timing and curriculum.
11 Likes
We delivered Milestone 4 of the grant, continuing to provide global light wallet infrastructure for the Zcash network.
Our vision is to further decentralize the Zcash network by financially incentivizing people to run more light wallet nodes. We also want to make it as easy as possible for non-technical people to host their own nodes.
This Monday, we begin our first workshop on how to host Zcash light wallet infrastructure. Everyone should sign up! We’re bumping the class size up to allow everyone to participate.
We will reward the top 20 server operators which have the best uptime with a $90/month reimbursement in ZEC for their time and hosting costs for six months. We’d love to extend this reward further and intend to publish a public donation address that any individual or granting body is welcomed to contribute to, even retroactively if desired.
To track the uptime of our growing network and to fairly reward new participants, we are developing Hosh. We maintain a Hosh instance at hosh.zec.rocks which we will continue to maintain throughout the six month class.
Our goal is for there to be so many nodes out there that we no longer will need to run our own infrastructure. Instead of hosting it, we will aggregate it and allow wallet developers to pull a list of nodes from Hosh’s API. I believe that we’re around six months out from that goal.
Thank you to ZCG for bootstrapping our efforts with the grant that we applied for in this thread. We intend to experiment with crowdfunding for our next proposal before asking for any ZEC from our community’s grantors.
Zcash rocks!
6 Likes
seems https://explorer.zec.rocks/ has not updated block info for 3 days now
2 Likes
Good catch, sorry about that! The explorer is fixed. We are adding monitoring of explorers in our ecosystem to Hosh so that we can catch things like this faster.
The issue was an out of date zcashd running in our infrastructure, the Helm charts have been updated.
3 Likes
Zebra v2.2.0 is now deployed across our infrastructure.
2 Likes
Are you still accepting proposals for infrastructure development and maintenance? looking to put together a proposal for 1 year based on the requirements
5 Likes
Yes. You can DM your proposal to @ZcashGrants on the Forum or email it to jason@zcashcommunitygrants.org and I’ll share it with the rest of the committee. Thanks.
4 Likes
The Zcash-stack Helm chart is updated to support Zaino on Kubernetes.
Our Zaino endpoints are also live for testing. Do not use these for anything important yet, these should not be shown as defaults in wallet apps.
- zaino.unsafe.zec.rocks:443
- zaino.testnet.unsafe.zec.rocks:443
We do not log IPs on any of our endpoints but it is possible that we will enable error logging on our *.unsafe.zec.rocks endpoints to assist with debugging.
Let’s deprecate!
4 Likes
Hi all,
I’m Jordan from Mlabs (mlabs.city). We are an AI & Blockchain software development team with experience in cloud infrastructure, DevOps, and distributed systems, primarily known for our open-source contributions and infrastructure work in the Cardano ecosystem.
We’ve been following the Zcash ecosystem with interest and recently came across the RFP for lightwalletd infrastructure development and maintenance. In response, we’ve submitted a grant proposal to provide an automated monitoring and on-call response service for the Zcash network, specifically aiming to enhance the reliability and performance monitoring of crucial components like lightwalletd .
Our proposal focuses on providing a reliable, automated monitoring system with proactive daily maintenance and on-call support for critical incidents. Here’s a brief breakdown:
- Goal: Establish robust, automated monitoring for Zcash network health, focusing on
lightwalletdperformance and availability. - Infrastructure: Deploy a geographically distributed Kubernetes (k3s) cluster on Hetzner Cloud (using nodes in EU, US, and Asia). Infrastructure will be managed via Terraform and Ansible for automation and reliability.
- Monitoring: Utilize automated tools (like Prometheus/Grafana) for continuous monitoring of key metrics and alerting on critical issues.
- Operations: Provide an on-call response service. The budget allows for approximately 1.27 hours of engineering time per day for proactive maintenance and incident response. This is not continuous 24/7 human monitoring.
- Response Time: Aim to acknowledge critical alerts within 12 hours and begin active investigation within 24 hours. Non-critical issues addressed within 1-2 business days.
- Budget: The total proposed budget is $75,000, split between infrastructure costs (~$33.2k buffered) and operations/maintenance (~$41.8k).
The full, detailed proposal has been submitted to the Zcash Community Grants platform (Issue #32 on the zcashcommunitygrants GitHub repository) and provides more specifics on the technical approach, budget justification, and deliverables.
We believe our expertise in building and maintaining robust, scalable infrastructure can benefit the Zcash network. We welcome feedback from the community on our proposal and look forward to the discussion!
2 Likes