Cool, the more the merrier! What’s your plan for maintenance after your grant expires, would the servers go offline, and would the devops scripts no longer be maintained?
Why create new devops tooling instead of contributing to what the community has already built and paid for?
Terraform could be interesting, but there’s little need for Ansible with our existing Kubernetes and Docker approaches which are already a lot to maintain.
I would support this if it used and contributed to our existing zcash-stack rather than reinventing the wheel.
Thanks for your proposal! The ZCG committee will discuss your proposal over the coming days.
ZCG’s goal with this RFP is to decentralise lightwalletd infrastructure and reduce the barrier/time/costs for new teams/deployments. Are you able to provide any extra details about:
Will any of the existing tooling (e.g. zcash-stack) be utilised?
Will the ansible and/or terraform tooling proposed be useful to the wider community?
We will review the proposal and its budget taking into account any new assets/tooling developed and its value to the community. Otherwise utilising/contributing to existing tooling is appreciated especially if it helps you fall within our original budget.
Cool, the more the merrier! What’s your plan for maintenance after your grant expires, would the servers go offline, and would the devops scripts no longer be maintained?
Great question! The sustainability of the project after the grant expires really depends on the ongoing support from the zCash granting organization. As long as we continue to receive funding, we would apply for extensions and keep the project running.
In the event that funding becomes unavailable, we would be open to open-sourcing the DevOps scripts, which would allow the community to contribute to ongoing maintenance and ensure that the servers stay up and running. This way, the community can play an active role in keeping everything going.
As Far as using the existing zcash stack,
This is a fair question. We want to leverage the existing tooling. We would use Ansible to provision the Hetzner instances, that is: Setting up SSH Users, configuring fail2ban, installing and setting up K3s in both the control plane and agent nodes, and all the general configuration management required at the host level that is not possible directly with Kubernetes or Docker.
For Terraform we plan to use it for setting up the Hetzner Infrastructure, including the placement groups, private network, firewall, servers, etc.
We may also leverage Terraform to manage the Kubernetes manifests and Helm charts. We believe this will be useful to the wider community as it could simplify the management of multiple Helm charts.
By no means we are trying to reinvent the wheel, our plan is to integrate new tools with the existing ones.
The costs I am seeing here should be an order of magnitude less, at most. What I am seeing here is not making any sense.
Wallets, how do you select a lightwallet node on startup? Do you ping each node and select the fastest one? If yes, great. If not, why not? Also, when you detect that the active lightwallet node is not responding, do you switch to the next available one? cc: @zancas@hanh@Honza
We do not need to spend anything on High Availability architecture.
Instead, we need a few motivated, experienced and reliable community members to run servers that are ideally not overlapping in location or network. They should certainly be compensated, appropriately. This could be a great warmup for people expecting to become validators once we switch to PoS.
Because they are experienced and reliable, they will rarely go down. But in the event that they do, the wallets would have the proper logic in place to switch to the next available lightwallet with little to no interruption to the end-user.
Ping measures RTT. Wallets need throughput. So no, I don’t automatically pick based on ping, but the metric is available and the user can decide.
There are < 1000 full nodes. Sure, ideally, people would run full nodes around the world and create a mesh of lwd. In practice, it didn’t happen and the ZCG had to offer a RFP because of the lack of common infra.
Even with the RFP, the offers were scarce, hence the high price. FYI, I run 6 lwd servers for a grant of 6k / year, but there is no uptime guarantee.
You can say they charge too much. It is a free market. They charge what the community is willing to pay. Honestly, IMO this is NOT the most outrageous expense I have seen by far.
That’s harder to do than you think. Network code is complex to design, and even harder to test and debug.
I actually don’t know how much throughput is needed per transaction and how many transaction you have to handle. However, I would assume your lightwalled servers list is composed of high bandwidth servers, so I don’t quite see how using ping to select the best server would be an issue.
How many do we need? Where are they located? Do we have strong decentralization of those nodes? We may have enough for our needs at this point?
I did not want to spend time reviewing things before, because why would I. But if stakeholders finally get to decide how their own money is spent, I will certainly take the time to review things a bit more going forward.
Got it. True, I should phrase things better so it’s clear that I am suggesting an idea that I am not sure how easy it would be to implement. Thank you for clarifying.
Is there an alternative way forward we should consider, long term?
You need to download several GB of data to sync a wallet from the start. I don’t see how we can assume the servers are high-bandwidth if they are supposed to be decentralized.
This was an optimistic estimation. It’s more likely in the few hundreds.
Sure, but the lightwalletd is under ECC and there is no plan to change that.
There are multiple possible levels to decentralization. Zcash nodes can relatively easily be decentralized using low powered, low bandwidth servers; that’s ideal. On the other hand of that, we have what we have here today with lightwalletd nodes, a High Availability setup; that’s the opposite of ideal. What I am arguing for, is a middle-ground. A dozen or so (would be a good start), of well distributed servers with relatively high bandwidth, managed by separate entities, over separate jurisdictions, etc.
There are lots and lots of datacenters all over the world, we can definitely use that. It’s where I run my nodes anyway.
The most important is how decentralized those are. Do we have data on geographic and network (to make sure it’s not all Hetzner or something) distribution of those?
Is it correct to say that, Fly.io, could eventually receive (if they have not already) a National Security Letter (NSL) with an accompanying gag order or non-disclosure requirement, that would force them to store transactions and metadata transiting through zec.rocks servers?
IPs may not seem like much, but once those are crossed with advertisement networks (Facebook, Google, etc) and using browser fingerprinting, it’s quite easy to reliably get a few names behind a single IP, without even calling any ISP to ask for anything (which they also can do).
Trezor uses Tor (optionally) to transmit transactions. Are our wallets doing something of the sort?
It is currently supporting Zcashd and lightwalletd, and we do use a Grafana dashboards showing metrics given by a Prometheus exporter querying the RPC endpoints of the Zcashd node we are running.
Zaino and Zebra are being added.
We do have plans to improve the code in the repository, we are currently working on it.
The endgoal would be to have a “one-click” deployment for any cloud provider or home servers.
This is not part of a grant.
We’re listening to your suggestions of improvements.
Just a quick note that I’ve interacted with @stakehold.rs for a few days and believe they know what they’re doing. I’d prefer that they contribute to our existing infrastructure-as-code rather than rolling their own, but from a pure “would they host reliable light wallet servers” I believe it’s a yes.
Perhaps we can merge our efforts, I’ll coordinate off-forum.
Welcome and thanks for contributing before applying!
The best way to test these endpoints for now is to use Zecping.
There are ongoing efforts to add Tor light wallet server support to Zcash wallets, hopefully our new onion (“hidden service”) endpoints will help speed up those endeavors. We would love to see Zcash work in the TAILS operating system and other Tor-only environments.
These endpoints are powered by the new Tor support in zcash-stack.
@mlabs011 thank you for your submission. After consideration from @ZcashGrants and sufficient time for the community to provide feedback on the forum, the committee has decided to reject this proposal.
The committee appreciates your grant submission efforts and encourages you to continue as an active member of the Zcash community going forward!