Zecwallet is a desktop and mobile Zcash wallet with full support for shielded and transparent transactions. Over the last month or so, traffic to the Zecwallet’s LightwalletD servers has increased ~4x (probably correlated with the price of Zcash) as old and new users refresh their wallets.
This has put a large amount of strain on the Zecwallet server infra, and Zecwallet needs to add a new server to cope with the increased traffic without which I’m worried the infra will melt if there is another large influx of users.
Motivation and overview
Zecwallet’s LightwalletD server served ~25 million blocks yesterday, with peak traffic exceeding 50,000 blocks/min (Note this is all server bandwidth usage, Zecwallet clients don’t have any logging, so it is hard to estimate how many new / existing users of Zecwallet) which is approx 4x what it was at the start of Jan.
This grant is based on the previous maintenance grant, but simplified down to only the basics needed to keep the lights on.
It features 3 work items:
- Add a new LightwalletD server in Asia
Add a new infra setup (LightwalletD server + zcashd fullnode *2) in a primary-backup setup in a Asia AWS region. This will need 2x t3.xlarge machines (Zecwallet’s LightwalletD has higher memory and disk requirements, please see previous grant application )
This work item also includes the devops needed to setup the new region
- Zecwallet Fullnode
To protect against any eclipsing attacks, the zcashd nodes powering the LightwalletD run with
maxconnections=500. A side effect of this is that the nodes are connected to a lot of new Zecwallet Fullnode users who are syncing the blockchain from scratch, and it ends up serving a lot of bandwidth to new Zecwallet Fullnode users.
Need to investigate and patch this in the embedded Zecwallet Fullnode zcashd node, which should help reduce server traffic costs and ease some infra pressure. (Note, this is just a hunch that this issue is causing a large amount of bandwidth to be used, I need to investigate to be sure first)
- Fix checkpoints
New Zecwallet users need to sync from a known ‘checkpoint’ to catch up to the latest block height on a lightclient. To help with decentralization, Zecwallet has so far embedded the checkpoint (which contains the block hash and the sapling commitment root) directly into Zecwallet’s code so that the users don’t have to trust the LightwalletD server.
A side effect of this is that Zecwallet needs regular releases to update the embedded checkpoint. The last Zecwallet release was a over a month ago, which causes new users to download ~50k blocks just to catch up to the chain tip. This is unnecessary, and can probably be optimized away.
For (1), we already have deployed infrastructure, so the technical approach is just to copy the deployment to a new AWS region and reconfigure the DNS, SSl etc…
AWS Server costs
This section describes funds that are spent directly on AWS, which hosts the Lightwallet infrastructure
USD 2,000 / month
6x AWS large instances (across 2 regions) for
LightwalletD(primary + backup)
1x AWS t2.small instance for load balancing, website and other static hosting
AWS bandwidth costs, including zcashd serving blocks
SSL certs, App store developer fees and other miscellaneous one-time costs
This grant requests funding for 6 months of server costs.
For (2), we’ll investigate bandwidth usage on both client and server with the patch, measure bandwidth usage, and deploy a new release if it reduces bandwidth usage
For (3), we’ll add a new checkpoint to the lightclients, make new desktop + mobile releases, and look at pulling in the work done by the ECC wallet team to use the sapling root delivered from zcashd → ligthwalletD → Zecwallet lightclient.
I estimate this is about 5 days worth of work (for all 3 items)
Since this is pretty small scope and the work items are well understood, the execution risk is minimal. There is a larger risk that Zecwallet’s LightwalletD is centralized, and needs to become compatible (at least on the API level) with the ECC’s lightwalletD so that users can easily switch between providers.
This grant will help ease the infra/bandwidth costs a bit, and they also make progress in reducing costs in the long run, but more can be done to address centralization and decreasing costs.
Success of this project is mainly measured around:
- Lightclient infrastructure uptime >99.5%
Budget and justification
Code and Devops :
USD 7,500 (@USD 187.5/hr)
USD 12,000 Cloud infrastructure (USD 2,000 * 6 months)