It wasn’t my pick.
1 Like
I am hearing that there are reservations in the ECC team about this grant? Do you have suggestions or questions @daira
@Dodger ?
Please reach out if there’s something to discuss. We are already more than a week later in the schedule than we intended.
Without more efficient communication we may not be able to contribute to the deprecation effort.
Have folks reviewed the most recent update to the proposal?
It includes the agreed upon revisions.
@zancas This was discussed during yesterday’s Arborist call. I don’t think there are reservations so much as your grant proposal may need some massaging or refinement.
It would be helpful if you were to join the weekly meetings we have to discuss zcashd deprecation, and the Arborist calls.
2 Likes
Thanks! I’m working to get all the right players together, and ensuring that we’re focused on doing the right work, right away.
We’ll be present at all LCWG, and Arborist calls going forward.
5 Likes
Totally went down a horse-color-in Spanish rabbit hole. Never heard of vaino but I’m familiar with alazán which seems to mean the same thing. It turns out there are as many Spanish words for horse colors as artic people have for snow. Oscuro is pretty cool. Free2z
No sé mucho sobre caballos y no soy hablante nativo, pero yo lo llamaría un caballo castaño, un caballo alazán, un caballo oscuro. “Oscuro” suena genial.
1 Like
Also the Zcashd Deprecation meeting that takes place on Thursdays at 3pm UTC.
1 Like
Sharing the annotated diagram from today’s Arborist call.
Annotated_Zaino_diagram.pdf (82.4 KB)
3 Likes
I like this architecture and plan. A Rust implementation of the light client protocol coupled with a remotely-accessible ReadStateService sounds wonderful.
My understanding of the plan for developing the zcashd replacement wallet (CLI wallet) is to reuse existing code that relies on lightwalletd, and to use the (Remote)ReadStateService to access any other data the CLI wallet requires from Zebra’s chain state. That sounds wonderful too, I think the (Remote)ReadStateService will offer the best access to Zebra’s state, require the least implementation effort to make available, be the easiest to extend during CLI wallet development, and makes it easiest to switch between directly querying the state from within the CLI wallet process and querying it over the network.
Using the ReadStateService to support the Lightwallet gRPC Service should also allow for reusing some code in zebra-rpc (I found that every method maps directly to a method in zebra-rpc except GetBlockNullifiers when researching librustzcash#1395, and now that we’re using the read state service, there shouldn’t be many changes needed except updating methods to return protobuf types with the same fields).
I’d like to make sure I understand the project correctly, the unimplemented and in-scope pieces of the latest architecture diagram that I see are:
- IndexerMempool
- IndexerStateService
- Lightwallet gRPC Service
- Indexer RPC Service
- RemoteReadStateService?
I may be missing something, but, I’m imagining the IndexerMempool would be very minimal: a task listening to mempool changes, updating a watch channel with the latest mempool data, and updating a broadcast channel with the latest changes to the mempool data since the last update.
On the Indexer RPC Service, I don’t know how to generate protobuf types for existing Rust types, I think it would be better to use an encoding with derivable (de)serialization methods (like rkyv, SCALE, or bincode, although hopefully not bincode). Presuming we could have derived implementations for (de)serialization, I was imagining a thin wrapper around the ReadStateService that: deserializes network requests into ReadRequests, calls the service, serializes the ReadResponse, and sends the response back to clients.
The RemoteReadStateService is in the out-of-scope section, but it would probably be useful for testing the Indexer RPC Service if it could be included. I was imagining something simple there too that: serializes ReadRequests, calls the Indexer RPC Service, then deserializes the ReadResponse.
I think my understanding of the project would benefit the most from clarification around the IndexerStateService’s intended role and an approximate possible design for the IndexerMempool, though, if possible, I’d be interested in an approximate possible design for all of the unimplemented and in-scope pieces of the latest architecture diagram. I’m also wondering if there have been enough design changes to warrant an update to the Zaino Grant Roadmap?
Thank you to everyone who’s been putting this together, it’s looking great.
7 Likes
I like the overall idea and the designs seems good.
The only thing that is not clear in my mind is to determine whether this is really the fastest path for Zcashd deprecation. An alternative would be to extend the existing Go lightwalletd server which whatever added functionality the CLI-wallet requires. But in order to do that, we need to map out exactly which functionality is missing (and this mapping out is needed regardless, since this functionality needs be implemented by Zaino anyway). I think ECC is better equipped at this point to map out this missing functionality since they are actively working on zcash_client_backend which will be used by CLI-wallet.
With that information, we could compare how much work each option would take (Zaino vs extend the Go lightwalletd) and make an informed decision.
2 Likes
Hey it might be easier if I break this down into a few sections.
ReadStateService backed lightwallet gRPC services:
With this I was under the assumption that we would have to implement the majority from scratch, if this isn’t the case and there is a lot of code in zebra we can reuse then I agree that the current estimate may be too long.
This would be partly dependant on the required indices being available in zebra, the main one I’m thinking of is compact blocks, and I know this may not be of the highest priority currently so have allocated time in the grant to implement temporary functionality if the required work cannot be implemented in zebra in time.
Remote ReadStateService / Indexer RPC Service:
I am still unsure of the exact form this will take, I believe there is currently a couple options being discussed: A (direct) wrapper around the ReadStateService, whether gRPC or something more simple, and/or adding a second set of gRPC services or extending the current lightwallet gRPC services to provide the indices that “cli-wallets” will require.
Because of the uncertainties with this and the possibility that both options may be required I would be hesitant to reduce the allocated time for this by too much.
IndexerMempool:
I agree that implementing the bare functionality for this will not be hard, and yes I agree re implementation details. There are a couple things which i would like to include that will slightly increase dev time. First, if possible it would be preferable for the IndexerMempool to get its data directly from the ReadStateService and only use the JsonRPC interface as a backup. And I also think its worth allocating time to writing a solid set of tests for this process.
IndexerStateService:
Part of our original proposal is that we would write this code so that functionality can be consumed as a library. The IndexerStateService is meant to be the endpoint to this library. The plan is that it will have the same set of methods as the remote services offered by a live Zaino Server (both the lightwallet gRPC services and the extended set of functionality for “cli-wallets”). This way “cli-wallets” or block explorers that want the option of accessing chain data directly don’t have to worry about TrustedChainsync, ReadStateService set up or mempool creation, they can just import an IndexerStateService.
Also for all of these points while getting the functionality implemented may not take too long I really think its worth putting in the extra time to implement a solid test set. While it may seam like extra work now I think it will lead to a much more robust service and a service which is much nicer to maintain.
I’m happy to discuss any details though.
7 Likes
Is this about the order of the tasks, their magnitude or both?
Because if the overall amount of work is the same then the grant proposal can be signed off for ZCG to decide on it.
And also: This is more like an implementation detail or anecdote, but I’ve worked with the LWD code to include small functionality and it’s not the greatest and most extensible code I’ve seen to be honest 
. That’s probably why the path of enhancing LWD never was a strong option in the menu
1 Like
Apart from ZcashD deprecation, it is urgent that lightwalletd by deprecated ASAP because it is a demonstrated critical attack surface that has been exploited to eliminate lightclient functionality across the ecosystem.
We urgently need to eliminate lightwalletd, as a separate concern from zcashd deprecation.
1 Like
It’s about time. This will take 5 months to complete, assuming it will not miss the deadline. But what if extending the Go lightwalletd would take 2 months? Then this would actually delay Zcashd deprecation.
I like the idea of deprecating the Go ligthwalletd. But that does not mean we need to do that right now.
(Of course, it’s also possible that everything else will take 5 months and getting this part done sooner wouldn’t gain us anything)
TL;DR
- If we’ve overestimated time to completion, then we’ll soon know more (in ~10 weeks), and we’ll take on more zcashd deprecation work.
- If we’ve underestimated time to completion, then we’ll soon know more (in ~10 weeks), and we’ll thoughtfully allocate more resources to parallelizeable tasks.
- My bet is that this estimate is realistic.
Longer
Does anyone have an estimate for zcashd deprecation where the other components take less than 5 months?
Is that a real possibility? I am willing to place a bet that zcashd deprecation is not within 5 months, that zaino will be done long before that, and that far from being a blocker it will be an invaluable tool that will accelerate the development of other components, especially by allowing for flexibility in test.
The following is an incomplete and informal list of some tasks that I believe remain for zcashd deprecation. I assume that there’re a surprises in each of the below-listed projects, undiscovered complexities.
zebrad regtest mode
When this is finished that lightclients can start validating zebrad for lightclient usage.
Script Engine
I last heard that the script engine would take about 2 months to be code complete, but that’s not counting review.
Last I heard review had not even started. (Though that was about a month ago… I think.)
NU6
My impression is that NU6 needs more time in testnet, with concomitant attention from (I assume) ECC engineers.
Wallet
Has wallet design even started in earnest?
As Far As I Can Tell, there’s not agreement in who/what/how wallet will be created?
ASIDE: I find it a bit jarring to hear about “the cli wallet”… when zingo-cli has been around for years, and has always had t-address support (BTW). I was also surprised to learn that the Zenith grant mentioned a vacuum being left in the wallet space by the disappearance of zecwallet… but, of course, both zingo-cli (a CLI wallet) and zingo-pc a desktop wallet, have existed all along.
… ← All the pieces I am not thinking of, because I am not the zcashd deprecation coordinator.
If all of the above (plus other necessary components I am not aware of) is going to be done in less than 5 months, then we might need to roll with lightwalletd as an emergency stopgap, even though it’s a critical attack vector.
If there are folks who have worked on lightwalletd in the past, and they want to play in the new space, let’s get them into extending zaino’s test capabilities for use by all.
Finally, in the event that zcashd deprecation seems lilkely to block, we can add developers to some parallelizeable tasks… particularly test.
2 Likes
I think the name “cli-wallet” is a bit confusing here as its not the CLI interface that is actually required. It is the JsonRPC interface that exchanges need. I think “Full Node” RPC/API wallet would be more descriptive (but maybe not as elegant…).
Im also unsure of the plans for Zenith and whether they fit the requirements. My belief was that ZF wanted a wallet that uses the ReadStateService (or IndexerStateService) to access chain data directly which is not how zenith currently operates.
2 Likes
I agree the name is confusing, I have raised a similar point somewhere. Maybe we should simply brand it, I think @Dodger suggested “Quagga”. Or just “RPC Wallet”?
I’m not sure about Zenith. I don’t think the issue is the desire to use ReadStateService to access chain data directly - that’s seems the ideal solution but is not a requirement IMO. I think the issue is that the overlap between Zenith and the CLI-wallet/RPC-wallet/Quagga is small enough that is not worth trying to reuse it somehow (i.e. if I understand correctly, Zenith is GUI and we want JSON-RPC, Zenith talks to a node directly and we want to talk to a lightwalletd-like process…). Also it being Haskell makes it difficult for everyone else to try to work on top of it.
3 Likes
Well, not everyone :).
2 Likes
4 Likes
@zancas at the most recent meeting, the @ZcashGrants Committee voted to approve your revised proposal and has requested that you provide monthly updates via the forum in this thread. This approval is contingent on @daira & @pili confirming the amendments to this grant via this thread.
10 Likes
And if you can get some details about what the exchanges currently use from zcashd, it would help a lot in ensuring the new tools provide the same capabilities.
1 Like