Grant Proposal: Zcash Extensible Wallet Interchange Format (ZeWiF)

Community Grants Applications
1

Blockchain Commons & Zingo Labs have partnered to put together a proposal to aid with the zcashd deprecation, to improve interoperability among wallets, and to help salvage legacy wallet funds that might otherwise be lost.

It’s the Zcash Extensible Wallet Interchange Format (ZeWiF), submitted in the new GH system:

As noted, this is a partnership between Blockchain Commons & Zingo Labs. Here’s some of the bio info that didn’t fit into the new grant format:

Blockchain Commons has been working for over five years on bringing together wallet developers across the blockchain ecosystem to create secure & interoperable systems that protect the human dignity of their participants. This project will allow them to extend that expertise to the Zcash ecosystem.

Some of Blockchain Commons’ major achievements for wallet design include the introduction of an interoperable Animated QR specification (Animated QRs - Developer Resources), formulation of the Lifehash visual hash specification (LifeHash - Developer Resources), creation of the security-tested Sharded Secret Key Reconstruction library (GitHub - BlockchainCommons/bc-sskr: Sharded Secret Key Reconstruction (SSKR) reference library in C), and development of the dCBOR (draft-mcnally-deterministic-cbor-11 - dCBOR: A Deterministic CBOR Application Profile) and Envelope structured data formats (draft-mcnally-envelope-08 - The Gordian Envelope Structured Data Format) for the IETF. More information on Blockchain Commons’ work for wallet design can be found in the 2023 Yearly Overview (Blockchain Commons 2023 Overview) as well as overviews from previous years (Posts by Tag - Blockchain Commons).

Zingo Labs is a well-known wallet and infrastructure development team in the Zcash community. They contributed to the early implementations of the NU5 and Orchard protocol clients, contributed to core code in librustzcash, zcash-stack, and zebra and are the primary stakeholders of the Zaino lightwallet-proxy project. They have extensive experience with zcash wallet formats, including lightclient wallets, that make them ideal zcash-specific partners for the blockchain-agnostic Blockchain Commons.
==

We think this is work that is crucial for the zcashd deprecation, but also something that will improve the whole Zcash infrastructure through new interoperability.

15 Likes
2

The wallet format is one thing, another thing is how secret keys are derived. For which there is a zip but not complete agreement from wallet developers. For instance, Ywallet doesn’t follow it for performance reasons. Transparent addresses are not scanned, etc.
Also, current wallets do not support the full extent of the keys from wallet.dat (and certainly not the sprout keys), therefore even if they understand the zwif format, the end user would not be able to recover their funds.
Many people user ywallet as a recovery tool and in 90% of the time, they have the seed so it’s not a format issue.

A common format is still a good thing to have moving forward, but I wanted to point out that backward compatibility may not be achievable and/or worthwhile. In which case, the work is much easier (tedious but simple).

2 Likes
3

Good points, and I agree with most of them in-principle, but differ on the degree.

We are considering different key derivation patterns and this is an issue that I believe the Block Chains commons team have successfully dealt with in multiple cases previously.

Per this assertion:

Are you referring to the “naive diversification” via secret-key-generation strategy that zecwallet employed? If that’s the problem you’re considering, then we agree that it’s significant, and intend to specifically design a tool to recover all knowable funds in that case. It’s exactly this that is motivating the development of ZExCavator.

I am in 100% agreement with you on this point. It’s important to not over-sell what can be recovered, it’s a function of both which wallet saved the data, and which version of that wallet. That having been said we can recover at least some and possibly all funds from the ZECWallet family of wallets with some caveats (mainly around “naive-diversification”). We believe that, in and of itself, makes the modest cost of the tool well worth the effort.

But even if the tool itself isn’t very successful, a real selling point of this work, is that it invites other instances of ZExCavator-like tools. So if our reference implementation can be improved on, for example, by an expert in a particular legacy wallet, then it will be quite easy to export the data from one tool, and load it in another.

1 Like
4

This grant addresses one of the paths of the Zcashd deprecation dag and sets Zcash wallets to have a common, well designed, extensible and interoperable

The new format can accommodate existing wallet data so that tools can be developed to make use of it. I understand it’s not plug-and-play backwards compatibility because as Hahn pointed out it’s hardly achievable.

It’s backed by renowned and experienced developers like @ChristopherA that has many years of experience in the field and has successfully designed and deployed many formats like these before.

Thanks you Blockchain Commons and Zingo Labs for putting all of this effort and ECC engineers for your help reviewing and always being well predisposed to help and give your honest opinion and feedback

Best of luck to the grant candidates!

2 Likes
5

My comment was about setting expectations regarding the usability of the grant.

The section about risks around adoption is my biggest concern. IMO, people would expect the Zewif format to be the ultimate interop solution and put the blame on the wallets when they can’t recover their funds. I don’t think two wallets that adopt zewif will always show the same balance. It feels that the technical description is not clear enough in this area, and end users could misinterpret the capabilities.

In other words: “don’t be upset if zewif can’t recover your funds in ywallet”.

My point is that these tools are going to be lossy. Wallet developers are aware of the technical details, but the users are not.

I wouldn’t say the cost is modest. I agree this tool is good, but it doesn’t depend on zewif. These users, more often than not, have lost access to their wallet file and just have the seed.

I think the beginning of the grant is good, but then it adds work that isn’t going to pan out.

IMO, it’d be good if the grant enumerated the use cases where it guarantees success and clearly showed its limitations.

PS: I had considered writing a free tool for Zecwallet, but after analysis, its scope was more extensive than expected and it was too challenging to implement efficiently (that was during the spam period). It’s based on a conversion tool from zcashd to zwl format and the parsing of zwl file in ywallet. https://www.youtube.com/watch?v=Shvfdx4aZwM

Good luck with the grant

3 Likes
6

OMG yes please!!!
Been trying to recover funds for well over a year, and keep hitting road blocks. And I know many others with the same issue who’ve come to me privately. This would be an amazing tool for the community, thank you so much for working on it!
:pray: :pray: :pray: :yellow_heart: :yellow_heart: :yellow_heart:

8 Likes
7

It’s important to note that we want to account for the broadest possible set of existing wallets, so if you have time please share any wallet schema for YWallet with @dorianvp !

1 Like
8

Great to hear from Real World Zcashers demanding real utility.

2 Likes
9

It just uses zip-32 and bip-44.

1 Like
10

The demand is real but I am still not convinced this would fix the use cases @NaomiBrockwell mentioned.

For ZWL, you need the seed + number of transparent addresses + number of shielded addresses, because it uses bip-44 and zip-32.

With the zingolib (or warp), you could write a recovery tool in less than 1h.

To be clear, I am not saying ZeWIF is a bad idea, I am arguing that the proposal states:

  1. People want a tool to recover their funds
  2. ZeWIF would offer that
  3. Let’s make a standard where for all wallets
  4. Then people can interop between wallets.

Imo, 1. is true. But it does not imply that you need 2, 3, 4. I am sure you will not get every wallet to switch to the zeWIF, because we can’t even get wallets to switch to UA (or even consensus branch id).

2 Likes
11

Push button recovery for multiple keys as generated by historical ZECWallet is explicitly covered in our plan.

In the case that the user has a wallet.dat, the ZeWiF format will account for the set of keys recorded. In the case where the user only has a seed, a gap rule heuristic can be used.

The output of the tool will be a wallet-file in a standard format allowing the user to select any wallet that supports the standard, eZcash, Zashi, Zingo, YWallet?

Let’s give the User control.

My understanding of @NaomiBrockwell 's problem statements is that she specifically cares about these cases. Assuming that she’s representing a group of private zcashers, that alone more than justifies the tool.

I have yet to meet a User that objects to the idea that they could then choose whichever (standard format supprting) wallet they want to subsequently manage their recovered funds.

1 Like
12

I understand the tool.
I don’t see the need of zeWIF. The user won’t object to it. I am saying it won’t be available in practice because wallet devs have to adopt it.

Anyway, I think I made my point clear. Good luck on your grant.

PS: I think we can crowdfund the recovery tool.

13

Yes. Please, build this. This is absolutely needed.

Zcash needs to become known as a community that takes care of its own.

I have had so many people I have tried to onboard be scared off by the idea of losing funds because of reliability concerns.

I’m a professional software engineer myself and have been trying to solve this problem by going back and building old images and it has been an absolute nightmare. I just don’t have enough time to dedicate to this. I can only imagine what it must feel like and look like to people without technical savy.

6 Likes
14

Cool!

Well the intention is that this tool will be something that folks like you can both use, and enhance, for any specific issues that you’re facing.

Maybe you’ll just open issues on the repo letting us know about specific problems…

but maybe you’ll extend the format yourself (because ZeWiF is Zcash extensible Wallet interchange Format) and enable the tool to recover even more funds, by enabling the tool to handle a new kind of wallet.dat.

Wouldn’t that be a fun PR?

4 Likes
15

sounds great :slight_smile:

2 Likes
16

If you are curious, I am attempting to do this here :rofl:

Stream is over.

7 Likes
18

@BC-Shannon at the most recent meeting, the @ZcashGrants Committee voted to approve this proposal and has requested that you provide monthly updates via the forum in this thread.

12 Likes
19

Thanks! We look forward to getting started with the new year.

5 Likes
20

We’ve figured out a schedule & begun work on the January phase 1 on this project.

To further support it, we’ll be holding a (virtual) meeting on January 15th (10am PT) for Zcash wallet developers & other interested parties.

I posted a full description here:

Hope to see some of you there. Our goal for the meeting is to do our best to make sure we’re not missing anything & ensure that we can accommodate anything that’s widely used (2+ wallets) in Zcash wallet metadata.

Our initial plan is to close out the January milestones by Wednesday, the 29th. We’ll report back afterward!

2 Likes
21

Thanks for all your work hanh. Many of us really appreciate it; as a Linux user since '96 it’s nice to see someone who understands what open source is truly about - even if you do use MacOS which ripped off BSD due to the MIT license as opposed to GPL2 :wink:

1 Like