ECC not requiring User-Defined Asset precursor support in Orchard

I speak only for myself when I say I am a bit offended by this framing.

I came here because @tromer asked a question about new NU requirements that suddenly appeared on a ticket with no community discussion, requirements that surprised me because as someone who has been trying to stay closely involved with the technical progress of Zcash I had no warning about them. Nor did @tromer.

When two of Zcash’s founding scientists show up on a board and say “hey folks what’s going on with this feature we are confused” and two ECC engineers’ entire response is to show up to explain how offended you are, this is not the right response. There is clearly a communication problem here that needs to be addressed.

ECC engineers talk among themselves regularly and don’t talk much to the Foundation or outside scientists anymore at all. It did not used to be this way, and in the past things worked better. (For example, I used to be able to design light wallet protocols for you.) Posting notes in obscure Github tickets while neglecting real time communications is not transparency. To the outside community (which now includes the founding scientists and all your technical counterparts at Zfnd, folks who have done plenty of work eagerly for ECC in the past) this doesn’t feel like real transparency. Instead it feels a lot like “posting your plans in a filing cabinet in a disused lavatory in the basement of the local planning office, why didn’t you know to look there” kind of transparency. Why not just talk to people? It’s easy. We’re really pretty nice.

Please, consider the fact that many people outside of ECC have poured their heart and soul into Zcash and want to continue to contribute despite the immense effort it takes even to get ECC folks to consider our feedback. And if Zcash’s scientists and basically everyone else in the community tell you that things are confusing and they don’t understand what you’re all up to, you should just try to listen to them and change things. We would love to help you and are not your enemies.

ETA: @tromer’s response says this all better and in much more detail, but does not as analogize the transparency of your requirements process to the Hitchhiker’s Guide and therefore this post will stand. (Actually he does. Maybe the metaphor really is apt.)

ETA2: Speaking of light wallet privacy, here’s some recent research work we’ve been doing that might be helpful. Cryptology ePrint Archive: Report 2021/089 - Fuzzy Message Detection

8 Likes

You are correct that there is a communication problem here. Please don’t misrepresent my post:

  • I replied before you had participated in this topic.
  • My response was in part precisely because the top post was not simply “hey we are confused”, but appeared to be mis-representing the requirements document (I appreciate @tromer clarifying that it was not his intention).
  • Well over half of my response was a substantive on-topic technical reply.
7 Likes

@tromer Thanks for your thoughtful response. I greatly appreciate it, particularly when contrasted against a lot of the negativity in parts of the community.

I opened up on how I felt–as I mentioned, I almost didn’t speak up. For better or worse, engineers adopt a certain level of “thick skin” and retreat to GitHub where we can focus on facts, rather than feelings.

I trust that people are voicing their concerns in good faith (as opposed to sabotaging), precisely because they care.

Judging by outcomes, something surely is broken yet some approaches are more destructive than constructive. As a founding scientist, your statements reverberate and can be corrosive to community and morale if they are inaccurate or appear inflammatory.

I strongly agree that fixing communication is key but the burden is probably not all on ECC.

RE: ECC’s effective (non-)transparency

A few things stand out to me as I read and deeply think about your post:

  1. Resource constraint. From a Wallet Team perspective: We have one Android Developer. One iOS. One person who updates lightwalletd as HALF of their job. I’m not sure our team could be any smaller without people further taking on too many roles. And I won’t even get started on how much work @str4d does!
    In contrast, 3 jobs ago, I worked on a coupon app that had 32 mobile devs. I know former coworkers on app teams with over 100 Android developers, alone.
    On the other end of the spectrum, the average small team cannot spare the time to do technical design “by committee.” Instead, they are trusted with a high level of autonomy, especially when it comes to complex technical decisions and complex trade-offs.

  2. Trust. Does the community really trust ECC and vice-versa? It’s risky to wade into the forums without a “Hazmat suit” because we’re all mired in a vicious cycle of distrust. Thankfully, everyone wants to fix this (modulo the occasional troll).
    So lets continue to work to repair the trust wounds.

  3. Negativity is a communicable disease. Toxicity in the forums is disproportionately spread like wildfire. Out of frustration, when those in position of influence make fairly imprecise assertions like, “ECC … no longer communicates the details of what it’s doing,” it gets amplified into other contentious threads and I’m not convinced that process is constructive. That post lead me here.
    On a personal note: I was a bit offended by this, in the moment, because I was awake past midnight on a Saturday collecting information from the forum so I could put it into a newsletter that I’m releasing Monday to explicitly communicate technical details for Light Clients. This will be my 7th such article. Yet two of my favorite scientists are taking actions that might do more to contribute to toxicity than to neutralize it.

  4. Help not harm. How can the “outside community” be a part of the solution? Ignoring that the plans were on display analogy probably disparages or at least trivializes the hefty efforts we do make, who can help us retrieve the content from that “filing cabinet in the lavatory” (given that we seem to agree it exists, publicly)?
    Could more of the roughly $15 million of ZF funds be put toward this reoccuring problem? Can ZOMG fund a team that specializes in technical communication to lend a hand?
    Perhaps we could try to use this interesting tool as the one source of truth?

Let’s get to a healthier place. I think that requires a concerted effort to improve communication not just from ECC but in all directions.

12 Likes

I’m personally fine with this being added as a requirement, given the general community interest. It’s certainly not something I plan to allow to regress.

8 Likes

@gmale, I hear ya. No easy constructive answers, so let me ponder this.

Also in the spirit of improving comms and reducing hazmat:

  • I changed the topic title to better reflect the (clarified) semantics.

  • It has just dawned on me that @str4d’s first post meant the same thing as @daira’s subsequent post, and thus was very relevant to the “should” question that I posed. I misconstrued it as talking about what would happen if we did include some requisite UDA precursor support contrary to the requirements, but not full UDA support. Sorry, @str4d!

7 Likes

Hey,
I don’t’ think i’ve seen any community comments or anyone complaining about the wallet team (other than why can’t we ship an actual wallet … which I know the wallet team worked their ass off to try and do and then were blocked from doing so).

You guys are doing an awesome job Yeah, some things can always go better, but none of this is directed at you. Or, as Eran made clear, even ECCs engineers generally.

5 Likes

Yes! Both of these things can happen. That’s why we have these funds. But for them to happen there has to be some engagement, and an assurance that ECC will cooperate with any additional efforts made in this area. It is very hard to contribute to a single software ecosystem (without forking it) when you aren’t sure that people you hire will be able to work closely with other teams. This is what’s lacking right now and preventing us from building exactly the big teams you want.

To be more specific with examples, last year we at the Foundation debated a major investment towards readying UDAs by taking the work @str4d had done and Tezos had paid to “productize”, and bringing it back to Zcash. This seemed like an obvious win since we could leverage the cash Tezos had already spent. We certainly had the budget and we had a contractor with experience who could have done a lot of the work. But ultimately this ran aground on the fact that the existing solutions are Sapling-based and wouldn’t necessarily be any good with Halo, which is prioritized over further Sapling upgrades. That’s fine, but it also means that spending any money on feature upgrades without ECC buy-in would be like setting money on fire. We don’t even have a full spec for what’s going into the next NU circuit so we can’t even think about features at this point. This approach and lack of coordination means that instead of Zfnd and ZOMG using their cash as a force multiplier, everything is bottlenecked on ECC who talks to us on a monthly 1-2 hour call.

With regards to the wallet: ZFnd is already funding multiple partial re-implementations of the light wallet, and I’m sure we (and I hope ZOMG too) would be thrilled to hire people and put resources into a coordinated effort to bring everything back upstream and invest in a larger team. This is almost the definition of what these funds are for! But to avoid yet another wallet fork, we need to do this in close cooperation with ECC’s team. This requires wide open communication channels and a guarantee that ECC will work closely with any devs we hire. I think that the wallet team would probably be open to this and we should talk about how to do it. (Real ‘action item’ — let’s set up a call this week to talk about how to do it, ok? Write me on my main email or ask @str4d for my Signal.)

Basically there’s money and a lot of people who want to help. But the current approach where ECC controls consensus dev and upstream and there’s loads of money that wants to help but can’t get bandwidth with ECC engineers and management drops requirements out of the blue — that’s strangling Zcash. And if recognizing a problem and trying to fix it is “toxic negativity” then for god’s sake, let’s all be more toxic.

5 Likes

I just wanted to thank you for all the work you do. Having a healthy set of Android clients is an absolute must for Zcash adoption. The SDK enables it .

8 Likes

Hey folks, real quick, it is not accurate to say that ECC management is distancing, or failing to engage with others, and I think the facts amply demonstrate otherwise. As Kevin eloquently said, everything we do is public, everything is transparent, everything is subject to scrutiny, and we are proactively working to empower people in the community to give useful feedback, as early as possible so that we can integrate it into our planning, and moreover to contribute to Zcash directly themselves, in the various ways that they can contribute, instead of contributing only indirectly by informing or influencing ECC.

With regard to Halo design, as we previously stated, we’ve collected a great deal of useful feedback/questions/objections from various channels, especially the most recent open community Arborist call, we’ve documented and published all of our current, evolving requirements/non-requirements documentation as we go (which is why this thread is possible), and we’re working hard to provide answers for the questions/objections we have answers to, and to organize and document which ones don’t yet have answers, which we expect to have ready to share at the next Arborist call.

There are also a lot of other statements in this thread that I think are inaccurate and if you’re reading along and you’re not sure, I’d urge you to double-check against the facts. Judge people by their actions, not (just) their words.

I’m sure we’re not succeeding perfectly! Of course not. Nobody is perfect. But as Kevin eloquently wrote, we’re pouring our hearts out, trying our hardest, working day and night and weekends, and in my estimation, even though we’re not reaching perfection we’re doing pretty damned good.

The specific technical issue in this thread is about whether we should include features into ZIP-224/Halo which could then be used in the future for UDAs is simply a misunderstanding or a disagreement about engineering strategy. One approach to engineering strategy is to “build in” future capabilities into the next version. Another approach is to build and ship a series of ship narrowly-scoped improvements. At ECC, we take the latter strategy.

We believe that building and shipping a series of narrowly-scoped improvements, as soon as we can safely do it, is a good engineering strategy for the long haul. We believe this is a good way to support the delivery of high-quality UDAs/ZRC-20’s, and other potential future features, as fast as is safely possible.

One of the biggest benefits of this strategy is that shipping Halo/ZIP-224, we eliminate trusted setup immediately! In addition to thus eliminating one of the most, if not the most, widely-cited reason for distrusting the soundess of Zcash’s monetary base, this also enables all possible future upgrades to launch without needing a setup ceremony that would put everyone’s money at risk and introduce a fresh reason for people to doubt the soundness of the Zcash monetary base.

But in any case, this issue is simply a disagreement or a misunderstanding about different strategies, both of which are valid possibilities. We could be wrong! We’re doing our best to learn from others and to be less wrong. Any insinuation that we are unwilling to engage or that we have an ulterior motive is simply false.

8 Likes

@zooko, rather than dwelling further on consternation and counter-consternation, I’d like to focus on the original specific topic of this thread. My concrete question to you is:

Suppose it were discovered that the simplest/easiest Orchard design was not compatible with UDAs, and adding UDAs would require new a address format and/or a new shielded pool. Yet with just a bit more complexity/effort, we could ensure that Orchard would allow for future addition of UDAs without those disruptions.

Which of these two Orchard designs should be chosen?

You seem to imply that we should choose the design that incompatible with UDAs, because of your belief that shipping a series of narrowly-scoped improvements, as soon as we can safely do it, is a good engineering strategy for the long haul. And if this delays UDAs, say by a year or more because of the disruption of new addresses and pools, then we’ll just think about that later.

Is that accurate?

(This is the dilemma as I originally understood it. @str4d and @daira later explained that this dilemma is unlikely to arise in their current technical approach. Yet it still makes sense for the Requirements to express the intended goals, rather than the opposite, lest they lead us to losing capabilities we thought go without saying.)

7 Likes

Wow this thread took a really constructive turn in the middle! Thanks @gmale for expressing vulnerability to adversarial suspicion vibes, thanks @str4d and @daira for explaining how chill and non-zero-sum a decision this is actually is, and thanks @Matthewdgreen and @tromer for switching modes!

@Matthewdgreen, I agree with what you’re saying about the sheer volume of discussion and documents created by ECC over the years being overwhelming in a way that hurts transparency.

But my sense is that it’s the organic result of a small team with some very prolific people who have a ton of ideas, working full-time and in the open. They put out a lot of stuff. This might be distracting to them somewhat internally from an efficiency standpoint, and it does create a lot to wade through, but I think it’s important to recognize that it comes from their being full-time, obsessed with what they’re working on, and prolific and doesn’t come from a desire to obfuscate.

In my experience as an outside contributor working on Zbay, it was difficult at first to get information and support about Zcash, and to get attention to PR’s (we had one about viewing keys that was languishing for a while) and to figure out what the priorities were.

I think I first reached out to Sonya at ZF, who then connected me to Elena, and they both oriented me. They mentioned they’d be at ETH Denver, and there I got to meet everyone, and everyone was really solicitous and helpful in getting me connected, and seemed to really be listening when I expressed difficulties about getting connected. And all of this was before I had any particular official position in the ecosystem: I was just some developer/user with a cool thing built on Zcash.

I think it might actually be the case that some of the current bi-monthly calls and async communication channels were partly due to my requests for such channels—or at least if they weren’t they fit the needs of my project for updates and connection to expertise. The weekly calls, and the newsletters @elenita and @gmale put together have been super useful for keeping up on things. Discord has been super helpful for getting timely support for problems with Zcash tooling or questions about capabilities from str4d and daira.

It took a little poking around to get answers, but this is natural and very common, and the willingness to make efficient connections between me and the information I needed was definitely there, and everyone at ECC followed through on the commitments they made to me, which is the most important thing.

I recognize that you’re both in different positions as folks with very deep connections to the project but also tons of other commitments, and without a full-time Zcash project you’re working on (is that right?) So I can totally see that your needs are different. We’ve been talking about the need for some high-level update for people at ZOMG too, since several of us don’t have time for all the calls— @ml_sudo brought this up. Perhaps we could work together on figuring out how to get you what you need?

@tromer My sense of the state of the art in software project management is that it’s actually (counter-intuitively) considered most efficient to work in a direct line to the most simply-scoped releasable piece of user value, even if intuitively it seems that preparing for future features would save you time—and that this is true even when you know you want to tackle a future feature.

The only safe way to use information about desired future features is when there is literally no cost to doing things one way or another—and then you only use it to make sure the door is open to the future feature; you don’t build for it or make it a requirement. (And it seems str4d and daira agree that the door is open to UDAs.)

The logic behind this rule of thumb has something to do with compounding delays, the risk of roadmaps changing, the cost of work that is never released, and a cost of managing “inventory” (unreleased features) similar to the cost physical goods supply chains have. But I suspect it comes less from logic and more from decades of really brutal empirical experiences with over-scoping, and people trying this approach instead, and it working better.

So I’m not sure if Zooko will respond here, but I think the approach he’s giving is the industry standard one.

This is a separate question from whether to prioritize UDAs or Orchard, and the same argument kind of pushes the opposite way towards releasing UDAs before Orchard. (Optimize for short-term releasable user value!) But the trusted-setup issue is user value as well, albeit in a competitive sense with other chains, so it’s a tough call!

The thing that convinced me about prioritizing Orchard/Halo over network layer privacy or UDAs is the fact that UTXO’s won’t need to be managed, and tx’s can be sent while funds are still pending. This is probably the biggest Zcash usability issue, from my perspective. It’s really hard to work around it in the lightwallet context, because the client side SDK doesn’t really give you much control over this. I’m not sure exactly what the functionality looks like post-Orchard but that seems like a great reason to prioritize it right there.

7 Likes

@holmesworcester, you touch on many important points. I agree with the spirit, but not with all of the facts or conclusions, so let met point out a few places where our view diverges.

Indeed! Except that @gmale’s superb newsletter, which is the more technical and developer-oriented of the two, is visible only to the Light Client WG and can’t be read by others.

Discord has been super helpful for getting timely support for problems with Zcash tooling or questions about capabilities from str4d and daira.

Yes, they’ve been amazing with their time and helpfulness. Just note that this is different than the problem of us even being aware of which major design and decision processes are undergoing, which is the (side) topic at hand.

I recognize that you’re both in different positions as folks with very deep connections to the project but also tons of other commitments, and without a full-time Zcash project you’re working on (is that right?) So I can totally see that your needs are different. We’ve been talking about the need for some high-level update for people at ZOMG too, since several of us don’t have time for all the calls— @ml_sudo brought this up. Perhaps we could work together on figuring out how to get you what you need?

:+1:
I made some specific suggestions above, and await a response to these.

You seem to imply that we should choose the design that incompatible with UDAs, because of your belief that shipping a series of narrowly-scoped improvements, as soon as we can safely do it, is a good engineering strategy for the long haul. And if this delays UDAs, say by a year or more because of the disruption of new addresses and pools, then we’ll just think about that later.

@tromer My sense of the state of the art in software project management is that it’s actually (counter-intuitively) considered most efficient to work in a direct line to the most simply-scoped releasable piece of user value, even if intuitively it seems that preparing for future features would save you time—and that this is true even when you know you want to tackle a future feature.

The only safe way to use information about desired future features is when there is literally no cost to doing things one way or another—and then you only use it to make sure the door is open to the future feature; you don’t build for it or make it a requirement. (And it seems str4d and daira agree that the door is open to UDAs.)

The question at hand was not whether to actually build the the UDA feature, which is indeed a large endeavor. The question was whether to require that the low-level protocol will have the capability for adding the UDA feature in the future without the huge disruptions such as new address format and new shielded pool. What is the cost of preparation is of course relevant, but was never publicly discussed until I opened this forum topic. Discarding a requirement because it doesn’t have a cost is ludicrous.

A good analogy is whether to require, in your database structure, that all all record counts and indices must be 64 bits, because you know that within a year you’ll need that, even though your first release only supports databases up to 4GB and that’s all your regression tests check. Sure, explicitly saying that 64-bit indices are not a requirement may get your product out the door a day earlier, because it’s one less thing to review and maybe fix, but you know that if you ship with 32-bit indices, you’ll be paying dearly by forcing your users into a major format migration within a year, and you’ll need to write the migration tool and do migration handholding. (Plus in the world of consensus rules, you’d be forcing all your users and developers to migrate.)

Now the database programmers show up and say: we’re not stupid, we’ll use 64-bit indices anyway, so we’re fine that it’s not a requirement. The only important thing is that the prioritization and requirements-setting process doesn’t get in our way of doing what we were planning anyway.

Does it still sound like state of the art in software project management?

To me it sounds like: the planning and decision process is a mere distraction; a performative ceremony whose only purpose is to retroactively appease the bosses and transparency freaks; we engineers only care that the process doesn’t get in the way of us doing the right thing. I emphasize, and yet I observe the inevitable fallout.

This is a separate question from whether to prioritize UDAs or Orchard, and the same argument kind of pushes the opposite way towards releasing UDAs before Orchard. (Optimize for short-term releasable user value!)

Totally separate question indeed, though I happen to agree with the conclusion.

The thing that convinced me about prioritizing Orchard/Halo over network layer privacy or UDAs is the fact that UTXO’s won’t need to be managed, and tx’s can be sent while funds are still pending. This is probably the biggest Zcash usability issue, from my perspective. It’s really hard to work around it in the lightwallet context, because the client side SDK doesn’t really give you much control over this.

Uhm? This would surely be a major improvement, but I had no idea that it’s planned. The relevant sections of the Orchard book don’t seem to mention this. Can you say/link more?

1 Like

So, from the document we’re responding to: “Getting precursor support right requires certainty about a subset of UDA requirements, and blocking Pollard on clarifying future UDA requirements introduces more deployment & execution risk.”

To me, this statement sounded in line with the state of the art in product management.

If you have a path to release a feature you consider valuable (e.g., eliminate trusted setup) you stay disciplined and don’t let a future feature (UDAs) add to the scope of that release.

You release, and then do the work to add the second feature in a future development cycle, even if this means re-doing or undoing some of the previously done work, because the cost of redoing it is usually worth the de-risking and process value.

I’m not sure where I saw this, and I can’t find it now in the descriptions I do remember reading, so I’m worried I might be totally confused or wrong. Embarrassing and sorry for the static!

Also, curious to know whether I heard/understood/remembered correctly!

1 Like

Also, agreed on making the lightwallet newsletter public, and I think your suggestions above all make sense, though I think it’s a tricky question what audience to focus on in this public communication and feedback solicitation. Should it be the forums? Insiders/experts? The general public?

There’s some extent to which these are different audiences with different needs, and you have scarce resources so you can’t allocate across all of them and do a good job.

On the topic of communication, one of the points I made earlier that I’d like to emphasize is resource constraint. Personally, I love the idea of enumerating everything in flight but who will curate and maintain it? In practice, doing so effectively would require reading countless forum posts, discord chats, github issues, research papers and attending community calls while being technical enough to understand complex issues yet eloquent enough to distill them for public consumption.

Since such content would become stale and inaccurate almost as fast as it’s created, I’d argue that this would be a full-time job of multiple people (in addition to the technical team to support the operation). Above in point #4, I suggested that the community could be part of the solution.

Put simply, nothing prevents a 3rd party from assembling all of ECC’s activity into a website. In addition to providing great value, such a team might also free everyone else to get more done!

2 Likes

My suggestion above still stands. Apparently there’s ambiguity about what the original phrasing (“precursor support for a future User-Defined Assets feature”) means. My suggestion is a clearer functional requirement and we already know it can be achieved at no extra cost.

1 Like

This is irritating. What you describe would be unprofessional, and we’re not unprofessional. We take planning very seriously.

The document said:

Non-Requirement: The protocol does not require precursor support for a future User-Defined Assets feature.

Rationale - Technical Strategy: Getting precursor support right requires certainty about a subset of UDA requirements, and blocking Pollard on clarifying future UDA requirements introduces more deployment & execution risk.

This is all completely correct: we can’t predict what support would be needed, and blocking on clarifying that would indeed introduce deployment and execution risk. As it happens, the current design does not present any obstacle to future UDA support, as far as we can tell (given a fuzzy idea of what the requirements are likely to be). But would we have gone ahead even if it hadn’t? Yes, absolutely, and that would be the right decision, in my professional opinion.

When I see @str4d et al. doing amazing work on UDA, that’s so effective and high-quality that it’s already deployed by Tezos, but then ECC management squelches it […]

This is inaccurate. First, the current best proposal is more efficient than what Tezos deployed (since it doesn’t require a hash-to-curve in the circuit). Second, it hasn’t been squelched. I would never have expected UDAs to be deployed without thorough investigation of the economic issues, which is happening.

@holmesworcester wrote:

The thing that convinced me about prioritizing Orchard/Halo over network layer privacy or UDAs is the fact that UTXO’s won’t need to be managed, and tx’s can be sent while funds are still pending. This is probably the biggest Zcash usability issue, from my perspective. It’s really hard to work around it in the lightwallet context, because the client side SDK doesn’t really give you much control over this.

I’m not sure what you’re referring to here @holmesworcester; Orchard doesn’t differ from Sapling in this respect. It’s likely that we’ll implement some form of automatic note management, but that would apply also to Sapling, and doesn’t require consensus protocol changes.

1 Like

But this is exactly what I disagreed with, in my top post. You dismissed this disagreement by saying it’s moot in the current design, and yet you ignore my suggestion to make it an explicit requirement to keep it moot. You can’t have it both ways.

Yes, we disagree. I didn’t “dismiss the disagreement”. I don’t think it should be a requirement.

You may have misinterpreted the statement “That’s part of the reason why I personally didn’t push back too hard on it being a non-requirement …”. Another important part is related to what @holmesworcester said: it simply isn’t good engineering or cryptographic practice to speculatively include feature support that might end up being inapplicable. The cost of unused complexity in a cryptographic protocol is extremely high. One of the reasons why Zcash and Sapling has had a good record in deploying complex functionality with very few bugs, is precisely that we avoid excessive generality. Zcash is not TLS, and never will be.

3 Likes