Request for Input: Securing the Zcash Ecosystem

Community Grants
1

Hi all,

I would like to open a discussion around securing the Zcash ecosystem in the context of ZOMG funding.

When I was campaigning for my seat on the committee I made the following policy suggestions:

And additionally:

Now that the @ZcashGrants is up and running, and proposals have started come in I would like to take some time to flesh out that vision for the future of securing zcash ecosystem projects, regardless of whether they currently exist independently, are funded by ZOMG or yet to be imagined.

Zcash ecosystem projects do not have the best track record on security and privacy, and the cryptocurrency world, as a whole, is not kind and will not hesitate to exploit any holes that might exist. As such it is critical that when ZOMG funds the zcash ecosystem that we take great care to help projects reach the highest levels of security and privacy assurance possible.

In my opinion it doesn’t make sense for the ZOMG to require projects to source their own security and privacy reviews, nor to account for these in their grant applications - this approach will result in duplicate work and funding inefficiencies. More fundamentally, such an approach diversifies the effort while doing nothing to mitigate the brand risk associated with any kind of major public ecosystem privacy flaw.

To that end, I would like to propose the creation of a dedicated zcash ecosystem security team. This team would be funded by ZOMG, and would be available for projects to consult with. Their ultimate responsibility would be to ensure that ecosystem projects like wallets, infrastructure, explorers, and zapps meet the high bar that Zcash requires as a privacy-first project.

I believe that Zcash is in a fairly unique position given the ZOMG and the strong technical foundations of the project to cement itself as an ecosystem that is unmatched when it comes to privacy, but that will only happen if we can back up exciting new applications with strong assurance.

To that end I have a few questions that I would like to put forward to my fellow committee members, and the wider community:

  • What are your opinions on the creation of such a team?
  • Do you have thoughts on the structure of such a team?
  • Do you have ideas on who you would like to see involved in the team?
  • Is this something you would prefer to see handled by e.g. a dedicated/contracted industry partner or constructed from scratch?

Please leave any and all thoughts, ideas and critiques in this thread. I will gather them all up to produce a provisional Request for the ZOMG committee (and community) to discuss (and maybe ultimately vote on).

10 Likes
2

This a great initiative, having a dedicated or contracted industry partner to pen-test and provide feedback to patch vulnerabilities would really help projects in the Zcash ecosystem as most of them (including @NighthawkApps) are working with bleeding edge tech wrapped around work-in-progress code on top of platforms (mobile/desktop) that are regularly facing 0 days. Since we can’t have bug bounties, using the ZOMG funds to have a respectable security researcher in mobile/desktop (primarily end user platforms) will help in avoiding possible blunders.

As for structure, it could be 1-2 rotational positions per quarter/year or hiring them one off per quarter for testing. Having dealt with contacts from https://krebsonsecurity.com in my past enterprise work, I’ve found them to be very professional and helpful for end-users and only going to the media when corporates fail to take action.

3 Likes
3

I agree this matters a ton.

People will use Zcash for privacy, which they can’t get using centralized services, so they’ll be using desktop or mobile apps. And these apps are built by small teams who would benefit immensely from support from a dedicated team of specialists who become experts in Zcash.

I have a couple thoughts on this:

On structure:

  • Ideally the group would be interdisciplinary enough that they could cover everything from verifying that the Zcash guts of a tool are working correctly, to network layer privacy (e.g. use of Tor), to looking at the app’s basic security properties, to reviewing the UX. Do you have a sense of how easy it will be to source these skills?
  • What’s the relationship between this team and an outside auditor? You’d typically want folks advising in the design stage to be different from the folks auditing, right? Or how does that work?

On who: are there any ZF or ECC alums who would be good fits, in a “from scratch” team? What’s the set of firms that have provided auditing so far? I know Least Authority is one.

Also, I think we could have ecosystem wide bug bounties, no?

3 Likes
4

I am in full support of this idea. A concern struck me after I read this:

Finding people who are cross-disciplinary enough to be able to audit both protocol and application level projects seems like a huge ask. It may be good to have someone or some people in the community who are a part of the team in a very limited role, but are responsible for helping source the auditors on a per-project basis. The idea of having 1-2 people who do shifts over the course of a quarter-year time is good if they are incredibly cross-disciplinary, but that will be a difficult find.

5 Likes
5

I would recommend collaborating with OSTIF to help identify and negotiate with security auditors.

1 Like
6

When you say team you are mixing lots of different skillsets.

  • I suggested this in my application. It would be better to have a general purpose team with crypto ability, like NCC and use them to farm out the rest.
  • Webapp pentesting is not cryptographic maths assurance which is not network segregation or HSM/container management, which is not the same as app development.
  • all carry unique issues that need specialists (this is why OWASP was made, hardened PHP, etc.)
  • Yes, I know a number of different companies that I could reach out to over this.
  • I would like to see it organised, as in “this software needs to pass these requirements” - as outlined by the mgrc, then tested for by people with insurance and reputation in that area.

btw - your link. kinda 404’s but shawn explained removed image.

A great example of of this would be the bug you found. That should never have made it to security review. it should have been part of release test procedures. There is a lot of room for automation and manual conformation. but these test harnesses need to be built on an technology or device undertest. (that might translate bad. I mean. You build a test harness to test zcashd, or zecwallet, or if you are particularly brave and really good, libraries.)

2 Likes
8

The link is good, Sarah will generate a CCR, there are none currently open @mistfpga , only proposals.

9

Hi Sarah

– Just me, you can skip this part.
I’m Benjamin Winston, I work for the ECC as their Director of Security. I’ve been lucky enough to work on the test team for several great consultancies, and I’ve worked directly for a few vendors in the past, most recently at Fitbit looking at their full stack and firmware. The transition from external assessment to internal vendors really taught me a lot about being a security practitioner, including making my fair share of mistakes along the way.

I had been hired while CVE-2019-7167 was being worked on, but I wasn’t briefed on it until just prior to Sapling launch, at which point I handled the disclosure process. Many sleepless nights and stress on that one lol :slight_smile:

I handle the engagement of ECC with external security assessors, and I run our internal security function doing stuff like this Fuzzing Zcash with Kubernetes - Electric Coin Company
– end of me, please stop skipping here

I’m glad that we’ve got representation of security interests on the ZOMG and I agree that forcing spend into silos will reduce the effectiveness of the effort to check the security of the Zcash ecosystem by doing external assessments.

It also makes sense to me that there be a resource for security input into designs ahead of the “oops we didn’t mean to do that” and reengineering that inevitably follows someone engineering without first doing some security planning.

Certainly I don’t want to see basic TLS security or any other table-stakes control omitted from any product, which is why we started an outreach program with Aditya, unstoppable, nighthawk around their security. Our (I keep saying our - I mean mine and Taylor’s) intentions around that was to provide maximum reward for the ecosystem with a minimum invetment of effort on our part.

After you dropped that bug in Aditya’s code on Twitter with no warning or forward disclosure, we reached out to him and tried to provide a service. We created a key result around our performance in this area in response to that event to the effect of “no simple-to-discover issue gets by this program”. We defined “simple to discover” that a competent attacker would find it in the first few moments of analyzing the software.

I must confess I do wonder about your commitment to Ethics Reviews around projects harming the communities they’re intended to support. I wonder whether you consider dropping that bug to not be harming the community, or whether you think it only doesn’t detract from your ability to make that determination in other people’s efforts.

Thus far this program, as ah-hoc as it has been has seemed to provide at least some assurance to projects that might otherwise have experienced none. We had problems regulating our time on it though - it’s hard to dedicate exactly half a day per week to something while we’re operating in what is basically a high throughput interrupt-driven environment like ECC’s security.

One thing we’ve been able to organize while the Zcash project was just ECC (well, ZcashCo at the time) and the foundation is to adapt the norms of responsible disclosure to the wilds of the cyptocurrency space. I was lucky enough to be invited to a working group that ultimately produced this standard: GitHub - RD-Crypto-Spec/Responsible-Disclosure: A standard point from which crypto projects may derive their responsible disclosure policy. and we committed to it here: https://github.com/zcash/zcash/blob/master/responsible_disclosure.md

Briefly, the standard is intended to provide a commitment scheme to the destinations and conditions under which we will disclose security issues, so as to be able to provide continuous user safety in a distributed consensus-critical environment. The intent is to separate, basically, the transport negotiations from the communication itself so as to speed up our response times and reduce overhead when dealing with critical bugs. This complements ECC’s internal processes on handling security issues.

We just completed negotiations (yesterday) with Deirdre and Teor at the foundation to add them to our public commitment (expect a PR for that in zcash/zcash later today, I’m delighted to have them on board for this) and with Nighthawk and Unstoppable who have adopted our wallet code (expect a PR for that later today or early next week in those repos).

My hope is that if you come into contact with any bugs, critical or otherwise, in any of the code I’ve been employed to coordinate the security of that you’ll work with us on our polite request to please report them via our security disclosure comms point, which is documented on our website here: Security Information - Zcash and we will deal with them seriously.

I stand ready to provide whatever other input might be helpful to ZOMG in the effort to secure the ecosystem, although of course ultimately we don’t have too much spare capacity in terms of individual contributions, I’m here to help you in any other way I can.

Thanks

bambam

6 Likes
10

This point was rehashed numerous times in my candidacy. I’m not sure what there is to add, and I can only speculate as to why you feel you need to add your voice to this now instead of then, but my answer remains the same.

On that latter point, I am glad that my actions were the prompt for ECC to take ecosystem software risks somewhat seriously.

If you had actually read my post you would understand that my goal here is to establish funding for a team that can proactively address these challenges. Instead you have chosen to impugn my ethics and make this about some nebulous concept of “responsible” disclosure in an inherently hostile environment.

Thank you for your comments. There are clearly deep divisions that I thought had been somewhat resolved through the candidacy period and election results, but which I now see through recent conversations are deeper and systemic to core parts of this ecosystem. Given such constraints I’ll have to consider if my participation can be ultimately useful or effective.

3 Likes
11

<rant>

I’m officially annoyed that we (I mean everyone) have failed to move on from that event.

Bruised egos, sore ethics…yeah, I get it, but I really don’t care & it was a long time ago.

Please can we now move towards what we want to happen in future ?

(not aimed at anyone in particular)

</rant>

6 Likes
12

I think @zebambam owes @sarahjamielewis a public apology. Her involvement with ZOMG was one of the things that made me most optimistic about the future of Zcash attracting more open source developers that make meaningful contributions.

Losing @sarahjamielewis is a huge blow. @zooko please think carefully when hiring and retaining people like @zebambam that are part of the reason for a toxic culture at ECC and create toxicity in the community.

1 Like
13

Sarah wants to take on the surveillance state but can’t handle a little criticism? This has to be a joke.

14

I was extremely surprised to hear this weekend of Sarah’s resignation, and sad to see that my reply was a contributing factor. I’m not active on these Forums at all and clearly there was a wider situation that I stepped into that I was unaware of. I am taking the time to listen, hear and reflect on all the comments around what I posted so that I can do better in future.

12 Likes
15

Good to hear that you are taking some time to reflect on how your reply may have impacted her.

Being more active on the forum would be a positive. Public comments from someone in your position acknowledging the importance of open security issues will help rebuild confidence from the community in your stewardship. Absent that, some many will falsely assume that you either don’t take some issues seriously or do not find it important to keep the community updated. Zcash is a decentralized blockchain with many stakeholders. Please interact with the wider ecosystem as much or more than you interact with ECC.

5 Likes
16

I lik what this guy said.

17

I think it would be a tragedy on top of a tragedy if Sarah’s resignation distracted us from this initiative that she’s kicked off with this request for input.

I’m really interested in pushing this forward and would still appreciate feedback!

The positive feedback from @aiyadt here—i.e. that this would be helpful for Nighthawk—is a very good sign. I feel exactly the same way for Zbay.

It seems like once we start reaching out to these firms, they’ll have ideas about the best ways to structure it. Ideally (and I’m cribbing this from Sarah’s suggestions) they’d be involved in the grants process, in the initial design of new applications or features, and in review and testing—both of projects ZOMG has funded and really any widely used projects in the ecosystem.

My vision for this also includes the security team publishing regular reports on the state of ecosystem privacy and security (like our own version of Google’s Threat Analysis Group) and on progress towards goals.

I think this can help focus our values and brand around privacy, as secparam suggests here, and communicate to both users and investors that—while it will be a long road and a ton of work to get there—Zcash is on a certain trajectory to becoming the leading private cryptocurrency.

“Zcash Ecosystem Security Team” has a zesty ring to it :slight_smile:

4 Likes
18

It doesnt work like that. @zebambam @earthrise @alchemydc have all explained in previous posts why not. im too tired to look them up sorry.

My NCC option was the best imho. but you will spend all your funds on “security testing” when normal testing and release procedures would fix them problem. now how do you get skilled people to test complex hardware across different platforms? you make a test department! im nt sure if the ecc or zfnd have dedicated test outside security.

19

Sarah is not professional on the bug report how is that even a loss for community Infact it’s good she left with such unprofessional behaviour she doesn’t deserve Associated with zcash anyway…losing her is good that we found such vicious behavior early better than late.

20

(My personal opinions, not necessarily those of ECC, are:)

What are your opinions on the creation of such a team?

Yes please! There are so many great projects in the Zcash ecosystem from hardware wallets to message boards to Slack replacements. It’s really hard to try to keep up with them all, and they all have different levels of security expertise, so the community could really benefit from a dedicated and well-funded effort here.

Doing after-the-fact audits alone will likely lead to lots of “oops, this design is totally broken, we need to re-do a lot of work.” IMO, it’s important for there to be security people working closely with the teams to help build a “safety culture”—a sense of pride around delivering reliable code and an eagerness to notice when things are security- or privacy-critical and call out for extra scrutiny.

Early and continuous input is especially important for privacy, because it’s nearly impossible to “add” privacy later to a design that’s already leaking a bunch of information in a bunch of different ways.

Do you have thoughts on the structure of such a team?
Do you have ideas on who you would like to see involved in the team?
Is this something you would prefer to see handled by e.g. a dedicated/contracted industry partner or constructed from scratch?

I would like to see it staffed by full time security and privacy people. Because:

  • There are enough different projects now that it’s a full-time job just to keep up with them.
  • Audit firms (NCC et al.) operate on a timeboxed contract basis, so I don’t think they don’t really provide the kind of from-the-ground-up continual service that fledgling projects need to grow into high-assurance software shops.
  • Full-time allows them to develop personal relationships with the people working on the projects, hopefully earning them some easy buy-in when they want to make recommendations around development practices and feature selection.
  • They can become Zcash security/privacy specialists over time, building up a deep understanding of the protocol and all of the subtle Zcash-specific things that can go wrong, as opposed to using third-party firms who won’t be as well-versed in the tech and privacy goals. (I love Holmes’ idea of something like Google’s TAG, I’m thinking take it even to the next level and create a Project Zero for the Zcash ecosystem.)

So perhaps a couple people on the team who are really great security engineers and communicators, combined with a budget for standard post-hoc auditing according to the team’s prioritization?

(By the way I’ve heard of https://latacora.com/ which does something like I’m imagining, but I have no experience of actually using them. It might be worth reaching out to them for an opinion on how best to implement this.)

7 Likes
21

This makes a lot of sense to me.

Is this similar to what @earthrise is saying about creating process and culture inside the team itself, and guiding design decisions?

Like, you’re saying to leverage the work of skilled people within the teams as much as possible and not wastefully consume the security consultants’ time, is that it?

I’ll reach out to them. Who else should we reach out to, besides NCC and Krebs as mentioned above? (Assuming they’d be interested in a more long-term retainer-type engagement than something time-boxed.)

1 Like