Bootstrapped and deterministic builds a la StageX

Hey everyone, I’m Anton, a security engineer and technologist who loves improving the security, privacy and freedom of the world (antonlivaja.com). I would like to submit a proposal for support around work I have been doing and will continue to do. I’m looking forward to your questions, concerns and comments, please don’t be shy. If this goes well, we have other ideas how we can help improve the zcash ecosystem through verifiable compute (for which reproducible builds are a requirement), some operational security level-ups and more!

Terms and Conditions

  • I agree to the Grant Agreement terms if funded
  • I agree to Provide KYC information if funded above $50,000 USD
  • I agree to disclose conflicts of interest
  • I agree to adhere to the Code of Conduct and Communication Guidelines
  • I understand all milestone deliverables will be validated and accepted by their intended users or their representatives, who will confirm that the deliverables meet the required quality, functionality, and usability for each user story.
  • I agree that for any new open-source software, I will create a CONTRIBUTING.md file that reflects the high standards of Zcash development, using the librustzcash style guides as a primary reference.
  • I understand when contributing to existing Zcash code, I am required to adhere to the project specific contribution guidelines, paying close attention to any merge, branch, pull request, and commit guidelines as exemplified in the librustzcash repository.
  • I agree to post request details on the Community Forum
  • I understand it is my responsibility to post a link to this issue on the Zcash Community Forums after this application has been submitted so the community can give input. I understand this is required in order for ZCG to discuss and vote on this grant application.

Application Owners

@antonleviathan

Organization Name

How did you learn about Zcash Community Grants

Working with ecosystem organizations

Requested Grant Amount (USD)

48000

Category

Non-Wallet Applications

Project Lead

Anton Livaja - Co-Founder Distrust.co, Security Engineer - https://antonlivaja.com/

https://mastodon.online/@anton
https://snort.social/nprofile1qqstl9pmw9jlefsk5jpudhrszer2995f4dn3zy8umkap9ga939xd59gkrdtdy
https://x.com/antonlivaja

Additional Team Members

Lance Vick - Co-Founder Distrust.co, Security Engineer - https://lance.dev/
Ryan Heywood - Distrust.co, Security Engineer - https://ryansquared.pub/

Project Summary

We’re working to establish bootstrapped and deterministic build environments of the core software components of the Zcash ecosystem as best in class: better than what exists currently in both Bitcoin and in Monero.

Leveraging our FOSS distribution StageX protects against supply chain attacks and powerful threat actors by methodically eliminating surface area for attack, making entire classes of attacks impossible.

Project Description

Distrust.co produces open source software tools for verifiable security.

Our manifesto includes the goals: decentralize trust, open source everything, create security beyond compliance, move thoughtfully & improve things, and create impact over profit.

Distrust collectively and individually have worked with BitGo, Ether.fi, Optimism Labs, Exodus Wallet, Fitbit, Unit410, OpenZeppelin, Ledn, and resolved issues for many Fortune 500 companies. We discovered and disclosed the ‘Milk Sad’ Libbitcoin Explorer CVE(milksad.info) in 2023, presented at the 38th Chaos Communication Congress which involved flawed entropy generation of private keys. Past publications also include CVEs in GPG, a vital open-source encryption tool that is the free alternative to Phil Zimmermann’s PGP, and in the Terraform AWS Provider, the most widely used provider for Amazon Web Services.

We are committed to using our skill sets and experience to bring better security to technology ecosystems that benefit and protect people worldwide.

Monero and Bitcoin both currently use Guix for reproducibility. Guix is good, but StageX takes things to a new level, with an even stricter approach.*

StageX is a security-first Linux distribution for reducing supply chain risks: minimal, bootstrapped, hermetic, deterministic, reproducible, and multi-party signed. It’s being used by Talos Linux, BitGo, on projects we’re involved with such as Turnkey.io and Mysten Labs, and we’re in other exciting but still confidential discussions right now as well. Check it out!

The Zcash ecosystem is, with this initiative already underway, now beginning to integrate deterministic/reproducible and bootstrapped builds.

*Guix has 3 significant disadvantages:

  • Use of a package management system that adds surface area for attacks, and introduce complexity by requiring Scheme programming language.
  • Allows non-deterministic and non-full source bootstrapped code in tree.
  • Allows a single maintainer to push changes into a release without another maintainer signing off.

Proposed Problem

Not using bootstrapped and reproducible builds in a critical public software project creates significant security and operational risks.

Without reproducibility, it is impossible to verify that the binary distributed to users was built exactly from the claimed source code, leaving the project vulnerable to undetected supply chain attacks.

A dramatic and famous example of this risk is the SolarWinds Orion attack, where attackers subverted the build environment. A verified reproducible build system could have detected or prevented this attack that modified build inputs that were not checked into version control.

A more recent example of an attack also could have been made impossible by use of StageX: the 2024 XZ Utils Backdoor, CVE-2024-3094. A backdoor was embedded in pre-generated build files, and this malicious addition was detected just in the nick of time! Before a silent, exploitable weakness was deployed in sshd across the entire internet, it was discovered by an observant developer, rather than being obvious as would have been the case with a verified build system.

Another prominent example was the Codecov breach in 2021, which ran the attacker’s malicious code in developer’s CI pipeline. It was ultimately discovered manually, months after the initial breach, because Codecov provides a signature for its proprietary code. If the attackers had changed the published code signature as well, this exploit could have continued, unexposed, for an indefinite period of time.

Finally, an older but frightening (and instructive!) malware called XcodeGhost infected about 3,500 apps at once using a type of dynamic library hijacking.

Proposed Solution

Reducing supply chain risks by introducing full source (both compiler and language) bootstrapped, deterministic builds for critical Zcash software via StageX. This strategy can be applied both to build and runtime.

This addresses the ‘Trusting Trust problem’ and similar issues stemming from the attacker manipulating the build environment.

Solution Format

Verified reproducible builds act as a cryptographic audit trail between source and binary. They are effective against attacks that modify build inputs that are not in version control (like XZ), tamper with the build process (like SolarWinds), or distribute malicious code through compromised release or testing pipelines(like the Codecov breach and XcodeGhost).

The Linux Foundation, source of the Open Container Initiative (OCI), recognizes verified reproducible builds as a long-term solution to such attacks.

Dependencies

Teams and individuals interested in utilizing our build toolchain will be engaged in an ongoing way, with discussions and PRs.

Technical Approach

Use containers (docker) backed by maintained StageX images, bootstrapped via ‘stages’ from the compiler, along with utility (bash) shell scripts.

StageX is

  • Container Native.
    Using layers of the OCI packaging standard instead a typical package manager.
  • Full Source Bootstrapped.
    Remarkably, not inheriting ANY trust from third party binaries, < 190 byte x86 assembly seed reproduced on multiple distros, and more…
  • Reproducible.
    Accountable, protecting against any compromised or malicious maintainer.
  • Multi-Signed.
    Every artifact is signed by multiple maintainers, using offline, hardware backed PGP keys.

This 2024 InCyber conference talk covers topics including bootstrapping compilers, deterministic builds, and StageX.

See also our threat model document.

Upstream Merge Opportunities

Full source bootstrapped and deterministic build by antonleviathan · Pull Request #301 · zcash/wallet · GitHub [merged]
See the README here.
use SOURCE_DATE_EPOCH for build date if present by antonleviathan · Pull Request #622 · zingolabs/zaino · GitHub [merged]
make zainod reproducible and use stagex build toolchain by antonleviathan · Pull Request #641 · zingolabs/zaino · GitHub [in review]
use stagex for full source bootstrapped and deterministic build by antonleviathan · Pull Request #10068 · ZcashFoundation/zebra · GitHub [in review]

We have our eyes on other software from Zingo, ZF’s FROST tools, Coinholder Voting systems (which could also involve verified compute!) and Crosslink (though we do not see much public at this moment) among others. We are open to feedback on which projects to select, which will have the broadest impact or the greatest need for our work.

Hardware/Software Costs (USD)

0

Hardware/Software Justification

n/a

Service Costs (USD)

0

Service Costs Justification

n/a

Compensation Costs (USD)

48000

Compensation Costs Justification

48000 / 3 Security Engineers = 128 hours at $125/hr

Total Budget (USD)

48000

Previous Funding

No

Previous Funding Details

n/a

Other Funding Sources

No

Other Funding Sources Details

None for Zcash ecosystem specific work.

Implementation Risks

Some of the up-front work is already done, as PRs have opened into the Z3 stack and conversations with those teams (ECC, ZF, ZingoLabs) are currently ongoing where needed. There is presently work on Zebra around updating their release and build scripts.

We would like to continue to roll out PRs and communicate more in the coming months to ensure that all interested parties are engaged and supported thoroughly while offering more ecosystem projects coverage. This grant would offer us support to do just this.

It’s possible that the work outlined in Milestone 2, directed in whole or in part by ZCG or the community, will number either too many projects or too few, as we have not resolved what projects to engage next.

Also, StageX cannot currently build on ARM, it only supports x86-64. This limits what projects are possible to integrate at this time. However, we just did a big refactor to add LLVM so we will support ARM in the near future, as there is interest from parties, including the Apache Software Foundation.

Overall, we are proud of our work, and will stand by it through open and respectful discourse and continued delivery wherever applicable.

Potential Side Effects

While accomplishing greater security for the software pipeline, integrating new tooling to projects can create more cognitive overhead and integration work for developers even when designed to be a drop-in replacement for existing solutions like Alpine, or Debian.

We pledge to support these considerations and transitions where applicable, by being available and communicative with the stakeholder teams that may accept this tooling into their stacks.

Success Metrics

Checked in code, allowing creation of StageX pipeline build tooling for core ecosystem tools with organizations and individuals.

Communication of theory and application of StageX, and closely related tooling.

Retainer: 20 hours total support including technical guidance, and broadly project-related Q & A for a period of 12 months.

Startup Funding (USD)

0

Startup Funding Justification

n/a - much of the work has already been done.

Milestone Details

- Milestone: 1
  Amount (USD): 25000
  Completed! as of 2025-11-09
  Deliverables:
    - PRs listed as complete above are in.
    - Consultations and review feedback cycles underway or completed.
  Acceptance Criteria: See what has been done already, poll community members about our team, and also of course ZCG agrees it is worth this amount. Reviews and chats are active, and we can engage also on the forum.

- Milestone: 2
  Amount (USD): 18000
  Expected Completion Date: 2025-12-15
  Deliverables:
    - PRs
    - Consultations and review feedback cycles.
  Acceptance Criteria: Not less than 3, and not more than 6 PRs will be opened in other ecosystem repos, with reviews and chats continuing.

- Milestone: 3
  Amount (USD): 5000
  Expected Completion Date: 2026-11-01
  User Stories:
    - "Distrust helped us with integration and was responsive when something came up! They are happy to talk about both theory and the practical side of the project and how it relates to the zcash ecosystem."
  Deliverables:
    - All outstanding inquiries and issues addressed.
  Acceptance Criteria: No outstanding issues. Ask community members about their experiences in communicating with the Distrust team, and their resolution of issues.

Supporting Documents

https://codeberg.org/stagex/stagex
https://stagex.tools/
5 Likes

I reviewed this PR mentioned above for ZingoLabs, where we’re discussing this and other related topics having to do with leveling up our attestation, and integrity in pipelines…

While I did not dig into the makeup of StageX directly, and so I cannot testify to the core of this project’s worth, I think this kind of direction is very exciting and a great step in the right direction. I noticed @str4d engaging with the PR that went into the ECC repo mentioned above, which gives me a good amount of confidence, as well as the bona-fides listed above (how awesome is that milk sad finding?? in addition to finding a flaw in GPG)…

And even though the Zcash Foundation with Zebra are moving more cautiously, I also take that as a good sign, in terms of having a diversity of views and also questioning major overhauls of mission critical projects.

And I’ve been impressed with both the public and private interactions I’ve had so far with @antonleviathan, who is clearly passionate about their work and is keen to discuss the tech and register with cool projects in our zone.

I strongly support attracting and maintaining relationships with engineers with such special skill sets and experience… Zcash should only benefit from such strong affiliations!

I support this initiative.

1 Like