Bootstrapped and deterministic builds a la StageX

+222 -11

This PR switches `zallet` to use StageX as the build toolchain distribution. …The primary benefits of this are that the StageX is: * Full source bootstrapped, all the way down to the compiler, including rust * The whole distro is fully deterministic. * Reproducing the whole tree is as simple as setting up `containerd` and running `make` in the [StageX](https://codeberg.org/stagex/stagex) repo - may want to use a beefy machine for the build, as it takes a while * All changes in StageX tree are always reviewed and signed by at least two maintainers * All releases are reproduced by at least two maintainers on at least two different chipsets, and signed The followup work would include updating the debian build and publish [workflow](https://github.com/zcash/wallet/blob/main/.github/workflows/deb-release.yml) and push to dockerhub [workflow](https://github.com/zcash/wallet/blob/main/.github/workflows/build-and-push-docker-hub.yaml). Closes #313.

zcash-devtool : Open. CLI app for developers prototyping Zcash functionality. Requires patched rage dependency. See associated forge items below.

ZingoLabs
zaino: Open. An indexer for the Zcash blockchain implemented in Rust.

github.com/zingolabs/zaino

make zainod reproducible and use stagex build toolchain (#641)

dev ← antonleviathan:bootstrapped_deterministic_build

opened 08:45PM - 02 Nov 25 UTC

+216 -0

## Motivation I introduced the use of the [StageX](https://codeberg.org/stage…x/stagex) build toolchain, and made the `zainod` build deterministic. It's worth nothing that runtime inherits a secure config from here: https://codeberg.org/stagex/stagex/src/commit/4a9058f6cc16dc81f762543d768176762b600dd7/packages/core/filesystem/Containerfile ### Tests I built the image twice using `utils/build.sh` and compared the resulting hashes ### Follow-up Work We need to figure out how to update the `Makefile.toml` according to team preferences to use the updated image for creating `zainod` images and for producing binaries as needed. I'm also happy to help write documentation as desired. ### PR Checklist - [x] The PR name is suitable for the release notes. - [x] The solution is tested. - [ ] The documentation is up to date.

zingolib : Open. This library also produces the wallet zingo-cli as a binary.

Zcash Foundation
zebra : Placed into Draft from Open by project maintainers. Zebra is the Rust full node implementation of the Zcash protocol.

github.com/ZcashFoundation/zebra

feat: introduce deterministic builds using stagex (#10491)

main ← antonleviathan:stagex-deterministic-bootstrapped-build

opened 08:46PM - 16 Apr 26 UTC

+399 -0

## Motivation Supply chain security for cryptocurrency node software is criti…cal. This PR follows up on #10068, which attempted to bring Zebra's build security in line with Bitcoin Core's approach ([bitcoin/bitcoin#15277](https://github.com/bitcoin/bitcoin/pull/15277)), adopted by them in 2018. [StageX](https://codeberg.org/stagex/whitepapers) is a bootstrapped, fully deterministic build distribution that goes further than Guix in supply chain security while adopting modern tooling. LLVM as the primary compiler, mold as the linker, and mimalloc as the allocator, for improved performance. This PR is related to the grant: https://forum.zcashcommunity.com/t/bootstrapped-and-deterministic-builds-a-la-stagex/53040 ## Solution This PR introduces a `Dockerfile.deterministic` alongside the existing `Dockerfile`, so maintainers can evaluate and test the deterministic build path without disrupting the current setup. Since #10068, StageX has matured to support LLVM as the default compiler, mimalloc as the default memory allocator, and mold as the primary linker, for dramatic speed improvements and cross-compiling support across the board, without compromising on determinism ## Follow-up Work - Determine target architectures to support and add cross-compilation stages - CI integration for verifying build reproducibility ## References - Previous PR: #10068 - Bitcoin Core precedent: [bitcoin/bitcoin#15277](https://github.com/bitcoin/bitcoin/pull/15277) - StageX whitepaper draft: https://codeberg.org/stagex/whitepapers ## PR Checklist - [ ] The PR name is suitable for the release notes. - [ ] The PR follows the [contribution guidelines](https://github.com/ZcashFoundation/zebra/blob/main/CONTRIBUTING.md). - [ ] The library crate changelogs are up to date. - [ ] The solution is tested. - [ ] The documentation is up to date.

frost-tools : Open. a set of command line demos that uses the ZF FROST libraries and reference implementation. Meant to be used in tandem by multiple collaborators to establish cryptographically secured group trust relationships. Reproducible builds will ensure that each participant has the same set of tools, and those tools correspond directly to the source code.

github.com/ZcashFoundation/frost-tools

Frost-tools should have reproducible builds.

opened 08:22PM - 17 Apr 26 UTC

[From reproducible-builds.org](https://reproducible-builds.org/) > Reproducible… builds are a set of software development practices that create an independently-verifiable path from source to binary code. > Reproducible Builds ensure that the software you trust is both safe and verifiable.. > Reproducible Builds provide certainty that software is genuine and has not been tampered with. [From Kernel.org](https://docs.kernel.org/kbuild/reproducible-builds.html): > It is generally desirable that building the same source code with the same set of tools is reproducible, i.e. the output is always exactly the same. Particularly with tools meant to be used in tandem across distributed collaborators, reproducible builds will ensure that each participant has the same set of tools, and those tools correspond directly to the source code. More on StageX: https://stagex.tools/ https://codeberg.org/stagex/stagex

Associated forge items:
zcash-devtool :

github.com/zcash/zcash-devtool

allow stdin input for wallet init (#156)

main ← ala-mode:stdin_wallet_init

opened 05:37PM - 09 Mar 26 UTC

+4 -1

Fixes #154 Aiming to create some basic container automation (wallet creation… without attaching a TTY) I noticed this change could be made easily within the same crate already being used. The `.lock()` [method aims to provide exclusive access to the standard input stream](https://doc.rust-lang.org/std/io/struct.Stdin.html#method.lock) by returning a locked handle. I tested this small change locally and it seemed to work as I expected.

github.com/zcash/zcash-devtool

Network required for wallet init

opened 05:58PM - 09 Mar 26 UTC

Running the same invocation `./zcash-devtool wallet init --name "generic_name" …--identity ./age_id.txt --connection direct --network test -s zecrocks` works with a network connection, but fails without it: ``` INFO zcash_devtool::remote: Connecting to testnet.zec.rocks:443 Error: transport error Caused by: 0: dns error 1: dns error 2: failed to lookup address information: Name or service not known ``` Omitting the `-s zecrocks` defaults to `lightwalletd.testnet.electriccoin.co:9067` (which also has an [identified issue](https://github.com/zcash/zcash-devtool/issues/148)). I think, minimally, wallet initialization should be allowed without a network connection for security (and potentially privacy) issues. I have not investigated other aspects of `zcash-devtool` to see what other systems require the network, but wallet initialization seems clear to me there should be an option to do without. Perhaps consider adding a `--connection none` option to explicitly disallow network access?

github.com/zcash/zcash-devtool

Cargo Update fails due to orchard dependency

opened 05:55AM - 17 Apr 26 UTC

running `cargo update`: ``` failed to select a version for the requirement `core…2 = "^0.3"` version 0.3.0 is yanked version 0.3.1 is yanked version 0.3.2 is yanked version 0.3.3 is yanked location searched: crates.io index required by package `orchard v0.12.0 (https://github.com/zcash/orchard.git?rev=6b12c77260aa7fac0d804983fc31b71b584d48e0#6b12c772)` ... which satisfies dependency `orchard = "^0.12"` of package `zcash-devtool v0.1.0 (/home/o/stagex/zcash-devtool)` ``` `core2` [has been yanked from crates.io,](https://crates.io/crates/core2) and [is deprecated](https://github.com/bbqsrc/core2/blob/main/README.md) : "No longer supported. Use `core` directly." See [the opened issue in the orchard repo](https://github.com/zcash/orchard/issues/481).

rage:

github.com/str4d/rage

age: Replace HashSet with BTreeSet for deterministic iteration (#612)

main ← antonleviathan:deterministic-iteration

opened 05:59PM - 17 Apr 26 UTC

+27 -27

This change is to remove a source of nondeterminism in age's label handling. …In downstream reproducible-build testing, we were seeing otherwise-identical builds produce different `zcash-devtool` binaries. `diffoscope` narrowed that down to code generation differences in `age::error::PluginError::fmt`, which pointed back to non-deterministic iteration order from `HashSet`-backed label metadata. Switching these paths to `BTreeSet` makes the iteration order deterministic. I tested this by pointing `zcash-devtool` at a local checkout of this branch and rebuilding with the patched age crate.

orchard:

zingolib:

Build Script requires network access

opened 07:17AM - 15 Feb 26 UTC

While working on [StageX Reproducible Build](https://github.com/zingolabs/zingol…ib/pull/2113) I noticed the current build script for `zingolib` [requires network access in at least one place](https://github.com/zingolabs/zingolib/blob/5a5393c205c9cad867e0f672d149060f2132556c/zingolib/build.rs#L39). This logic looks like it went in about 9 months ago and was last adjusted about 4 months ago. There is [one other place it looks likely that a network call is needed too](https://github.com/zingolabs/zingolib/blob/5a5393c205c9cad867e0f672d149060f2132556c/zingolib/build.rs#L6), but I did not observe this working, directly. This logic makes it impossible to control the hermeticity of the build. That is, it defies the structure which is used to make the build reproducible: this is a blocking problem for this PR. More abstractly, this pattern implicates a number of issues: -network calls during builds introduce flakiness, additional security risks, possibly add latency. -it can break or make impossible offline or cached builds. -possible non-determinism. -it runs counter to rust's compile-time guarantees (could possibly bring in unverified external code, bypassing Rust’s safety checks). -adds debugging complexity. -create 'hidden' dependencies that are not tracked by `cargo` or `git` directly. I am not sure of the pattern to fix this issue right now, but I am fairly certain this is a bad pattern beyond it blocking the PR I am looking at. Here's a few rough thoughts: -check for params, and download params at (first) runtime, not build time. -provide a utility script or cli subcommand to download params. -if needed for CI run as pipeline script

Panic on connection to server

opened 07:04AM - 02 Mar 26 UTC

bug

Panic on connection to server, noted during wallet creation across multiple serv…ers, including the default `zec.rocks` on `dev`: ``` Creating a new wallet Error during startup: Failed to create lightclient. Error getting client: Transport(tonic::transport::Error(Transport, ConnectError(Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssuer) }))) thread 'main' panicked at zingo-cli/src/lib.rs:517:13: explicit panic ``` I can connect to this server with `wget` over `https` on the same `443` port. Also, I confirmed via `openssl s_client -connect zec.rocks:443 -servername zec.rocks` ``` ... Server certificate -----BEGIN CERTIFICATE----- MIIDhjCCAwygAwIBAgISBdshwVszK16YwmbjFlXuGuRWMAoGCCqGSM49BAMDMDIx CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF ODAeFw0yNjAxMTgyMzI2MzRaFw0yNjA0MTgyMzI2MzNaMBQxEjAQBgNVBAMTCXpl Yy5yb2NrczBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABMOyXUn3FfWdQSQJJ4dM jMDOOi7SJbIf/L/NbAD/n7xwwntmQ14iXQDQplj5kFXbQr2M2coU/TSkvQfWU65m AOSjggIeMIICGjAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEG CCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFKxgl8DSoDPlEXvOL5YF yLcq7DUbMB8GA1UdIwQYMBaAFI8NE6L2Ln7RUGwzGDhdWY4jcpHKMDIGCCsGAQUF BwEBBCYwJDAiBggrBgEFBQcwAoYWaHR0cDovL2U4LmkubGVuY3Iub3JnLzAUBgNV HREEDTALggl6ZWMucm9ja3MwEwYDVR0gBAwwCjAIBgZngQwBAgEwLQYDVR0fBCYw JDAioCCgHoYcaHR0cDovL2U4LmMubGVuY3Iub3JnLzg3LmNybDCCAQsGCisGAQQB 1nkCBAIEgfwEgfkA9wB+ABqLnWlKV5jImaDKiL30j8C0VmDMw2ANH3H0af/H0ayj AAABm9OkFewACAAABQA5WkouBAMARzBFAiBHC8T7vkptVn5Plt4YxGhE9bTZJCLy 7f9FSoGaslyJawIhAK1E3sioiX1RVA/vvPMAKIWV6kN4vyhkRiDYtQUb2SMTAHUA ZBHEbKQS7KeJHKICLgC8q08oB9QeNSer6v7VA8l9zfAAAAGb06QbvQAABAMARjBE AiBXi+YMEM6ntoJolCT+kbSBaR8DVrWfzH3Go9I/LxNdOgIgPmbHIUtr6dEx5aSK X6abKnWY8B5TshErqFaQ00rrlH0wCgYIKoZIzj0EAwMDaAAwZQIwNZjjJizguNOB s6Jux7pirTl21b3elFM9L6Cs1L5dUVyVqRb8X+ImeaIB8L60EX7DAjEAm/gQtmZ2 o9c6qCSjFOLlisP2RaOkew+/lszTfzPIUgJLJxf6m9uDLGRrpO0LahVR -----END CERTIFICATE----- subject=CN = zec.rocks issuer=C = US, O = Let's Encrypt, CN = E8 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 2334 bytes and written 391 bytes Verification: OK ```

`cargo clean && cargo update` fails on `dev`

opened 08:15PM - 28 Mar 26 UTC

While working to refine and update #2113 I encountered: ``` ~$ cargo clean && ca…rgo update Removed 0 files Updating git repository `https://github.com/time-rs/time` Updating crates.io index Updating git repository `https://github.com/zingolabs/infrastructure.git` Updating git repository `https://github.com/zingolabs/grpc_proxy.git` Updating git repository `https://github.com/zingolabs/zaino.git` error: failed to select a version for the requirement `zingo-netutils = "^3.0.0"` version 3.0.0 is yanked location searched: crates.io index required by package `darkside-tests v0.1.0 (/home/o/stagex/zingolib/darkside-tests)` ``` This may be related to https://github.com/zingolabs/zingolib/issues/2114

Build script writes to a custom folder in the user's home directory

opened 09:45PM - 29 Mar 26 UTC

`rm -rf ~/.zcash-params && cargo clean && cargo check && cargo build -v --releas…e --frozen && ls -lat ~/.zcash-params` will confirm the files and directory are newly created. This violates the principle of least surprise (and makes reproducible builds more difficult by expanding the filesystem areas effected). It introduces git-untracked side effects, breaking isolation expected in containerized or CI environments. Generally, Cargo itself only writes to `~/.cargo` (for registry/git data) and `target/` (for build artifacts). https://doc.rust-lang.org/cargo/reference/build-script-examples.html gives the example of using the `OUT_DIR` environment variable. under https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts under `OUT_DIR`: "This folder is inside the build directory for the package being built, and it is unique for the package in question."

Build script writes (copies) Sapling parameters to crate source directory

opened 04:16PM - 30 Mar 26 UTC