"ZIP Proposal: Deterministic Orchard Wallet Scanning Standard"

Technology
1

Hello Zcash community,

I’m proposing a new ZIP for Deterministic Orchard Wallet Scanning Standard. This ZIP addresses a critical reliability issue that affects all Zcash wallet users: inconsistent scanning results across wallet implementations and restores.

ZIP: Deterministic Orchard Wallet Scanning Standard

Introduction

This ZIP specifies a standard algorithm for deterministic scanning of Orchard notes in Zcash wallets. The specification ensures that wallet implementations produce identical results when scanning the same block range, regardless of implementation details, scan order, or timing. This determinism is critical for wallet reliability, user trust, and interoperability between wallet implementations.

Motivation

Problem Statement:

Current Zcash wallet implementations lack a standardized approach to scanning Orchard notes. Recent discussion about issue with Zingo wallet is a good explain of this leads to several critical issues:

  1. Non-Deterministic Results: Different wallets, or even the same wallet on different runs, may produce different results when scanning the same block range. This creates user confusion and loss of trust.

  2. Inconsistent Wallet State: When a user restores a wallet from a mnemonic, the resulting balance and transaction history may differ from the original wallet state, even when scanning the same blocks.

  3. Interoperability Issues: Users cannot reliably switch between wallet implementations because each may produce different results for the same wallet.

  4. Testing and Verification Challenges: Without a deterministic standard, it is difficult to verify wallet correctness or compare implementations.

  5. User Trust: Inconsistent results erode user confidence in wallet software and the Zcash protocol itself.

Solution:

This ZIP specifies a deterministic scanning algorithm that ensures:

  • Identical results across wallet restores
  • Consistent state regardless of scan order
  • Reliable transaction history
  • Trustworthy balance calculations
  • Interoperability between wallet implementations

Why This Matters:

Deterministic scanning is a core Zcash invariant for wallet reliability. Without it, wallets cannot be trusted to accurately represent user funds. This ZIP establishes a standard that all wallet implementations should follow to ensure consistency and reliability. If this not a major issue great I’m proposing this zip to ensure user getting the best user wallet from Zcash.

Specification

Terminology

  • Block Height: The sequential number of a block in the blockchain (starting from 0 for genesis block)
  • Orchard Action: A spend or output action within an Orchard transaction
  • Note Commitment: The commitment to an Orchard note (cmx)
  • Nullifier: The nullifier of a spent Orchard note
  • Incoming Viewing Key (IVK): The key used to decrypt incoming notes
  • Scope: Either External or Internal scope for Orchard addresses
  • Deterministic: Producing identical results given identical inputs, regardless of implementation or execution order

Scanning Algorithm

Overview

The deterministic scanning algorithm processes blocks in strict sequential order by height, processes transactions within each block in deterministic order, and processes Orchard actions within each transaction in deterministic order.

Step 1: Block Range Determination

  1. Start Height: The scan must begin at a specific block height. This may be:

    • The Orchard activation height (block 1,687,104 for mainnet)
    • A user-specified start height
    • The last scanned height + 1 (for incremental scans)

  2. End Height: The scan must end at a specific block height. This may be:

    • The current chain tip
    • A user-specified end height
    • A height limit for safety (e.g., start_height + 1000)

  3. Block Range: The range must be [start_height, end_height] inclusive, processed in ascending order.

Step 2: Block Processing Order

Blocks MUST be processed in strict ascending order by height:

for height in start_height..=end_height:
    process_block(height)

Rationale: Processing blocks in height order ensures deterministic results regardless of network conditions or implementation details.

Step 3: Transaction Processing Order

Within each block, transactions must be processed in deterministic order:

  1. Primary Order: By transaction index within the block (0, 1, 2, …)
  2. Secondary Order (if needed): By transaction ID (txid) in lexicographic order

Rationale: Transaction index is the most reliable deterministic ordering within a block.

Step 4: Orchard Action Processing Order

Within each transaction, Orchard actions must be processed in deterministic order:

  1. Primary Order: By action index within the transaction (0, 1, 2, …)
  2. Scope Order: Process External scope actions before Internal scope actions (if both exist)

Rationale: Action index provides deterministic ordering within a transaction.

Step 5: Note Decryption

For each Orchard action:

  1. Extract Action Data:

    • Nullifier (if spend action)
    • Note commitment (cmx)
    • Ephemeral key
    • Encrypted note ciphertext

  2. Attempt Decryption:

    • Try decryption with External scope IVK
    • Try decryption with Internal scope IVK
    • Use the first successful decryption

  3. Note Validation:

    • Verify note commitment matches
    • Verify nullifier (if spend action)
    • Extract note value and memo

  4. Note Storage:

    • Store note with block height, transaction index, action index
    • Track nullifier to detect spent notes
    • Deduplicate by nullifier (if note appears multiple times)

Step 6: Note Deduplication

After processing all blocks:

  1. Deduplicate by Nullifier:

    • If multiple notes share the same nullifier, keep only the first occurrence (by block height, then transaction index, then action index)

  2. Spent Note Detection:

    • Mark notes as spent if their nullifier appears in a later block’s spend actions
    • Track nullifiers across the entire scan range

Step 7: Result Compilation

Compile the final result:

  1. Unique Notes: All deduplicated notes
  2. Total Balance: Sum of unspent note values
  3. Spent Count: Number of notes marked as spent
  4. Unspent Count: Number of notes not marked as spent

Deterministic Guarantees

The algorithm guarantees determinism through:

  1. Fixed Block Order: Blocks processed in ascending height order
  2. Fixed Transaction Order: Transactions processed by index within block
  3. Fixed Action Order: Actions processed by index within transaction
  4. Fixed Scope Order: External scope processed before Internal scope
  5. Fixed Deduplication: First occurrence kept when duplicates found

Verification:

Two implementations following this specification must produce identical results when:

  • Scanning the same block range
  • Using the same viewing keys
  • Processing the same blockchain state

Test Vectors

Test Vector 1: Simple Scan

Input:

  • Start Height: 3,146,400 (NU 6.1 activation)
  • End Height: 3,146,500
  • Viewing Keys: [Test keys for known addresses]

Expected Output:

  • Notes Found: [Specific note commitments]
  • Total Balance: [Sum in ZAT]
  • Spent Count: [Number]
  • Unspent Count: [Number]

Verification:

  • Multiple implementations must produce identical results
  • Results must be reproducible across runs

Test Vector 2: Incremental Scan

Input:

  • First Scan: Blocks 3,146,400 to 3,146,450
  • Second Scan: Blocks 3,146,451 to 3,146,500
  • Combined Scan: Blocks 3,146,400 to 3,146,500

Expected Output:

  • Results from combined scan must equal results from incremental scans
  • Note counts and balances must match exactly

Test Vector 3: Wallet Restore

Input:

  • Original Wallet: Scan blocks 3,146,400 to 3,146,500
  • Restored Wallet: Scan same blocks with same mnemonic

Expected Output:

  • Restored wallet MUST produce identical results to original
  • All notes, balances, and spent status must match

Reference Implementation

NozyWallet Implementation:

A reference implementation is available in the NozyWallet codebase:

Implementation Notes:

  1. Blocks are processed sequentially from start_height to end_height
  2. Transactions within each block are processed in order
  3. Both External and Internal scope IVKs are tried for each action
  4. Notes are deduplicated by nullifier after all blocks are processed
  5. Spent status is determined by tracking nullifiers across the scan range

Security Considerations

  1. Privacy: The scanning algorithm does not leak information about which notes belong to the wallet. Decryption attempts are made for all actions, but only successful decryptions reveal ownership.

  2. Performance: Sequential block processing may be slower than parallel processing, but determinism requires sequential order. Implementations may optimize by caching block data.

  3. Network Reliability: The algorithm assumes reliable access to blockchain data. Implementations should handle network errors gracefully and resume scanning from the last successful block.

  4. Memory Usage: For large block ranges, implementations should process blocks incrementally rather than loading all blocks into memory.

  5. Nullifier Tracking: Tracking nullifiers across the entire scan range is necessary for accurate spent status, but may require significant memory for large ranges.

Backward Compatibility

This ZIP does not require any changes to existing ZIPs.

Impact on Other ZIPs

No modifications required:

  • ZIP 32 (HD Wallets): No changes needed - This ZIP uses ZIP 32 keys but doesn’t modify ZIP 32
  • ZIP 224 (Orchard): No changes needed - This ZIP uses Orchard protocol but doesn’t modify it
  • ZIP 316 (Unified Addresses): No changes needed - This ZIP uses unified addresses but doesn’t modify ZIP 316
  • ZIP 315 (Best Practices): No changes required - This ZIP is complementary, not conflicting

Optional coordination (not required):

  • ZIP 315: May optionally reference this ZIP in future revisions, but not required
  • Future wallet ZIPs: May reference this ZIP for scanning guidance, but not required

Why No Changes Are Needed

  1. Standards Track, Not Consensus Track:

    • This is a wallet implementation standard
    • Does not change protocol rules
    • Does not require network upgrade
    • Does not affect consensus

  2. Additive Standard:

    • Adds a new standard for scanning
    • Does not remove or modify existing standards
    • Wallets can adopt it without breaking existing functionality

  3. Uses Existing ZIPs, Doesn’t Modify Them:

    • Uses ZIP 32 keys (doesn’t change ZIP 32)
    • Uses ZIP 224 Orchard protocol (doesn’t change ZIP 224)
    • Uses ZIP 316 addresses (doesn’t change ZIP 316)

  4. Backward Compatible:

    • Existing wallets continue to work
    • New wallets can adopt the standard
    • No breaking changes to any existing ZIP

Wallet Implementation Impact

This ZIP does not affect existing wallet functionality. It specifies a standard algorithm that wallets SHOULD follow for deterministic scanning, but does not break compatibility with existing implementations.

However, wallets that do not follow this standard may produce different results, which could cause interoperability issues. This is a reason to adopt the standard, not a breaking change.

Implementation Recommendations

  1. Block Caching: Cache block data to avoid repeated network requests during scanning.

  2. Progress Tracking: Track the last successfully scanned block height to enable resumable scans.

  3. Error Handling: Handle network errors gracefully and allow scanning to resume from the last successful block.

  4. Performance Optimization: While determinism requires sequential processing, implementations can optimize by:

    • Caching block data
    • Batch processing transactions
    • Parallel decryption attempts (but maintaining deterministic order)

  5. Testing: Implementations SHOULD include test vectors to verify deterministic behavior.

Deployment

This ZIP is a Standards Track proposal. Once approved:

  1. Wallet implementations SHOULD adopt this standard for deterministic scanning
  2. Test vectors SHOULD be included in wallet test suites
  3. Interoperability testing SHOULD verify deterministic behavior across implementations

Activation: This ZIP does not require a network upgrade. It is a wallet implementation standard that can be adopted immediately.

Acknowledgments

This ZIP is based on implementation experience from NozyWallet and discussions with the Zcash community about wallet reliability and interoperability.

Related ZIPs

The following ZIPs are related but do not cover deterministic scanning:

  • ZIP 32: Shielded Hierarchical Deterministic Wallets - Covers key derivation (HD wallets), not note scanning
  • ZIP 315: Best Practices for Wallet Implementations (Draft) - General wallet best practices, may include guidance but not a specific scanning algorithm
  • ZIP 307: Light Client Protocol for Payment Detection (Draft) - Covers light client protocols, not full node scanning
  • ZIP 310: Security Properties of Sapling Viewing Keys (Draft) - Covers viewing key security, not scanning algorithms
  • ZIP 316: Unified Addresses - Covers address encoding, not scanning
  • ZIP 224: Orchard - Covers the Orchard protocol itself, not wallet scanning

This ZIP is the first to specifically address deterministic Orchard note scanning as a standard algorithm.

References

1 Like
2

While I understand and agree with the motivations, I can’t help but wondering whether the solution is covering an issue with a given implementation.

1 Like
3

ZIP-32 (Shielded Hierarchical Deterministic Wallets) ensures correctness in note scanning by providing deterministic key derivation from a seed phrase. When NozyWallet uses ZIP-32 to derive Orchard spending keys, it guarantees that the same mnemonic will always generate the same keys, which means all notes belonging to that wallet will be found during scanning—preventing lost ZEC from missed notes. However, ZIP-32 does not make scanning faster; it only ensures that scanning is complete and correct.

The sequential block-by-block scanning process can still take days for a full blockchain sync, especially for new wallets. To address speed, NozyWallet implements incremental scanning (only scanning new blocks after the initial sync) and could potentially add optimizations like Warp Sync in the future, which can reduce initial scan times from days to seconds. The key takeaway is that ZIP-32 is essential for correctness (no lost funds), while speed requires separate optimization techniques.

4

I meant that a wallet that doesn’t restore / sync deterministically and gives different balance or tx history has a bug imo.
I understand that not everything can be recovered from the seed phrase but balance and tx history is expected.
Not sure why there needs to be a ZIP for that.

5

Just a suggestion I proposed this hopefully to help with long delay in restore/ sync wallets such as Zingo and was hoping to avoid that same problem as nosy develop. If there then I can move on to my next task

6

What specific wallet apps have this problem?

7

Not sure, but this zip could be useful in case if some apps does have a problem in the future. Hanh you know better than me you help me with Nozy.