You can try running the git branch here the classic way
This pr explains the fix and links to the branch
main ← fix/5709-sync-stall
opened 12:31AM - 05 Jun 26 UTC
## Summary
Fixes the sync stall where Zebra nodes freeze for minutes during ini… tial block download, requiring repeated restarts to make progress. Root cause: `FindBlocks` responses didn't register block availability in the inventory router, so `getdata` requests were load-balanced to peers that didn't have the block — causing a NotFound storm that poisoned the registry and left frontier blocks permanently un-routable.
Four coordinated changes that together produce a 0-error genesis-to-tip sync. The active mechanism depends on the sync phase:
**During checkpoint mode** (genesis through ~3.3M, including the sandblasting region ~1.4M–2M):
- **Stop poisoning the inventory registry on transient errors** (`client.rs`). Only mark inventory as missing for explicit `notfound` responses — timeouts and connection drops no longer poison routing.
- **Re-request dropped blocks** (`sync.rs`, `handshake.rs`). Re-queue a block whose download failed with `NotFound` instead of silently dropping it. Bounded by `MAX_BLOCK_REOBTAIN_RETRIES`. Also register multi-block `inv` availability from `FindBlocks` responses in the inventory monitor.
- **Duplicate-tolerant batch dispatch** (`sync.rs`, `downloads.rs`). Catches `DuplicateBlockQueuedForDownload` errors and continues processing the remaining batch instead of dropping unprocessed hashes — preventing frontier gaps from missed blocks.
Note: `CONTIGUOUS_PREFIX_P2C = 2000` exceeds the checkpoint lookahead (400–1000), so source-aware routing does not fire during checkpoint mode. The 0-error result through the sandblasting region is delivered by no-poison + re-request + duplicate-tolerant dispatch.
**During post-checkpoint full verification** (concurrency 20, near tip):
- **Source-aware block routing** (`request.rs`, `response.rs`, `set.rs`, `connection.rs`, `sync.rs`, `inbound.rs`). Track which peer announced each block hash from `FindBlocks`/`ExtendTips` responses. Route block requests to the announcing peer via `BlocksByHashFrom`. The first 2000 hashes in each batch use reliable P2C routing (contiguous-prefix guard) so the checkpoint verifier always gets its next-needed block. Sourced blocks are batched up to 8 per peer to reduce busy-peer fallback.
## Why this surfaced now
The underlying bug was always present — `FindBlocks` responses never registered block availability, and the inventory registry always poisoned on transient errors. But several converging factors turned this latent defect into a visible stall:
- **Sandblasting-era blocks** (~1.4M–2M) take 5–8 minutes to verify. Before sandblasting, blocks were cheap and the misrouting was invisible — a dropped block was recovered on the next restart cycle before anyone noticed the 8-minute timeout.
- **Degraded peer set.** As more nodes stalled or ran old versions, at-tip peer ratio dropped as low as 2–5%. Fewer good peers means more misrouted requests hit peers that genuinely don't have the block, accelerating the poisoning cascade.
- **Post-checkpoint full verification** switches from checkpoint mode (contiguity-tolerant, concurrency ~1000) to full verification (strict ordering, concurrency 20). The lower concurrency means fewer blocks in flight, so one stuck frontier block parks the entire pipeline immediately.
## Validation
Genesis-to-tip sync experiments on v5.0.0 (`c2d-standard-4`, non-SPOT, persistent disks). Each combination was tested to isolate which pieces are essential:
| Branch | Elapsed errors | Time to tip | What it tests |
|---|---|---|---|
| **This PR** | **0** | **~14h** | All four changes together |
| no-poison + re-request only | 41 | ~18h | Missing: duplicate-tolerant dispatch. Reaches tip but drops batch remainders on duplicates |
| no-poison + routing + re-request (prefix=32) | 37-42 | ~17-19h | Source routing fires but prefix too small — busy-announcer fallback causes errors |
| no-poison + routing only (no re-request) | 128 | stuck at ~2M | Missing: re-request. Dropped blocks never recovered |
| stock v5.0.0 | 127 | stuck at ~2M | Baseline |
## Credits
- Root-cause diagnosis (registry poisoning + silent block drop): @zmanian ([#5709](https://github.com/ZcashFoundation/zebra/issues/5709))
- Source-aware routing design, no-poison approach, contiguous-prefix guard: @gustavovalverde
- Minimal reimplementation and fleet validation: @oxarbitrage
Closes #5709.
Clone it down or you can also just run from the existing zebra dir ‘git fetch origin’ and then ‘git checkout fix/5709-sync-stall’ and rebuild.
if you install with cargo, then you need to run
(Sorry for the pic, the website wouldn’t show the command string correctly)
Another thing I did was also delete the existing mainnet.peers in .cache/zebra/network. i don’t know if it helped or not but node was going way faster overall and finished syncing very quickly.
2 Likes
