Zebrad stalling

[Autotunafish]

This /fix branch along with the changes listed here seems to do well

Linux TCP tuning for block propagation - Troubleshooting - The Zebra Book

[the_zero]

(저의 시스템 요약 - 2세대 스레드 리퍼 2950x + 인텔 옵테인 905p 1.5tb + 16gb ecc 메모리 + zebra 4.4.1 ) 저 역시 비슷한 증상을 겪고 있습니다. 어제 나의 pc가 99.9%넘는 구간에서 15분 넘게 새로운 블록 생성이 멈춰있는 것을 발견했어요. 모든 데이터를 지우고 다시 동기화를 시작했습니다만 10시간 이상 시간이 지났음에도 20% 미만의 블록 데이터 동기화 과정을 진행중입니다.

I am experiencing a similar issue with my system (Threadripper 2950X, Intel Optane 905P 1.5TB, 16GB ECC RAM, running Zebra 4.4.1). Yesterday, I noticed that the sync process got stuck at over 99.9% for more than 15 minutes, failing to catch up with the latest blocks.

Consequently, I cleared all the data and restarted the synchronization from scratch. However, even after more than 10 hours, it is still progressing very slowly and has not even reached 20%.

[Roy101061]

I have also encounter stalling problem when setting up on WSL2, tried multiple fix but not yet resolved, here’s my current situation summary by gemini:
Environment:

• OS: Windows 10 running WSL2 (Ubuntu)

• Hardware Allocation: 20 CPU Cores, 16GB RAM

• Zebra Version: 4.4.1

Issue Description:

1. Syncing from scratch: The node consistently stalls at around block height 467k.

2. Syncing from a snapshot: I used a snapshot to jump to ~3.06M. With checkpoint_sync = true, it managed to reach ~3.18M but is now permanently stalled.

Instead of high CPU or RAM usage, my iostat shows CPU and Disk %util are both at 0%, meaning the node is completely idle waiting for network responses.

Specific Logs / Errors: I am repeatedly getting these warnings before the tip set exhausts. It seems peers are dropping the connection when requested to send these heavy blocks:

• temporary error obtaining tips: deadline has elapsed

• error downloading and verifying block e=ValidationRequestError { error: Elapsed(())...

• error downloading and verifying block e=DownloadFailed { error: SharedPeerError(ConnectionClosed)...

• error: SharedPeerError(Serialization(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })))

• Eventually leading to: exhausted prospective tip set

Troubleshooting Steps Already Taken (To rule out local bottlenecks): To ensure this isn’t a WSL2 or local networking issue, I have applied the following configurations:

1. Network Layer: Disabled Windows RSC (Disable-NetAdapterRsc) for WSL vEthernet to prevent large packet drops. Set net.ipv4.tcp_slow_start_after_idle = 0 and TCP congestion control to cubic.

2. Firewall & AV: Port 8233 is confirmed open and forwarded. Hyper-V firewall is disabled. PC-cillin (Antivirus) is completely turned off to prevent I/O or network interception. Tried with and without VPN.

3. WSL Resources: Forced swap=0 in .wslconfig to strictly prevent disk I/O latency.

4. Zebra Config (Bulldozer Mode): To prevent Tokio thread starvation, I drastically reduced concurrency and hardcoded parallel_cpu_threads:

   • download_concurrency_limit = 2

   • full_verify_concurrency_limit = 2

   • max_concurrent_block_requests = 2

   • lookahead_limit = 50

   • parallel_cpu_threads = 20

5. Peer Management: Cleared ~/.cache/zebra/network/* multiple times and hardcoded reliable seeders (e.g., dnsseed.z.cash, zec.rocks) in initial_mainnet_peers.

Despite all these optimizations, the node still asks for block ~3.18M, peers reset the connection, and the tip set exhausts.

Current conf:

[consensus]
checkpoint_sync = true

[mining]
internal_miner = false

[network]
crawl_new_peer_interval = “30s”
initial_mainnet_peers = [
“18.27.125.103:8233”,
“35.172.185.187:8233”,
“dnsseed.str4d.xyz:8233”,
“dnsseed.z.cash:8233”,
“mainnet.seeder.shieldedinfra.net:8233”,
“mainnet.seeder.zfnd.org:8233”,
“mainnet.is.yolo.money:8233”,
“zec.rocks:8233”
]
max_connections_per_ip = 1
peerset_initial_target_size = 25

[state]
cache_dir = “/home/roy/zebra-state/zebra-snapshot/zebra-2025-09-12”

[sync]
full_verify_concurrency_limit = 1
parallel_cpu_threads = 20
lookahead_limit = 100
max_concurrent_block_requests = 1

[the_zero]

sir! If we apply all of these measures, will the node operate normally? Has the effectiveness of this countermeasure been verified?

[Autotunafish]

It seems to help but the fix is unverified

[the_zero]

I would like to clarify one important context before the main content of this message.

I do not have permission to create a new post in this community. Therefore, I am unable to open a dedicated thread for this issue. As a result, I am posting my observations and technical concerns as a comment under an existing discussion.

Because of this limitation, I would like to respectfully request that if other community members or moderators find this content meaningful or relevant, they consider reposting or consolidating it into a dedicated discussion thread for broader visibility and structured review.

The content below represents a structured technical observation regarding Zebra’s synchronization behavior under intermittent node operation conditions.

본 메시지의 본문에 앞서 중요한 상황을 먼저 설명드립니다.

저는 해당 커뮤니티에서 새로운 게시글을 작성할 권한이 없습니다. 따라서 별도의 스레드를 열어 이 문제를 제기할 수 없는 상황입니다. 그 결과, 현재의 관찰 내용과 기술적 문제 제기를 기존 게시글의 댓글 형태로 작성하고 있습니다.

이러한 제한 때문에, 만약 커뮤니티 구성원 또는 운영진이 해당 내용이 의미 있다고 판단하신다면, 더 많은 사람들이 볼 수 있도록 별도의 게시글로 재구성하거나 정리해 주시기를 정중히 요청드립니다.

아래 내용은 간헐적 노드 운영 환경에서 Zebra 동기화 동작에 대해 관찰된 기술적 사항을 구조적으로 정리한 것입니다.

Empathy-Oriented Technical Proposal for Zebra Sync Liveness Stability Improvements

One-line conclusion

The observed late-stage synchronization instability in Zebra is best understood as a liveness degradation pattern emerging from the interaction of peer-based synchronization, failure classification behavior, and retry orchestration under intermittent node operation, and it may be mitigated through targeted improvements without conflicting with Zcash’s core privacy and decentralization principles.

---

Summary

This document is written with full awareness of the architectural constraints and tradeoffs inherent in Zcash, particularly its commitment to privacy, decentralization, and cryptographic correctness. Within this context, the purpose of this report is not to challenge these design priorities, but to share a recurring operational pattern observed in Zebra under realistic intermittent node usage.

In such conditions, late-stage synchronization (typically near 99% completion) may exhibit severe degradation in responsiveness, where a large portion of peers appear temporarily or effectively non-responsive. As a result, the system may appear stalled for extended periods, despite eventual recovery through repeated retry cycles.

Importantly, this report is not intended to challenge the foundational design philosophy of Zebra or Zcash, but to highlight a real-world operational boundary where existing architectural tradeoffs interact with intermittent usage patterns, leading to reduced liveness and predictability during late-stage synchronization.

This behavior does not indicate a correctness failure, but rather a liveness and operational stability issue that becomes more visible under intermittent usage patterns.

---

Reproduction Conditions

The issue is observed in environments where nodes are not continuously online, but instead operated intermittently with periodic shutdowns and restarts. After restart, nodes typically resume synchronization from a near-complete state (often above 95–99%).

Early-stage resumption behaves normally. However, as the system approaches final synchronization, performance becomes highly unstable, and in some cases appears to enter a near-stalled state.

---

Observed Behavior

During synchronization, throughput becomes increasingly irregular as the node approaches completion. A significant portion of blocks experience severe delays or repeated failures, while only a small subset completes successfully after multiple retries.

In the final synchronization region (around 99% and above), a more severe condition may occur where many peers become effectively non-responsive, leading to what appears to be a full or near-full stall in progress.

Unlike earlier degradation phases, recovery in this stage is less predictable and often requires repeated retry cycles before forward progress resumes.

---

Acknowledged Design Considerations

It is fully recognized that Zcash and Zebra operate under non-trivial constraints, including:

• Strong privacy-preserving architecture requirements

• Decentralized peer-to-peer network design

• Cryptographic verification complexity

• Avoidance of overly deterministic or centralized synchronization paths

Within this context, some degree of variability in synchronization performance may be an expected tradeoff rather than a defect.

---

Core Hypothesis

The system appears to degrade into a late-stage liveness instability under intermittent operation patterns, where peer responsiveness becomes uneven and retry dependency increases significantly.

In this state, synchronization does not fail in correctness terms, but loses temporal predictability and becomes heavily dependent on a small subset of responsive peers discovered through repeated retries.

---

Impact

While correctness is preserved, operational reliability is significantly reduced. Nodes may appear stalled during critical late-stage synchronization, even though eventual recovery is possible.

This reduces predictability for operators, increases monitoring burden, and negatively affects the perceived stability of node operation in realistic environments.

---

Why This Is a Problem

This issue does not represent a correctness failure, but rather a liveness and usability degradation. The system continues to function, but forward progress becomes inconsistent and difficult to anticipate.

This is particularly impactful near completion stages, where users expect stability rather than degradation, leading to a mismatch between expected and observed behavior.

---

Improvement Proposals

Rather than suggesting architectural redesign, the following incremental improvements may help stabilize observed behavior within existing constraints:

• More stable peer scoring under intermittent restart conditions

• Improved differentiation between transient and persistent failures

• Bounded retry and fallback strategies to avoid excessive oscillation

• Improved stability in late-stage synchronization behavior

• Enhanced observability of sync health metrics (retry rate, peer distribution, throughput variance)

[RaphiXL]

When I apply your fix(Linux TCP tuning for block propagation from the Zebra Book), then I do get some blocks occasionally.
Like every hour or so I get between 10 and up to 500 blocks.
I am no longer 5000 blocks behind, but I am staying around 2000 blocks behind despite running zebrad for about 8 hours a day.
It feels very random and sluggish.
But at least it did not completely stop.

[gustavovalverde]

The ZFND team is still working at identifying and fixing this chain stalling issue. We will formally inform about our updates soon enough.

[0xaaa]

Hi people, new here!

Also same issue. I sarted zebrad something like 2 or 3 days ago and it goes on ~34%. I thought it was a normal condition on the network, but after reading this thread, I realize that it is not.

Anyway, I feel like building things on Zcash, I don’t know what things but things.

[Autotunafish]

Release Zebra 4.5.0 · ZcashFoundation/zebra · GitHub i’m not sure if this fully addresses the sync issues, but you should still upgrade

[Roadies]

Upgraded. Still synced extremely slow due to stall issue

[Roadies]

Zcash Open Development Lab

Does this have anything to do with the stalling issue? Since the stalling issue started, there seem to be a lot of urgent updates to the network / zebra?

[Autotunafish]

The recent rash of upgrades is because of security vulnerabilities being discovered by ai since it has been developed and made more widely available in the past months. I would not guess that it is immediately connected to the stalling issue but I’m not sure.

[Roadies]

Yeah I just find the timing pretty sus..

[the_zero]

I’ve been trying to sync for more than 5 days and I’m still at 55%. I really doubt when this is going to get better. Clearly, the network is not functioning properly at the moment.

[nsheep]

Hi, we are actively in the process of investigating and trying to resolve this issue. There’s a bit more information in our recent engineering update here if you’d like to take a look. We will provide another update soon so that everyone is in the loop.

[Roadies]

Some additional info: I noticed that zebrad is still syncing but very slowly, because it is syncing in burst (once in a while it will sync ~1,000 blocks and then stop and then resume…)

2026-06-04T09:46:13+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:46:43+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:47:14+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:47:44+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:48:14+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:48:44+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:49:14+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:49:44+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:50:14+07:00 /Zebra:5.0.0/ height=2580806 delta=0

2026-06-04T09:50:44+07:00 /Zebra:5.0.0/ height=2582006 delta=1200

2026-06-04T09:51:14+07:00 /Zebra:5.0.0/ height=2582406 delta=400

2026-06-04T09:51:44+07:00 /Zebra:5.0.0/ height=2582406 delta=0

[Autotunafish]

It is a config option called checkpoint_verify_concurrency_limit, and it defaults to 1000

Config in zebrad::components::sync - Rust

[the_zero]

Has the Zebra sync stall issue actually been fixed?

I’m still stuck at 69% sync, and my PC has been running nonstop for about 9 days trying to catch up. Version 5.0.0 has been released, but honestly it doesn’t seem like this issue has been resolved on my end.

I’m still seeing the same sync stall behavior as before.

[Autotunafish]

You can try running the git branch here the classic way

GitHub - ZcashFoundation/zebra at fix/5709-sync-stall

This pr explains the fix and links to the branch

fix(network,zebrad): cure genesis-to-tip sync stall (#5709) (#10679)

Clone it down or you can also just run from the existing zebra dir ‘git fetch origin’ and then ‘git checkout fix/5709-sync-stall’ and rebuild.

if you install with cargo, then you need to run

(Sorry for the pic, the website wouldn’t show the command string correctly)

Another thing I did was also delete the existing mainnet.peers in .cache/zebra/network. i don’t know if it helped or not but node was going way faster overall and finished syncing very quickly.