Zebrad stalling

[Roadies]

Hello,

Since yesterday my local zebra node has stalled and lagged ~2,000 blocks compared to the tip. I tried everything ChatGPT suggested but nothing worked. Anyone could help? Thanks.

Here’s the log. I’m running zebrad 4.4.1 on Linux

journalctl -u zebrad --since “30 min ago” --no-pager | tail -n 250

May 25 06:57:16 Ticket zebrad[7409]: 2026-05-24T23:57:16.904587Z WARN zebrad::components::sync: error downloading and verifying block e=Invalid { error: Block { source: Transaction(InternalDowncastError(“downcast to known transaction error type failed, original error: Elapsed(())”)) }, height: Height(3351798), hash: block::Hash(“00000000009d19b4881702e2fa5115b2dad3945984d6de2ee83f5aadc78218f2”), advertiser_addr: Some(v4redacted:8233) }

May 25 07:05:33 Ticket zebrad[7409]: 2026-05-25T00:05:33.230810Z WARN zebrad::components::sync: error downloading and verifying block e=Invalid { error: Block { source: Transaction(InternalDowncastError(“downcast to known transaction error type failed, original error: Elapsed(())”)) }, height: Height(3351804), hash: block::Hash(“0000000000646edb5ed872ae0c8bf58900d721e8e05900d292b30c9ae889e416”), advertiser_addr: Some(v4redacted:8233) }

May 25 07:10:15 Ticket zebrad[7409]: 2026-05-25T00:10:15.093286Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 10 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=10m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:11:15 Ticket zebrad[7409]: 2026-05-25T00:11:15.108213Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 11 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=11m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:12:15 Ticket zebrad[7409]: 2026-05-25T00:12:15.121871Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 12 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=12m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:12:41 Ticket zebrad[7409]: 2026-05-25T00:12:41.461448Z WARN zebrad::components::sync: error downloading and verifying block e=Invalid { error: Block { source: Transaction(InternalDowncastError(“downcast to known transaction error type failed, original error: Elapsed(())”)) }, height: Height(3351785), hash: block::Hash(“000000000007bb2b686b603746a3d6c0007361d10b3567cd1357f363061f6970”), advertiser_addr: Some(v4redacted:8233) }

May 25 07:13:15 Ticket zebrad[7409]: 2026-05-25T00:13:15.137145Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 13 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=13m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:14:15 Ticket zebrad[7409]: 2026-05-25T00:14:15.152630Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 14 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=14m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:15:15 Ticket zebrad[7409]: 2026-05-25T00:15:15.166160Z WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 15 minutes. Hint: check your network connection, and your computer clock and time zone sync_percent=99.930% current_height=Height(3351773) network_upgrade=Nu6_1 time_since_last_state_block=15m 35s target_block_spacing=1m 15s max_block_spacing=None is_syncer_stopped=false

May 25 07:22:05 Ticket zebrad[7409]: 2026-05-25T00:22:05.222398Z WARN zebrad::components::sync: error downloading and verifying block e=Invalid { error: Block { source: Transaction(InternalDowncastError(“downcast to known transaction error type failed, original error: Elapsed(())”)) }, height: Height(3351786), hash: block::Hash(“00000000000b46d3dc8a311a043340e7348ffb80de099d5dfdec3d77d203598c”), advertiser_addr: Some(v4redacted:30841) }

[Lowo88]

What are the RAM disk space on this PC? Check RAM, swap, CPU, and disk usage while stalled.

[Autotunafish]

Can you check your peers?

Doing anything with the node besides just running it?

[shieldedmark]

It could be this: feat(zebrad): warn at startup if Linux TCP slow-start-after-idle is on by evan-forbes · Pull Request #10513 · ZcashFoundation/zebra · GitHub

Can you run sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0 and restart?

[Roadies]

32GB RAM, 2TB disk, CPU i7 so I don’t think they are the problems. There are 100+ peers as well

[Roadies]

It didn’t work..

[Roadies]

OK I have decided to reinstall zebra clean and sync node from the scratch. Wish me luck!

[Ec1ipse]

I am also having this same issue, since the beginning of the weekend. I was able to restrict my node to just 2 IP addresses that I was able to confirm worked. I let it run over night and sync’d about half the chain, but some other peers leaked through and eventually it stopped downloading.

I’ve restarted from a fresh format on a 8gb Raspberry Pi 5 using a 1tb NVMe drive. It sync’d the first 300,000 blocks perfectly fine and then it just hit a wall. Here is my log, showing very similar details to OP.

2026-05-25T03:27:34.604047Z  INFO sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(310435))
2026-05-25T03:27:34.747773Z  INFO init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: state is not fully synced yet, remaining checkpoints will be verified during syncing
2026-05-25T03:27:34.747797Z  INFO init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: finished state checkpoint validation
2026-05-25T03:27:43.903036Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=475 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(310435))
2026-05-25T03:27:44.605736Z  INFO zebrad::components::sync::end_of_support: Checking if Zebra release is inside support range ...
2026-05-25T03:27:44.605761Z  INFO zebrad::components::sync::end_of_support: Zebra release is supported until block 3391480, please report bugs at https://github.com/ZcashFoundation/zebra/issues
2026-05-25T03:27:46.164253Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=966 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(310435))
2026-05-25T03:27:48.832100Z  WARN sync:try_to_sync:try_to_sync_once: zebrad::components::sync: error downloading and verifying block e=DownloadFailed { error: SharedPeerError(ConnectionClosed), hash: block::Hash("00000000027035796ab2cbf3533c82a9b53c9027f9bb466d8e58ab481062c512") }
2026-05-25T03:27:48.833279Z  INFO sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(310435))
2026-05-25T03:27:54.604024Z  INFO peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=74 peer_cache_file="/home/ec1ipse/.cache/zebra/network/mainnet.peers"
2026-05-25T03:28:34.619587Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.209% current_height=Height(310435) network_upgrade=BeforeOverwinter remaining_sync_blocks=3060742 time_since_last_state_block=1m
2026-05-25T03:28:55.834599Z  INFO sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(310435))
2026-05-25T03:28:57.844118Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=496 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(310435))
2026-05-25T03:28:57.850596Z  INFO sync:try_to_sync:obtain_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=400 current_range=(Excluded(Height(310435)), Included(Height(310835)))
2026-05-25T03:28:58.586823Z  INFO sync:try_to_sync:try_to_sync_once:extend_tips:checkpoint: zebra_consensus::checkpoint: verified checkpoint range block_count=400 current_range=(Excluded(Height(310835)), Included(Height(311235)))
2026-05-25T03:28:59.589396Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=866 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(310469))
2026-05-25T03:29:00.309763Z  INFO sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2026-05-25T03:29:00.309807Z  INFO sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(310487))
2026-05-25T03:29:34.633804Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059929 time_since_last_state_block=20s
2026-05-25T03:30:07.311217Z  INFO sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(311235))
2026-05-25T03:30:10.602285Z  INFO sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2026-05-25T03:30:10.602390Z  INFO sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(311235))
2026-05-25T03:30:34.647420Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059930 time_since_last_state_block=1m 20s
2026-05-25T03:31:17.603364Z  INFO sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(311235))
2026-05-25T03:31:18.263420Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=742 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(311235))
2026-05-25T03:31:19.766765Z  INFO sync:try_to_sync:try_to_sync_once: zebrad::components::sync: extending tips tips.len=1 in_flight=999 extra_hashes=0 lookahead_limit=1000 state_tip=Some(Height(311235))
2026-05-25T03:31:34.663382Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059931 time_since_last_state_block=2m 20s
2026-05-25T03:32:34.677547Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059931 time_since_last_state_block=3m 20s
2026-05-25T03:32:54.605636Z  INFO peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=75 peer_cache_file="/home/ec1ipse/.cache/zebra/network/mainnet.peers"
2026-05-25T03:33:34.691445Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059932 time_since_last_state_block=4m 20s
2026-05-25T03:34:34.705361Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059933 time_since_last_state_block=5m 20s
2026-05-25T03:35:34.717479Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059934 time_since_last_state_block=6m 20s
2026-05-25T03:36:34.730054Z  INFO zebrad::components::sync::progress: estimated progress to chain tip sync_percent=9.232% current_height=Height(311235) network_upgrade=BeforeOverwinter remaining_sync_blocks=3059935 time_since_last_state_block=7m 20s

[Autotunafish]

@Roadies is this zebra modified in any way?

[Roadies]

Yes same issue. I reinstall zebra and it stucks at 340,000 blocks. Same error log:

May 25 18:10:23 Ticket zebrad[113081]: 2026-05-25T11:10:23.776026Z WARN zebrad::components::sync: error downloading and verifying block e=ValidationRequestError { error: Elapsed(()), height: Height(339458), hash: block::Hash(“000000000848f0cab4dfa42f2d2c38f4d63a0303a446204d3a225ef09ff574be”) }

May 25 18:19:38 Ticket zebrad[113081]: 2026-05-25T11:19:38.452460Z WARN zebrad::components::sync: error downloading and verifying block e=ValidationRequestError { error: Elapsed(()), height: Height(340373), hash: block::Hash(“000000000673331ad79e079e8d7602ec499bac45ab1551eba7dcb8fd163c251e”) }

[Roadies]

I don’t think so…

[Autotunafish]

What installation procedure are you following?

[Ec1ipse]

I am following the process 100% to the letter at zechub.wiki/developers/quick-start

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt update
sudo apt install libclang-dev clang pkg-config openssl protobuf-compiler npm

cargo install --git https://github.com/ZcashFoundation/zebra --tag v4.4.1 zebrad

zebrad generate -o ~/.config/zebrad.toml

[consensus]
checkpoint_sync = true

[mempool]
eviction_memory_time = "1h"
tx_cost_limit = 80000000

[network]
cache_dir = true
crawl_new_peer_interval = "1m 1s"
listen_addr = "[::]:8233"
network = "Mainnet"
peerset_initial_target_size = 25

[rpc]
listen_addr = "127.0.0.1:8232"
cookie_dir = "/home/ec1ipse/.cache/zebra"
enable_cookie_auth = false

[state]
cache_dir = "/home/ec1ipse/.cache/zebra"
delete_old_database = true
ephemeral = false

[sync]
checkpoint_verify_concurrency_limit = 1000
download_concurrency_limit = 50
full_verify_concurrency_limit = 20

[tracing]
buffer_limit = 128000
use_color = true

zebrad start

I can say I spend the night attempting some changes to the syncing code, reducing the time it takes to restart since this tends to take ~8+ mins to start again. This started the Syncing process again, but it was very slow as something would cause the blocks to either stop downloading or something where it would force restart and it would start syncing for a moment, then stop and force restart. This is not a fix.

[Roadies]

This is what I did. I believe nothing unusual here.

sudo apt update

sudo apt install -y git build-essential pkg-config libssl-dev clang cmake

command -v rustup >/dev/null || curl https://sh.rustup.rs -sSf | sh -s – -y

source $HOME/.cargo/env

rustup update stable

git clone GitHub - ZcashFoundation/zebra: Zcash - Financial Privacy in Rust 🦓 · GitHub ~/zebra

cd ~/zebra

git fetch --tags

git checkout v4.4.1

cargo build --release -p zebrad

sudo install -m0755 target/release/zebrad /usr/local/bin/zebrad

sudo mkdir -p /etc/zebra

sudo tee /etc/zebra/zebrad.toml >/dev/null <<‘EOF’

state

cache_dir = “/var/lib/zebra”

network

listen_addr = “127.0.0.1:8233”

rpc

listen_addr = “127.0.0.1:8232”

enable_cookie_auth = false

tracing

filter = “warn”

EOF

sudo tee /etc/systemd/system/zebrad.service >/dev/null <<‘EOF’

Unit

Description=Zcash Zebra Node

Wants=network-online.target

After=network-online.target

RequiresMountsFor=/var/lib/zebra

ConditionPathIsMountPoint=/var/lib/zebra

Service

User=xxx

Group=xxx

ExecStart=/usr/local/bin/zebrad -c /etc/zebra/zebrad.toml start

Restart=on-failure

RestartSec=5

LimitNOFILE=1048576

Install

WantedBy=multi-user.target

EOF

sudo systemctl daemon-reload

sudo systemctl enable --now zebrad

[Autotunafish]

Other reports of stalls and/or extreme slowdowns elsewhere so may not be just yall.

[Roadies]

aight so wen fixed?

[scalar]

fix(consensus): InternalDowncastError(Elapsed) near-tip sync stall after mempool bypass removal

• fix(consensus): InternalDowncastError(Elapsed) near-tip sync stall after mempool bypass removal · Issue #10629 · ZcashFoundation/zebra · GitHub
• fix(consensus): handle Elapsed timeout in BoxError downcast by oxarbitrage · Pull Request #10630 · ZcashFoundation/zebra · GitHub

@oxarbitrage has made few patch for your issue. Should be merged once it’s reviewed by maintainers.

[gustavovalverde]

If someone having this issue can deploy their nodes from this branch and confirm if the behavior is gone, it would help us cross-check fix(zebrad): retry explicit missing inventory without poisoning peers by gustavovalverde · Pull Request #10633 · ZcashFoundation/zebra · GitHub

[itz]

Hey, we’ll test on some fullnodes that we have for Maya validators

[RaphiXL]

I do have the same problem since Friday (2026-05-22)
I am running zebrad inside WSL on Debian-Trixi (version 4.4.1)
I tried every fix Gemini suggested:
→ deleting the peer list (rm -f /root/.cache/zebra/network/mainnet.peers)
→ Verify only one block at a time: full_verify_concurrency_limit = 1

Here is a summary from Gemini about what I have tried:
Bug Report: Zebra Sync Stall on Mainnet under WSL2 (Debian) near Chain Tip

Environment

• Zebra Version: 4.4.1+11.gd5ccbbd (Commit: d5ccbbd, Built with Rust 1.93.0)

• OS: Linux (Debian Trixi) running inside WSL2 on Windows 11

• CPU: AMD Ryzen 7 9800X3D (12 threads allocated to Rayon)

• Network: Dual-stack (IPv4 & IPv6 available via host), UPnP/NAT routing via WSL virtual switch.

• Database State: Valid, running version 27.0.0, total size ~274.5 GB.

Symptom Description

The node successfully syncs ~99.8% of the blockchain, but completely stalls exactly 5,497 blocks before the chain tip (at Height 3350744 to 3350761).

The node enters an infinite loop of attempting to sync, failing to download/verify the next prospective blocks, logging a timeout or a block verification failure, and then outputting exhausted prospective tip set. It waits for the 67-second retry cooldown and repeats indefinitely without making forward progress.

Detailed Logs of the Two Core Behaviors

Behavior A: Cryptographic Verification Timeout (Elapsed(())) When full_verify_concurrency_limit was left at its default value (or tuned to 20), Zebra attempted to process multiple complex near-tip blocks concurrently. This resulted in internal transaction verification timeouts:

Plaintext

WARN sync:try_to_sync:try_to_sync_once: zebrad::components::sync: error downloading and verifying block e=Invalid { error: Block { source: Transaction(InternalDowncastError("downcast to known transaction error type failed, original error: Elapsed(())")) }, height: Height(3350780), hash: block::Hash("0000000000579fc134ffddcf7c798903fabc7a59257793b743253942ebc3c2e3"), advertiser_addr: Some(v4redacted:8233) }
INFO sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(3350744))

Behavior B: Exhausted Prospective Tip Set After mitigating the concurrency issue, the node safely bypasses the validation crash but immediately stalls slightly later (Height 3350761) with a network peer starvation symptom:

Plaintext

INFO sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
INFO zebrad::components::sync::gossip: height=Height(3350761) request=AdvertiseBlock(block::Hash("..."), None) log_msg="sending committed block broadcast"
INFO sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(3350761))
WARN zebrad::components::sync::progress: chain updates have stalled, state height has not increased for 12 minutes. Hint: check your network connection, and your computer clock and time zone

Troubleshooting Steps Attempted

1. WSL Network Routing / Self-Connection Mitigation

   • Problem: In early logs, Zebra was repeatedly trying to dial its own public IP address because of how WSL2 handles outbound virtual network loops, causing widespread connection resets (os error 104) and connection timeouts.

   • Attempted Fix 1: Adding deny_list to [network]. This threw a fatal startup error because deny_list is an unknown/unsupported configuration field in Zebra.

   • Attempted Fix 2: Configured external_addr = "MY_PUBLIC_IP:8233" under [network]. This successfully resolved the self-dialing loop, allowing clean, external P2P handshakes with DNS seeds and initial peers.

2. Linux TCP Kernel Optimization

   • Executed sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0 to prevent the Linux kernel from clamping the congestion window during the sync idle/retry periods. This did not fix the stall.

3. Concurrency Bottleneck Mitigation

   • Action: Reduced full_verify_concurrency_limit from 20 down to 1 in zebrad.toml.

   • Result: This successfully eliminated the Elapsed(()) transaction verification errors on heavy blocks, proving that concurrent execution of Halo2 proofs on these specific blocks causes internal timeouts even on high-end hardware (AMD 9800X3D). However, the node still fails to advance past 3350761 due to the tip set exhaustion.

4. Time & Synchronization Verifications

   • Verified the internal WSL clock integrity via date. The Linux system time is accurately synchronized to the exact second with the Windows host and network time, ruling out Zcash block timestamp validation rejections due to clock drift.

5. Peer Cache Purging

   • Repeatedly deleted the compiled peer cache file via rm -f /root/.cache/zebra/network/mainnet.peers before fresh boots to force clean DNS seed lookups. Zebra successfully finds up to 75 fresh peers, but still claims the prospective tip set is exhausted.

Current Configuration (zebrad.toml)

Ini, TOML

[consensus]
checkpoint_sync = true

[network]
cache_dir = true
crawl_new_peer_interval = "1m 1s"
initial_mainnet_peers = [
    "dnsseed.str4d.xyz:8233",
    "dnsseed.z.cash:8233",
    "mainnet.seeder.shieldedinfra.net:8233",
    "mainnet.seeder.zfnd.org:8233",
]
listen_addr = "0.0.0.0:8233"
external_addr = "82.135.83.251:8233"
max_connections_per_ip = 1
network = "Mainnet"
peerset_initial_target_size = 25

[sync]
checkpoint_verify_concurrency_limit = 1000
download_concurrency_limit = 50
full_verify_concurrency_limit = 1
parallel_cpu_threads = 0