Zcashd error: ThreadStartWalletNotifier(): Assertion `pindexFork != nullptr' failed

Technical Support
1

Hello there. I’m new here and sort of new to Zcash. I have ran a full node on and off over the past year or so. I am currently running version 5.7.0 on Debian 12 stable.

Recently I discovered I had a faulty RAM stick. After extensive tests I have determined that the 2 sticks I have now are good, so I’m working with 16GB now. I did have a problem running the node since it looked like I had corrupted blockchain data due to that stick becoming faulty recently.

I ended up having to run zcashd with -reindex -rescan.
This was successful, however, after running for a few hours synchronized, the RAM usage was exhausted and then daemon was killed.

Now when I run zcashd I am prompted by the following error:

zcashd: init.cpp:715: void ThreadStartWalletNotifier(): Assertion `pindexFork != nullptr' failed.
Aborted

Here is the debug.log:

2023-12-05T06:51:47.514275Z  INFO main: Zcash version v5.7.0
2023-12-05T06:51:47.514285Z  INFO Init: main: InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2023-12-05T06:51:47.515457Z  INFO Init: main: Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2023-12-05T06:51:47.515909Z  INFO Init: main: Using BerkeleyDB version Berkeley DB 6.2.23: (March 28, 2016)
2023-12-05T06:51:47.515915Z  INFO Init: main: Default data directory /home/alexis/.zcash
2023-12-05T06:51:47.515932Z  INFO Init: main: Using data directory /media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata
2023-12-05T06:51:47.515956Z  INFO Init: main: Using config file /media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata/zcash.conf
2023-12-05T06:51:47.515959Z  INFO Init: main: Using at most 125 connections (1024 file descriptors available)
2023-12-05T06:51:47.518510Z  INFO Init: main: Using 16 MiB out of 16 requested for signature cache, able to store 524288 elements
2023-12-05T06:51:47.520157Z  INFO Init: main: Using 8 MiB out of 8 requested for Sapling bundle cache, able to store 262144 elements
2023-12-05T06:51:47.521760Z  INFO Init: main: Using 8 MiB out of 8 requested for Orchard bundle cache, able to store 262144 elements
2023-12-05T06:51:47.521769Z  INFO Init: main: Using 6 threads for script verification
2023-12-05T06:51:47.522179Z  INFO Init: main: Sprout parameters will be fetched from /home/alexis/.zcash-params/sprout-groth16.params if needed
2023-12-05T06:51:47.522186Z  INFO Init: main: Sapling parameters are bundled in this binary
2023-12-05T06:51:47.522189Z  INFO Init: main: Orchard parameters are generated deterministically
2023-12-05T06:51:47.522248Z  INFO main: metrics-ui thread start
2023-12-05T06:51:47.522291Z  INFO main: scheduler thread start
2023-12-05T06:51:47.653676Z  INFO Init: main: Loading Orchard parameters
2023-12-05T06:51:50.884384Z  INFO Init: main: Loaded proof system parameters in 3.362182s seconds.
2023-12-05T06:51:50.884587Z  INFO Init: main: HTTP: creating work queue of depth 16
2023-12-05T06:51:50.884627Z  INFO Init: main: Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
2023-12-05T06:51:50.884662Z  INFO Init: main: HTTP: starting 4 worker threads
2023-12-05T06:51:50.884945Z  INFO Init: main: Using wallet wallet.dat
2023-12-05T06:51:50.884962Z  INFO Init: main: CDBEnv::Open: LogDir=/media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata/database ErrorFile=/media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata/db.log
2023-12-05T06:51:50.888723Z  INFO Init: main: Bound to [::]:8233
2023-12-05T06:51:50.888739Z  INFO Init: main: Bound to 0.0.0.0:8233
2023-12-05T06:51:50.888751Z  INFO Init: main: Cache configuration:
2023-12-05T06:51:50.888756Z  INFO Init: main: * Using 2.0MiB for block index database
2023-12-05T06:51:50.888759Z  INFO Init: main: * Using 120.0MiB for chain state database
2023-12-05T06:51:50.888761Z  INFO Init: main: * Using 328.0MiB for in-memory UTXO set
2023-12-05T06:51:50.888783Z  INFO Init: main: Opening LevelDB in /media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata/blocks/index
2023-12-05T06:51:52.521647Z  INFO Init: main: Opened LevelDB successfully
2023-12-05T06:51:52.521668Z  INFO Init: main: Opening LevelDB in /media/alexis/Starseed-Data/.starbits/.zcash/zcash/zdata/chainstate
2023-12-05T06:51:53.527740Z  INFO Init: main: Opened LevelDB successfully
2023-12-05T06:52:09.904237Z  INFO Init: main: LoadBlockIndexDB: last block file = 1948
2023-12-05T06:52:09.908483Z  INFO Init: main: LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=10604, size=76592305, heights=0...2302808, time=2016-10-28...2023-11-19)
2023-12-05T06:52:09.908490Z  INFO Init: main: Checking all blk files are present...
2023-12-05T06:52:10.285507Z  INFO Init: main: LoadBlockIndexDB: transaction index disabled
2023-12-05T06:52:10.285518Z  INFO Init: main: LoadBlockIndexDB: insight explorer disabled
2023-12-05T06:52:10.285520Z  INFO Init: main: LoadBlockIndexDB: light wallet daemon disabled
2023-12-05T06:52:10.745381Z  INFO Init: main: Initializing databases...
2023-12-05T06:52:10.987840Z  INFO Init: main:  block index           20099ms
2023-12-05T06:52:11.022710Z  INFO Init: main: nFileVersion = 5070050
2023-12-05T06:52:11.022724Z  INFO Init: main: Keys: 107 plaintext, 0 encrypted, 107 w/ metadata, 107 total
2023-12-05T06:52:11.022729Z  INFO Init: main: ZKeys: 4 plaintext, 0 encrypted, 4 w/metadata, 4 total
2023-12-05T06:52:11.076613Z  INFO Init: main:  wallet                   89ms
2023-12-05T06:52:11.077663Z  INFO main: txnotify thread start

I have tested the two RAM sticks I have numerous times with memtest86+ and they passed 2 times with 4 passes each time. So I don’t think its that memory that is the issue. But this error is new to me and did begin happening after the memory exhaustion crash. It was using over max RAM capacity and swapfile filled up too. The OS did not crash or anything, since zcashd process was killed.

I don’t want to reindex if there is something I can do to get it running again. Thanks so much for reading this and for your help.

3 Likes
2

I have posted this into the Zcash R&D Discord where the devs hang out more, which is here.

You can also file an issue on github (which, if you don’t know, is basically just a report about the error).

3 Likes
3

Haven’t seen this before thanks for posting.

3 Likes
4

@Lexis Here’s what I think that have happened: the block whose hash was recorded in wallet.dat as the best block no longer exists in the blockchain (due to a reorganization or faulty RAM at that time).

I suggest you try the following:

  • make a backup copy of your original wallet.dat,
  • delete wallet.dat so that zcashd creates a new empty one on next startup,
  • start zcashd and wait for it to fully sync,
  • stop zcashd, delete debug.log,
  • restore your original wallet.dat from backup location,
  • start zcashd & keep your fingers crossed.
3 Likes
5

Thank you very much for the suggestions. I have backed up the wallet.dat

I’m currently working on getting both Zcash on Monero nodes fully re-indexed so it will take a while.
Monero is working for now and almost done. I’m syncing that one first because I only want to work on one at a time. Zcash was causing a memory exhaustion so I’ll do it by itself when I can monitor the process.

I will try these steps out and update as soon as I can.

3 Likes
6

Memory exhaustion during re-indexing is a known issue https://github.com/zcash/zcash/issues/6651 with low probability of getting more attention. Also, it takes almost the same time as sync from scratch. That’s why I usually prefer fresh sync, if not on a metered internet connection.

2 Likes
7

Sorry late reply. Had problems with Debian stable kernel, then holidays came.
Is this what that looks like when running dmseg?

[  +0.000001] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/session.slice/vte-spawn-c4e4559b-4130-4033-9397-54cbbfbda3a6.scope,task=zcashd,pid=438760,uid=1000
[  +0.000045] Out of memory: Killed process 438760 (zcashd) total-vm:11491880kB, anon-rss:8707628kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:18776kB oom_score_adj:200

I followed your tips but it never finished syncing and now when starting zcashd, memory exhaustion and process is killed every time.

Should I rebuild in new location? It was on a HDD but I just got an SSD.

8

I haven’t heard how much RAM is your pc equipped with?

If I were you, I would start sync from scratch (with empty datadir and wallet, on SSD) and closely track available memory during the process. You can use htop for that purpose. When RAM is almost full, you should stop zcashd gracefully (zcash-cli stop) regardless of block height. When you run zcashd again, it will use less memory than before stopping because of specific optimization (Reduce memory usage of CBlockIndex by daira · Pull Request #6192 · zcash/zcash · GitHub). Keep watching at mem. consumption and restarting the daemon as long as you don’t reach current chain tip.

IMO, it’s a terrible user experience but the only way unless you expand RAM capacity.

P.S. When you finish chain sync with success, stop zcashd and restore your wallet from backup. Be ready to wait for rescan to finish, because best block height written in the wallet is way below chain tip.

P.P.S. Since reindex is “mission imposible” on your system, be sure to enable indexes you might need later before you start the sync. Check for txindex or insightexplorer conf settings according to your use case and make sure that your SSD has enough of free space (260+ GB with txindex)

9

I’m not sure I understand how to enable indexes I’d need.
And how do I check for txindex or insightexplorer conf settings?

The SSD has 2TB
I only have 16GB memory. :frowning:

10

If you intend to use getrawtransaction rpc command, it would require txindex=1 line appended to the zcash.conf file. It’s disabled by default.

You probably won’t need other indexes I’ve mentioned, unless you plan to run an explorer or light wallet server.

2TB SSD is ok and 16GB of RAM isn’t too bad IMO, you should be able to sync with just a few interruptions.

Good luck! :crossed_fingers:

2 Likes
11

Thank you so much! I managed to sync entire chain to SSD and restored the wallet.dat.
Yay I have my Zcash again. Happy New Year to you and everyone in the thread.

3 Likes
12

Sorry I’m not done here after all. I ran into a separate issue with the wallet.
When I try to create a new address I’m prompted with this error:

Screenshot at 2024-01-01 21-03-39
Screenshot at 2024-01-01 21-03-39887×197 10.8 KB

I couldn’t find any documentation on this at https://docs.zecwallet.co/troubleshooting/ So I attempted to run ./zcashd-wallet-tool

I did not have a backup file of the exported phrases I have written down on a printout.
I was given a new set of recovery phrases.

After restarting zcashd and connecting with the wallet, I was still unable to make new addresses.

1 Like
13

It seems like you’re running zecwallet full node? That particular wallet has been deprecated for sometime now and will likely not start. You should import your seed phrase into a lightwallet and recover your account. Ywallet has recovery options that will allow for the proper restoring of a zecwallet derived seed phrase.
If you would like to utilize your existing full node, then you can start a lightwalletD server connected to your zcashd instance and then connect your light wallet locally. Otherwise you can connect to the public network.
Zcashd itself does not support importing a seed so any native solutions will not work here.

I’m going to stress here that the seed phrase that you had when you originally created this wallet that you are trying to restore is THE seed phrase.
The one that you got when you ran the wallet tool was a new phrase associated with the zcashd internal wallet, not zecwallet.

14

Hey I ran into this error before, hopefully this helps you!

If not, can try Autotuna’s suggestion

2 Likes
15

What is your end goal?
Getting your funds out? Running a node?

1 Like
16

I came across that video @dismad , thank you so much.
It was helpful, but I couldn’t solve my issue using the method in the video.
I did run ./zcash-cli z_getnewaccount successfully but still can’t use the zecfullnode wallet to create and addresses.

I was just trying to get my funds off the wallet @hanh .
I made it in the past using a zcashd and the zecwallet fullnode v1.8.7

Since recovering the wallet.dat that I saved from when I had corrupted data, I’m now unable to make any new addresses with the GUI front end.

I found this useful list of zcash-cli commands here
I made unified and transparent addresses through zcash-cli, and upon restarting the zecwallet GUI, I am able to utilize the addresses.

Funds are successfully moved to another wallet completely. But not sure how to make new shielded addresses with the zcash-cli commands, and since I cant do it with the GUI anymore. Maybe I’ll just stop hosting the fullnode. I’m really just learning as well as trying to support the network of my favorite projects.

1 Like
17

Personally, I use zebra as a full node and a seperate wallet app. I’m not a huge fan of the zcashd wallet functionalities. It’s too error-prone for my taste.

1 Like
18

Since the activation of NU5, the account model that was previously unused in zcashd was reactivated to accommodate unified addresses. Since then, generating a legacy sapling address is now deprecated and creating unified addresses is done through the z_getaddressforaccount call. z_getaddressforaccount - Zcash 5.5.0 RPC Docs
Directly using the zcashd internal wallet is definitely very advanced level. Zecwallet made that easier with a nice Gui front end and there are some other tools like Zenith that make using zcash-cli a bit easier. So for learning, sure, just read the docs.
If you run lightwalletD with your node, then you can use much more user friendly wallet software while also still running your own node. The zcashd internal wallet functionality doesn’t even need to be enabled to serve a lightwalletD server.
Zebra can do this as well and usually faster but the advantage of zcashd currently is that it supports many more RPC calls, which if you’re learning, you can play with and see what kinds of information that you can get. Maybe stuff that you might find on a block Explorer or something (there’ll also be alot of gibberish!). Zebra has a build feature that provides metrics as well in a different way.
Haven’t even scratched surface (maybe I suppose lol), glad you got your funds back👍.

1 Like