Testnet stuck on blk 903799 (v2.1.2) -- (Fixed by v2.1.2-3)

Ooops… it got stuck on the Heartwood activation block.

2020-05-04 20:20:40 UpdateTip: new best=001296b0b05b3818f58d87bc6b24fe7a247771c9fd59c656a49163e94d7ec1fb height=903799 bits=525394216 log2_work=37.237533 tx=1295308 date=2020-05-04 20:20:37 progress=1.000000 cache=0.0MiB(39tx)
2020-05-04 20:21:15 ERROR: CScriptCheck(): 063fcaffbad27430603d39bc8e00187e751ecc1ac0fbb4b8f054f013ceb8d80a:0 VerifySignature failed: Script evaluated without error but finished with a false/empty top stack element

Then went into banning nodes…

2020-05-04 20:25:43 Misbehaving: (0 -> 100) BAN THRESHOLD EXCEEDED
2020-05-04 20:25:43 ERROR: invalid header received
2020-05-04 20:25:43 ProcessMessages(headers, 4465 bytes) FAILED peer=6
2020-05-04 20:26:02 ERROR: AcceptBlockHeader: prev block invalid

…and… now its back!!

2020-05-04 20:29:43 UpdateTip: new best=05688d8a0e9ff7c04f6f05e6d695dc5ab43b9c4803342d77ae360b2b27d2468e height=903800 bits=537395199 log2_work=37.237533 tx=1295309 date=2020-05-04 20:29:09 progress=1.000000 cache=0.0MiB(44tx)
2020-05-04 20:29:43 Warning: Found invalid chain at least ~6 blocks longer than our best chain.

…and apparently not, it seemed to stall and no new blocks were being received.

Shutdown, enabled CPU mining & restarted - this gave an error message complaining of corruption & a reindex requested.

Re-index complete, still seeing the warning about an invalid chain being detected however its now up to blk 903809. Only two peers connected & lots of ‘obsolete version’ warnings - guessing many didn’t upgrade.

I’ll leave it running to see what happens (mining at 0.0280 sols/s! woo!!!)

Whatever the issue was its now resolved. Node is running normally, even managed to mine a block.

Edit: (sigh) Shutdown, restarted to try mining to a zaddr & got an error about corrupt chainstate & have to reindex again. (doing that now)


ECC’s testnet node (which is sometimes the only node running testnet) wasn’t upgraded prior to Heartwood activation (Heartwood did activate earlier than expected). Being fixed now.


I think this is https://github.com/zcash/zcash/issues/4496. Are you saying you have needed to reindex after every restart since testnet Heartwood activation, or only some of them?

Not sure. It’s on the second reindex but conditions were the same as the first time (before ECC node upgraded).

Slow machine so takes a while, will let you know.

EDIT: Yep! Somethings pooched. The reindex completed, so did a shutdown & restart. Got the following error :-

2020-05-05 02:21:12 ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot != hashChainHistoryRoot): CBlockIndex(pprev=0x55d5b5bd4910, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)
2020-05-05 02:21:12 Aborted block database rebuild. Exiting.

EDIT: Confirmed. Did another reindex & get the same error after shutdown/restart.

EDIT: Noticed v2.1.2-1 is available, upgraded & reindexing…

EDIT: Ooops! Shutdown & restart gives ‘Error loading block database’ & reindex requested

EDIT: Confirmed. Reindex completed with 2.1.2-1, shutdown & restart gives the following error :-

ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot 626444395cd5963d3dba2652ee5bd73ef57555cca4d9f0d52e887a3bf44488e2 != hashChainHistoryRoot 0000000000000000000000000000000000000000000000000000000000000000): CBlockIndex(pprev=0x55e40f3e5030, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)
2020-05-08 19:58:34 Aborted block database rebuild. Exiting.

EDIT: v2.1.2-3 released.
Upgraded & ran zcashd without the reindex flag.
Watching debug.log it did a rescan, puny test machine took ages (11 mins)
Rescan completed, waited for a few new blocks to arrive, then shutdown.

Restarted… moment of truth…and…
Woo! Its working :slight_smile: