Testnet stuck on blk 903799 (v2.1.2) -- (Fixed by v2.1.2-3)

General
1

Ooops… it got stuck on the Heartwood activation block.

2020-05-04 20:20:40 UpdateTip: new best=001296b0b05b3818f58d87bc6b24fe7a247771c9fd59c656a49163e94d7ec1fb height=903799 bits=525394216 log2_work=37.237533 tx=1295308 date=2020-05-04 20:20:37 progress=1.000000 cache=0.0MiB(39tx)
2020-05-04 20:21:15 ERROR: CScriptCheck(): 063fcaffbad27430603d39bc8e00187e751ecc1ac0fbb4b8f054f013ceb8d80a:0 VerifySignature failed: Script evaluated without error but finished with a false/empty top stack element

Then went into banning nodes…

2020-05-04 20:25:43 Misbehaving: 138.68.230.245:18233 (0 → 100) BAN THRESHOLD EXCEEDED
2020-05-04 20:25:43 ERROR: invalid header received
2020-05-04 20:25:43 ProcessMessages(headers, 4465 bytes) FAILED peer=6
2020-05-04 20:26:02 ERROR: AcceptBlockHeader: prev block invalid

…and… now its back!!

2020-05-04 20:29:43 UpdateTip: new best=05688d8a0e9ff7c04f6f05e6d695dc5ab43b9c4803342d77ae360b2b27d2468e height=903800 bits=537395199 log2_work=37.237533 tx=1295309 date=2020-05-04 20:29:09 progress=1.000000 cache=0.0MiB(44tx)
2020-05-04 20:29:43 Warning: Found invalid chain at least ~6 blocks longer than our best chain.

…and apparently not, it seemed to stall and no new blocks were being received.

Shutdown, enabled CPU mining & restarted - this gave an error message complaining of corruption & a reindex requested.

Re-index complete, still seeing the warning about an invalid chain being detected however its now up to blk 903809. Only two peers connected & lots of ‘obsolete version’ warnings - guessing many didn’t upgrade.

I’ll leave it running to see what happens (mining at 0.0280 sols/s! woo!!!)

Whatever the issue was its now resolved. Node is running normally, even managed to mine a block.

Edit: (sigh) Shutdown, restarted to try mining to a zaddr & got an error about corrupt chainstate & have to reindex again. (doing that now)

2 Likes
2

ECC’s testnet node (which is sometimes the only node running testnet) wasn’t upgraded prior to Heartwood activation (Heartwood did activate earlier than expected). Being fixed now.

4 Likes
3

I think this is Chain inconsistency from testnet Heartwood activation · Issue #4496 · zcash/zcash · GitHub. Are you saying you have needed to reindex after every restart since testnet Heartwood activation, or only some of them?

4

Not sure. It’s on the second reindex but conditions were the same as the first time (before ECC node upgraded).

Slow machine so takes a while, will let you know.

EDIT: Yep! Somethings pooched. The reindex completed, so did a shutdown & restart. Got the following error :-

2020-05-05 02:21:12 ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot != hashChainHistoryRoot): CBlockIndex(pprev=0x55d5b5bd4910, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)
2020-05-05 02:21:12 Aborted block database rebuild. Exiting.

EDIT: Confirmed. Did another reindex & get the same error after shutdown/restart.

EDIT: Noticed v2.1.2-1 is available, upgraded & reindexing…

EDIT: Ooops! Shutdown & restart gives ‘Error loading block database’ & reindex requested

EDIT: Confirmed. Reindex completed with 2.1.2-1, shutdown & restart gives the following error :-

ERROR: LoadBlockIndex(): block index inconsistency detected (hashLightClientRoot 626444395cd5963d3dba2652ee5bd73ef57555cca4d9f0d52e887a3bf44488e2 != hashChainHistoryRoot 0000000000000000000000000000000000000000000000000000000000000000): CBlockIndex(pprev=0x55e40f3e5030, nHeight=903800, merkle=d2277b2d66b2faa3f6ac6cd2fa6f0655130f4a7a19cab97076ce5024ee203e9b, hashBlock=0044f3c696a242220ed608c70beba5aa804f1cfb8a20e32186727d1bec5dfa1e)
2020-05-08 19:58:34 Aborted block database rebuild. Exiting.

EDIT: v2.1.2-3 released.
Upgraded & ran zcashd without the reindex flag.
Watching debug.log it did a rescan, puny test machine took ages (11 mins)
Rescan completed, waited for a few new blocks to arrive, then shutdown.

Restarted… moment of truth…and…
Woo! Its working :slight_smile:

3 Likes