Recent chain fork (not actually a chain fork)

General
#1

Folks, apparently there was a chain fork within the last 24 hours. Thanks to Wang Chun of F2Pool for reporting it.

The Zcash core developers — notably @str4d — reproduced the problem and confirmed that it is an instance of the cache invalidation bug from zcashd (“MagicBean”) v1.0.2.

I suspect that this was triggered by a malicious act, because I don’t think the cache invalidation bug would be likely to strike by accident, but that’s just a guess on my part. Others (especially Zcash core team engineers) might know better than me about that. In any case, I don’t think it is most important to determine whether it was triggered maliciously or accidentally, but it is most important to make yourself safe against it. As Sun Tzu said “The art of war teaches us to rely not on the likelihood of the enemy’s not coming, but on our own readiness to receive him”.

I’m not aware of anyone having been robbed or harmed during this event, but if you have any concerns please feel free to reach out to me privately or to the customer support team.

The most important thing you can do to protect yourself to upgrade! Go upgrade to zcash “MagicBean” v1.0.5 (the current stable release as of this moment) right now, and then come back and read the rest of this thread.

I’ll post more in a follow-up message on this thread about some lessons-learned from this and improvements we’re going to make, but for now, go upgrade to to the latest stable release, check whether your systems are functioning correctly, and join the community chat for real-time communication about security issues.

8 Likes
February 10, 2017 - Dev update
#3

Note: the bug in question was found and fixed in zcash “MagicBean” v1.0.3, released November 17, 2016: https://z.cash/blog/new-release-1-0-3.html

The only people at risk from this issue are those who are still running an older version than that.

4 Likes
#4

Update: Sean and Eran tell me that this really shouldn't be called a "chain fork", since the old nodes (≤1.0.2) don't follow an alternate chain, instead they just stop and refuse to follow any chain after they see the triggering block.

This also makes me think that I was probably wrong to speculate that it was triggered maliciously, but whatever…

3 Likes
#6

Most miners appear to have updated to the most recent versions, so everything is functioning properly for updated nodes as far as I can tell.

I will do a more detailed analysis of the cause, whether or not there was a sustained fork, etc. as soon as possible. I believe there wasn't a chain fork. Based on the nature of the bug, nodes that encounter the problem should just stall and fail to process new blocks. Indeed, this is how we discovered the bug in the first place.

In the mean time, we have updated the security page on our website: https://z.cash/support/security.html

We are likely to send an alert out soon to nodes that are vulnerable to these cache bugs.

4 Likes
#8

If you were running ≤1.0.2, you will likely get stuck on block 59372, and will repeatedly see a message like this in your ~/.zcash/debug.log file:

ProcessMessages(headers, 238081 bytes) FAILED peer=6

If that is the case, simply upgrading to 1.0.5 will not be enough, since the problem is that your chainstate database is corrupted. In order to recover, you will need to upgrade to 1.0.5 and then run zcashd -reindex in order to clear out and rebuild your chainstate database.

1 Like
#9

For all hurrying to update nodes, remember to retain a backup of ~/.zcash directory.

2 Likes
#10

Two network alerts were sent out just over two hours ago.

Alert 1000 notified users of 1.0.0-1.0.2 to upgrade. The alert also put their node into RPC safe mode. Safe mode is where RPC commands related to creating and sending transactions are disabled.

Alert 1001 notified users of 1.0.3 to upgrade. This alert does not put the node into safe mode.

1 Like
#11

I have two transactions from suprnova to my wallet at bittrex that haven't shown up. I'm guessing it's the network problem. Not a lot of money but will they go through?

#12

What the hell ?

What could be the consequence ?

#13

Mea culpa [*]. I shouldn't have used the F word, and I shouldn't have speculated that it was malicious (even though I flagged what I said as speculation when I said it). False alarm.

We're going to set up a more structured process for the steps to take in cases of possible or alleged emergency.

[*] “Mea culpa” is Latin for “my bad”.

4 Likes
#14

One of the major pieces I want to have for our new improved process is a General Chain Fork Detector:

The next time someone says to me "I see this or that weird thing on my Zcash node.", I want to just hit reload on "https://chains.z.cash" or "https://chains.zcha.in", and look at the red/green/yellow light on there, and if it is green, I can just tell them "Nothing unusual is detected by the nodes connected to this node, so whatever you're seeing must be local to your node.".

4 Likes
#15

LOL, I like the "whatever" :slight_smile:

so this wont be a worry for people using other wallets like Ledger Nano S?
Thanks

1 Like
#16

Nope, there is no reason for anyone (other than the operator of that one out-of-date node) to worry about this issue. Sorry for the false alarm.