I suspect that this was triggered by a malicious act, because I don’t think the cache invalidation bug would be likely to strike by accident, but that’s just a guess on my part. Others (especially Zcash core team engineers) might know better than me about that. In any case, I don’t think it is most important to determine whether it was triggered maliciously or accidentally, but it is most important to make yourself safe against it. As Sun Tzu said “The art of war teaches us to rely not on the likelihood of the enemy’s not coming, but on our own readiness to receive him”.
I’m not aware of anyone having been robbed or harmed during this event, but if you have any concerns please feel free to reach out to me privately or to the customer support team.
The most important thing you can do to protect yourself to upgrade! Go upgrade to zcash “MagicBean” v1.0.5 (the current stable release as of this moment) right now, and then come back and read the rest of this thread.
I’ll post more in a follow-up message on this thread about some lessons-learned from this and improvements we’re going to make, but for now, go upgrade to to the latest stable release, check whether your systems are functioning correctly, and join the community chat for real-time communication about security issues.
Update: Sean and Eran tell me that this really shouldn’t be called a “chain fork”, since the old nodes (≤1.0.2) don’t follow an alternate chain, instead they just stop and refuse to follow any chain after they see the triggering block.
This also makes me think that I was probably wrong to speculate that it was triggered maliciously, but whatever…
Most miners appear to have updated to the most recent versions, so everything is functioning properly for updated nodes as far as I can tell.
I will do a more detailed analysis of the cause, whether or not there was a sustained fork, etc. as soon as possible. I believe there wasn’t a chain fork. Based on the nature of the bug, nodes that encounter the problem should just stall and fail to process new blocks. Indeed, this is how we discovered the bug in the first place.
If that is the case, simply upgrading to 1.0.5 will not be enough, since the problem is that your chainstate database is corrupted. In order to recover, you will need to upgrade to 1.0.5 and then run zcashd -reindex in order to clear out and rebuild your chainstate database.
Two network alerts were sent out just over two hours ago.
Alert 1000 notified users of 1.0.0-1.0.2 to upgrade. The alert also put their node into RPC safe mode. Safe mode is where RPC commands related to creating and sending transactions are disabled.
Alert 1001 notified users of 1.0.3 to upgrade. This alert does not put the node into safe mode.
I have two transactions from suprnova to my wallet at bittrex that haven’t shown up. I’m guessing it’s the network problem. Not a lot of money but will they go through?
Mea culpa [*]. I shouldn’t have used the F word, and I shouldn’t have speculated that it was malicious (even though I flagged what I said as speculation when I said it). False alarm.
We’re going to set up a more structured process for the steps to take in cases of possible or alleged emergency.
One of the major pieces I want to have for our new improved process is a General Chain Fork Detector:
The next time someone says to me “I see this or that weird thing on my Zcash node.”, I want to just hit reload on “https://chains.z.cash” or “https://chains.zcha.in”, and look at the red/green/yellow light on there, and if it is green, I can just tell them “Nothing unusual is detected by the nodes connected to this node, so whatever you’re seeing must be local to your node.”.
