Some users are experiencing performance problems due to increased load on the network. We’re investigating and so far there is no evidence of malice, just increased load. If you are experiencing problems please join us on the #zcash channel on the community chat (https://chat.zcashcommunity.com/channel/zcash).
Hopefully the issue with signatures on taddrs that was inherited from bitcoin that’s causing the poor performance has a fix that’s not too extensive.
On May 1 a user started generating transactions with many inputs (presumably “sweeping” up Zcash that was stored in many different addresses of theirs). These transactions have increased the computational and network load, and this has caused problems for some users. Reported problems include Zcash nodes using more RAM (potentially exceeding the RAM available and crashing), transactions taking longer to confirm, and a higher orphan rate causing losses to miners.
We’ve investigated and determined that there’s no reason to think anyone is behaving maliciously, and there’s no evidence that any security vulnerability exists.
We’re adding additional monitoring to help us understand how the live network is behaving (orphan rates, RAM usage of Zcash nodes, transaction time-to-confirm, etc.). We’re also adding more benchmarks which exercise this code inside our Continuous Integration system so that we can optimize the Zcash software to handle this load better.
We have the following recommendations for users:
• Provision 8 GB of RAM for Zcash full nodes.
• Take into account that calls to getblocktemplate may take up to 15 seconds to complete when the blockchain is under load.
• Don’t treat a transaction as final until the transaction has 10 confirmations (which typically takes about half an hour after the transaction is posted).
• If you experience any difficulties, either with the performance issues described here or anything else “weird” or problematic, contact us! Join the Zcash community chat (https://chat.zcashcommunity.com) and ask for support.
6 Likes
I can confirm that post.
Over the last 48h I see largely increased RAM usage on suprnova but no issues so far, mining is working nicely and we’re finding blocks. The blockchain is also moving fine according to the block explorer however slower nodes might have issues with longer gbt calls.
2 Likes
Mining on Nanopool and no balance is being generated despite hashing at full tilt. Is this a result of these network issues? What should I do in the meantime?
1 Like
Update: this issue increased orphan rates for miners, and some miners started throttling blocksizes or excluding transactions from blocks approximately 2017-05-04_08:00Z, and this caused a backlog of transactions not getting mined.
We’ve written a simple patch against 1.0.8-1 which miners could use if they want. It excludes transactions with > 100 inputs from being mined.
We’re continuing to investigate and will keep you updated.
3 Likes
Okay, folks, we’re going out of “incident response mode”. There is no active security threat and the network is currently stable. We will address the underlying engineering issues in our normal (non-incident-response-mode) engineering process. The ZcashCo employees who were tasked with focusing on this situation are now to go back to their regular tasks. I’ll be following up with miners, exchanges, etc. to try to make sure the network stays stable in the meantime.
If you see anything (a) “weird”, (b) scary, or (c) problematic for your usage then let us know! 
Thanks for your support. 
2 Likes
jameshilliard commented 3 days ago
The next step on this issue is diagnosis, experimentation, and benchmarking. Why does this issue not happen in Bitcoin?
For some historical context 0.11.x bitcoind had major performance issues(it couldn’t properly handle mining/propagation of 1MB blocks without the relay network and had some pretty severe performance degradation with mempools larger than a few megabytes) these were fairly well known issues at the time, and these issues were largely resolved with the 0.12 bitcoin core release(diagnosis, experimentation, benchmarking and of course writing the code to resolve them was done).
When I was operating a pool with 0.11.x I had to do a lot of mostly manual mempool monitoring and limiting to prevent GBT call times from being excessive. The performance issues were not due to a single cause but multiple factors and required major rewrites of a number of internal bitcoin core components to resolve.
Since zcash is a fork of 0.11.2 I would expect it to have those same issues bitcoin had back then.
You may want to rebase on top of bitcoin 0.14.1 since it has many performance improvements since 0.11.2.
2 Likes
Cherry picking of bitcoin 0.12.x commits is in progress…but 0.13 and up have changes that make merging them in a lot more difficult (I’m mostly looking at you, SegWit)
1 Like