Somebody already has a GPU rig?

From about 6:15 am EST to 6:45 this morning somebody added the equivalent of about 150 desktop CPUs to the network. The graph below is the difficulty’s response. The large gap is from it going back to “1” when they stopped. The high yesterday was about 700, which correlates to about 90 PCs. It might have been an addition of only 75 CPUs, with an equal likelihood that it was 300 CPUs. (BTW there is a spike up before the rise that is a gap in my collection of last night’s data.)

2 Likes

Perhaps, or someone could be playing around with setting up a AWS instance.

1 Like

Out of curiosity, have you tried comparing the number of your estimated number of PC’s and Difficulty to the values in the block explorer? https://explorer.z.cash/status

My data is about 30 seconds behind the block explorer. With my data, I just copy and paste to excel to graph and do statistics.

It looks like we got our first “attack” the past hour. 22 blocks were found in 15 minutes when the difficulty was near a low, starting at block 3521. The network was indicating about 50 computers before the attack. Since the avg difficulty was about 300, their hash rate was about (22/15*2.5)*300/4 - 50 = 225 CPU equivalents. I did not change my previous post, but 200 was my best guess for what happened this morning. So I would guess it’s the same person.

For 1.5 hours so far (as of this post), the rest of the network has had to suffer a difficulty of 800 (now 980) when the appropriate difficulty should be 200. You can go back a month on github and here to see how much text I’ve spent trying to get this fixed.

I recommended Daira mention in their engineering meeting the problems and a fix, but she said they did not have time. [edit: she was very polite in saying she’ll try to bring it up again and shaw says str4d is looking into it] The red bar below is how my recommendation would have reacted (red) verses the digicash v3 difficulty algorithm that Zcash is using (blue). You can see the attacker is heavily penalized almost immediately by my recommendation, and it comes back down quickly, not forcing regular miners to pay so much more. The fix is the simplest and most mathematically correct:
next difficulty = avg of past 8 difficulties x 150 seconds / average of past 8 solve times. or in the lingo of the code, next D = avg past 8 diff x TargetTimespan (for 8) / nActualTimespan (for 8)

It should include orphaned blocks in the 8 because they are a measure of the attacker’s power. That will set back the coin release schedule by that many orphaned blocks, but that’s ok if there are not a lot of orphans. My biggest beef is how mathematically ugly and engineeringly stupid difficulty algorithms coins are using are. It’s like they were designed to be attacked. Zcash of course is not stupid, but they are simply copying what previous coins have done.

1 Like

Starting at block 3521, someone “mined” 15 blocks in 3.5 minutes at an average difficulty of 200, indicating a hash rate of about 480 CPU equivalents (the block explorer gives better data that allows a better estimate than 200). Either that or the testnet was hacked. It’s an “attack” because the attacker gets blocks at D=200, but for 1 or 2 hours afterwards, everyone else has to suffer over D=800. Blocks took twice as long to solve for the following two hours. This usually occurs when mining pools jump around to different coins, so they do not have a “down time”, but it may also occur with AWS EC2.

Below shows the difficulty verses block number from 2:30 PM EST to 6 PM yesterday in blue. My proposed fix to the difficulty is in red. Instead of D=200, the attacker would have had to suffer an average of D=2000 if my algorithm would have been in place and the orphaned blocks included. At 480 CPUs, the difficulty needed to be set to 3000, so the benefit to the attacker and cost to everyone else is nearly negligible. With some statistical refinement, a > 10x attack like this could be detected in 4 blocks, completely eliminating any profit motive and keeping blocks on time better.

The block times below show better what the attacker gets away with and the cost to everyone else. “10” on the scale is one 2.5 minute interval. My method (which can be improved) in red takes only 8 blocks (as expected) to correctly adjust the difficulty to make the attacker wait a longer time to find a solution. Where the blue lines are up to 20 are where blocks were coming in half as fast as they are supposed to and everyone else is having to suffer a higher difficulty for two hours afterwards. The x-axis is again per block instead of per time.

1 Like

@zawy I have moved your post to this existing topic, there is no need to create a new one. I PM’d @str4d yesterday and he is aware of this topic and the security issue , and is working on it.

1 Like

To make it make it a lot more responsive while keeping the good aspects, the following should work. I’ll test it on real data later.

My averaging window “8” in the above can be more dynamic in the following manner. This is finding the shortest time scale for averaging that is statistically significant. It should meet the important criteria: 1) stay on time 2) do not allow cheating. Item 1) means no swinging around too much and sticking to the long term average. Item 2) means reacting quickly but having a long memory.

# Perl
# N= number of past blocks in the above time & diff averages

for ($N=3; $N<19; $N++) {
    if ($ActualTimespan[N] > $TargetTimespan[N]*(1 + 1.5/SQRT(N) ) || 
        $ActualTimespan[N] < $TargetTimespan[N]*(1 - 1.5/SQRT(N) ) ) {
           last $N;  
   }
 }

The + /- 1.5/SQRT(N) is good approximation to the Poisson distribution (with 1.5% error between 3 and 18) that will cause this code to finish on any particular N’s time scale 20% of the time if the hash rate is stable. If it takes 3 blocks more than 14 minutes or less than 1.3 minutes the difficulty will change. This will occur by random chance in a stable network once every 75 minutes. N=18 will be reached 0.9^(18-3) = 20% of the time if the network is stable.

In short, you can cheat only about 1.5 blocks every 3 hours.

The above integers may be modified to other real numbers and the (1+k*SQRT(N)) function modified to fit the goal. If I do not receive an offer for the above algorithm and equation of the previous post in 3 weeks from 9/1/2016 it falls under CC license, Attribution-ShareAlike 4.0 International.

1 Like

If you are concerned about copyright, I suggest removing this information and contacting the Zcash team directly. This is a public forum, anyone can see.

2 Likes

I’m not really concerned about copyright because I have no real hope for it. If Zcash and no one else wants to own it, then so be it. Besides, every time I’ve check, every idea I have is already patented.

The statistical adjustment worked really well. Here is an additional adjustment: Let N go from 3 to 8. Change the 1.5 to 1.3x0.9^(N-3) so that it is less restrictive in general. This is the best I can do. The 0.9^(N-3) factor gives subsequent intermediate averaging periods a better chance of being selected, as each N is a filter blocking the probability of the next N being selected, but it’s not as pure as equal probability. This additional adjustment is the first step away from pure theory, but it is half way between 2 competing ideas. This data set indicated that half way between them was optimal, but it is not crucial. You can see it is still only a refinement to the simple average of 8.

Here is the time to solve and difficulty again around the attack period, comparing this adjustment over the prior simple method of averaging the past 8 blocks. The max is not at 3k because I changed how I interpret block arrival time. On my PC 4 of the solves came in 15 seconds.

You can see the adjustment is not different from the simple “average of 8” equation during stable times but it is more reactive against the attacker and keeps block solve time on track better. The region of red not overlapping blue is showing it reacted to the attacker faster. The regions of blue are where honest miners were penalized less. They are not equal area because the attack was stopped in its tracks so that there was less need to “reimburse” honest miners.

First is solve time improvement, second is difficulty, around the attack period.

where did you get that pic? is there any website to see the Zcash difficulty’s response ?

He is generating the images from his own proprietary algorithms

Here is some code to record difficulty. I posted it a month ago, but this is a better version. I copy and past this output to a spreadsheet to get the difficulty chart. The recorded time to get a block is as it arrives to your node which will have some error. It can have up to an additional 10 seconds of error. The official data can be found at https://explorer.testnet.z.cash Maybe there’s some way to get the data in a chartable format from there.

When difficulty says “1” it appears to be the previous difficulty, not actually 1. For example, it takes the same amount of time to solve d=1400 as d=1 when it is switching back and forth between the two. I never could get the devs to comment on that except to say going back to 1 will not exist in version 1.0.

#!/bin/bash
    # WARNING: this may slow your miner
    # Logs time, "apparent" secs to get block, block num, and difficulty
    # Execute this file log.sh with "bash log.sh". 
    # Assumes zcash folder is in home folder.
    # Output is to terminal and to log.txt   
    echo "Time, seconds, blocknum, difficulty" | tee -a log.txt
    c=1
    prevblock=$(~/zcash/src/zcash-cli getblockcount)
    prevseconds=$(date +"%s") # seconds since 1970
    until [ $c -gt 5760 ]; do # do for 10 days 
        block=$(~/zcash/src/zcash-cli getblockcount)
        if [ $block != $prevblock ]; then
            prevblock=$block
            let c=c+1
            mytime=$(date +"%T")
            myseconds=$(date +"%s") # seconds since 1970
            let myseconds=myseconds-prevseconds # seconds since 1970
            prevseconds=$(date +"%s")
            difficulty=$(~/zcash/src/zcash-cli getdifficulty)
            difficulty=${difficulty:0:5} 
            echo "$mytime $myseconds $block $difficulty" | tee -a log.txt
        fi
        sleep 10 # seconds
    done

@zawy select difficulty adjustment algorithm · Issue #147 · zcash/zcash · GitHub

The issue with reporting the difficulty as 1 in the case where minimum-difficulty blocks are allowed on testnet was resolved in Separate concepts of block difficulty and network difficulty in RPC by str4d · Pull Request #1308 · zcash/zcash · GitHub , which will be in beta 1.

Yep, so for the purposes of your graphing @zawy, you can consider "1"s to be equal to the previous non-min difficulty (it’s not exactly that, it’s the difficulty algorithm adjustment applied to the last non-min difficulty).

1 Like

This one is pretty darn good. The charts below speak for it.

# Perl
# zawy difficulty algorithm 0.1.

$N = 3 if abs( $ActualTimespan[3] / $TargetTimespan[3] -1 ) > 1.3 / sqrt(3);
$difficulty = $SumPastDifficulties[$N] * $TargetInterval / $ActualTimespan[$N];
$N++ if $N < 25;

The first chart is today for this algorithm (red) compared to the current v3 algorithm (blue). The spike is where someone got 5 blocks in 1 minute and then stopped. If they’re going to do that, I can make it begin at 2 blocks instead of 3. With this algorithm he would have had to suffer D=1000 on his last 2 blocks and other people had to suffer only 1 block at D= 1000.

The next chart shows this algorithm (red) verses the v3 algorithm in terms of how long each block should take. The single spike was the single block honest miners had to suffer when the attack suddenly withdrew. 10 on this scale = the 2.5 interval. Notice how often v3 takes 7.5 minutes, but this one rarely does.

The next chart is the simple N=8 average compared to this one for difficulty. For times they seemed to be similar.

The slow rises and a few sudden drops in the red (new algo) look enough to be suspicious as if there is room for improvement. It could be powerful miners shutting off.

The following is the previous attack. This new algorithm shows there was a second leg to the “attack”, the lower plateau.

There is no doubt that this algorithm is faster, doesn’t overshoot, and is generally better, but I need to point out something that makes v3 seem worse than it is. The long periods of staying high when it is supposed to drop faster than it rises are because it is getting stuck on “1”, unable to drop 16% for that block, having to wait. At least half of the excess blue area during the attack above is because of that.

Better against attacks, but maybe a little riskier as it relies on N=2 instead of N=3. Could simply switch to $N=3 and make 2.5 → 2.2 and 0.18 → 0.2 in order to reduce the drops in v0.1

# Perl
# zawy difficulty algorithm v0.11
# More aggressive against attacks. Smoother curve. Otherwise same as v0.1

$N=2 if $ActualTimespan[2] / $TargetTimespan[2] > 2.5;
$N=2 if $ActualTimespan[2] / $TargetTimespan[2] < 0.18;
$difficulty = $SumPastDifficulties[$N] * $TargetInterval / $ActualTimespan[$N]; 
$N++ if $N < 25; 

Images below are 2 day’s. Middle spike in first image could have been chance instead of attack. 2nd image shows how Zcash and Digish v3 algos differ.