Zcash P2Pool Mining Development

Rather than working towards a centralized mining solution, I think we should focus our early efforts on a decentralized solution: p2pool.

Zcashd, p2pool, and a standalone miner will all run on the same machine (no third party server needed). Not only does this prevent centralization of mining, but every miner will also be a full node.

To achieve this we need to do the following:

Fork/Modify P2Pool

P2Pool should have it’s ports modified (so the p2p mining network doesn’t interfere with the bitcoin p2pool mining network).

Also, I’m not sure if P2Pool verifies blocks/shares itself or whether it leverages zcashd to do that. If it verifies the validity of blocks itself then we’ll also need to modify it so it can verify zcash blocks.

Create a Standalone Zcash Miner

Zcashd has an internal miner (see line 400 here). It should be modified so that it gets it’s block headers from P2Pool, instead of building them from scratch on it’s own. Moreover, it needs to be able to communicate with p2pool using the stratum protocol.

There are already a few standalone miners (for bitcoin and various alts) that use the stratum protocol (and are therefore compatible with p2pool). BFGminer and cgminer are two that stand out. It may be tempting to fork one of those projects and swap out their native hashing algorithm with Equihash. But I wouldn’t recommend that.

Existing standalone miners come with a LOT of code we don’t need. Much of the code complexity of bitcoin miners comes from the fact that SHA256 hashing is so fast that miners quickly exhaust their nonce space. They then have to tweak things and recompute their block headers often. We don’t have this problem in zcash. Equihash takes dozens of seconds to complete a single hash. It is unlikely any zcash miner will ever exhaust their nonce space and need to tweak/recompute the block headers. So a standalone zcash miner can omit much of the code complexity found in the existing miners.

Also, the Stratum protocol isn’t very complicated. This is why I think it makes sense to write a standalone miner “from scratch”, starting with the native zcash internal miner.

##Moving Forward
P2Pool is written primarily in Python. The native zcashd miner is written in C++.
I’m a python developer and feel comfortable jumping into the P2Pool code to modify it as needed.

On the other hand, I have next to zero C++ experience, so it would be great if someone else could volunteer to help with the standalone miner.

Please let me know if you are willing to help.

5 Likes

I'm not sure if this is relevant but I was looking at the Zcash github issues page and noticed this thread:

which linked to this:

https://github.com/str4d/zcash/tree/standalone-miner

Also, perhaps those of us who are interested should post on our social media to put the word out among our contacts.

I'll happily accept PRs to add getblocktemplate or Stratum support to the standalone miner (as long as it can still run on its own without any real block info for testing purposes). And if it gets to a usable state, I'll see about potentially having it merged into the main repo.

2 Likes

Not related to pool development:

The issue jl777 described on June 25 that no one replied to is serious. He's indicating Equihash is biased strongly against slow machines. But his argument is based on the assumption that there is a hard minimum solve time that can be a large percentage of the 2.5 minute block time. Is that true? Is solve time = minimum + difficulty x slowness or is it = difficulty x (minimum + slowness). If it's the 2nd case then typical home PCs and laptops would not be able to mine after enough fast machines are online.

Right now my slow machine is showing 10 times more coins gained tonight than my fast machine. I guess that means they're all stale.

jl777:

    if you have a relatively slow system, then your odds of

finding a block goes down nonlinearly as compared to benchmark speed
differences

there are some miners running with 40 seconds benchmark times, mine
runs at 80 seconds, with a block being ~150 seconds, then I usually get 1
shot at winning a block. the 40 second machine gets 3 or even 4. so
instead of 2x, it can get 3x to 4x the blocks on a 2x faster system.

this effect is independent of interrupting mining threads, since the
mining has to restart at the beginning of each new block. As an example
if your system takes on average longer than the blocktime you simply
never get a chance to submit a block and so never have a chance to mine
the block, ie 100% stale.

but it gets worse...

If current diff hasnt adjusted yet, then the faster machines are
simply finding blocks faster than slow systems can generate even a
single set of solutions.

1 Like

I just had a brief look at the P2Pool source; it seems already to be designed to support multiple networks and proof-of-work functions (see the p2pool/bitcoin/networks and p2pool/networks directories).

One advantage of each Equihash solution run being expensive, is that the overhead of using IPC to get a solution, or even creating a separate process for each run, is likely to be negligable in comparison to the solution cost. (Unfortunately the Bitcoin RPC interface is blocking, so that wouldn't suffice unless we add asynchronous calls, which we may do anyway.)

2 Likes

The issue jl777 raised is being taken seriously; @str4d and I have been working on it recently. Remember that development discussions are all on github; not this forum. It's easy for the developers to miss particular posts here, and that doesn't imply we haven't responded to the issue elsewhere.

In this case, the main github issue about stale blocks was fixed yesterday: https://github.com/zcash/zcash/issues/1055 , and there are a few other issues open concerning remaining optimisation and parameter choices (search for Equihash on the issue tracker).

4 Likes

Daira, is your second paragraph saying time to solve = minimum + diff x slowness where minimum is small for slow machines compared to 2.5 minutes? (I made a mistake in previous post that I corrected, I got the two cases backwards)

@zawy please try to keep on topic in threads. If you have new queries, feel free to open a new thread :smiley:

2 Likes

The analysis on https://github.com/zcash/zcash/issues/764 indicates that provided the Equihash solution run time is less than about a quarter of the target block time, the advantage of an adversary who has a miner that is twice as fast remains small.

This analysis doesn't yet take into account block propagation and verification time, but making the solution time less than a quarter of (target block time minus pessimistic estimate of propagation+verification time) should be sufficient. We're not there yet but it's perfectly feasible.

2 Likes

Yeah, I'm hoping it will be as simple as adding a properly constructed zcash.py file in the /network folder and pointing to the correct ports.

I noticed that to get litecoin working with p2pool they had to add everything you see in the litecoin_scrypt folder. I'm not sure why those extra additions were needed for litecoin and not the other altcoins currently supported by p2pool.

I'm hoping it will be straightforward.

1 Like

That's just to support the scrypt proof-of-work.

For Zcash, the approach I'd suggest would be to have a mining process that takes block headers (hex-encoded, one per line) on stdin and spits out solutions (similarly encoded) on stdout. We can modify @str4d's standalone miner to do that. Then the POW_FUNC in the P2Pool Python code can run that miner in a subprocess.

2 Likes
2 Likes

AFAICT POW_FUNC is only used to check whether a block is under the desired target. So P2Pool doesn't actually require Equihash solver support inside it; it simply needs to be able to verify the validity of a block, which requires two parts:

  • Validating the Equihash solution (which only requires a validator, which can be written in Python)
  • Checking the block header satisfies the difficulty check (which currently uses double-SHA256)

In terms of actually supporting Equihash as a mining PoW, it looks like the standalone miner needs to support the Stratum protocol (rather confusingly, P2Pool uses getblocktemplate to fetch work from the upstream bitcoind but Stratum to serve that work (modified for its needs) to the miners).

2 Likes

As an update (since I noticed some new/renewed interest in a standalone miner in other threads): I don't have the time to work on this at the moment. A new semester started and my teaching responsibilities have taken the place of all my fun zcash playtime.

So this is wide open for anyone to work on. AFAIK nobody is currently working on this.

TODO: Build a standalone miner that works with the Stratum Protocol.

After that, everything else (getting P2Pool working and/or centralized mining up and running) is relatively easy.

1 Like

I'd be willing to discuss a bounty if it gets a serious dev hacking.

1 Like

Me too but it's going to take a lot more contributors to motivate a competent developer. Then there's the issue of a trusted party to hold contributions and assess whether or not the bounty has been met.

Given how well received the proposed GPU mining crowdfund was, where's the love for this mining initiative?

I've basically finished adding Stratum support to my standalone miner. I'm going to have a go at P2Pool next :slight_smile:

8 Likes

Hi,
is it this one:
https://github.com/str4d/zcash-pow

I'm just amazed about the support and passion that nearly all members are showing here.

very enriching community, hope this project will be fruitful for all.

@Dusares no, see https://forum.zcashcommunity.com/t/zcash-p2pool-mining-development/1051/2?u=str4d


A very good point was raised in #mining on the community slack:

[peter_zcash] Will p2pool mining even be feasible given the already low planned block time of 2.5 minutes?

We were struggling to find parameter choices that worked for Zcash itself with a 2.5-minute block interval. I wonder whether the P2Pool will run into issues given it needs to have a much shorter block interval. At the very least, the P2Pool block interval would need to be above 30s with the current miner, and that will have some discretisation issues which could maybe adversely affect the relative ratios of different miners in the P2Pool chain. But then, the incentive structure is different for a pool than for solo mining, so it's maybe not as much of a concern...