Physically locating GPU Servers at Massive Scale

I’m doing some cost modeling of what it’d look like to significantly scale up my mining operation. I’m at a handful of rigs now, and it is clear that if I continue to invest in mining equipement, I will very soon reach a scale that is not safe or economical to host at home and/or I will run out of patience for my wife :-).

One of the things I’m considering is where to host such an operation. Looking at office space around the area (I’m in Colorado USA), there is very little that would meet the power and cooling requirements of a 30+ rig system. ( Depending upon the card I choose, I’d be looking at a continual 25 kw load at 30 rigs. )

Has anyone looked into this, and if so, what did you find? How did you approach the problem of moving to a pro space?

Did you find it was more cost effective to colocate at a professional-level data center, or did you simply rent office space and work out of that?

If you colocating, how did you deal with the fact that your servers are not standard “blades” that fit into a rack in most data centers?

Thanks,
Owocki3

what is considered a “massive scale”?

Any residential 200 Amp service can handle 30 rigs easily if you run them on 240V. Even 125 Amp service will work if you dedicate the entire house to mining (i.e. no water heater, stove or anything else). You can always look for a house with a 400 Amp installation to scale up. I wouldn’t worry about it unless you have enough rigs to qualify as “large demand” (> 350kW).

1 Like

I wouldn’t worry about it unless you have enough rigs to qualify as “large demand” (> 350kW).

Wow, in my modeling spreadsheet, I have that 100 rigs at 800 watts a rig would be 80 kw. What do you mean “large demand”, is that a electricity company term?

Any residential 200 Amp service can handle 30 rigs easily if you run them on 240V.

Yes, but this assumes I can put rigs in any room of my house without my wife taking notice. And that I can cool them safely in each area of the house… :slight_smile:

Right now, I’m modeling 30 rig, 100 rig, and 200 rig scenarios. Out of curiousity, what scale are you mining at?

Wherever you choose to relocate to, our recommendation is that you focus on redundancy and safety.

Sound advice. Safety first, especially when dealing with tens of kilowatts and massive heat.

We have a friend who scaled up his mining operation and went on holiday. While away for a week his cooling failed.

Wow, what a horror story! Did (s)he lose it all? Did (s)he have insurance?

Professional setups take this into account.

Curious if you would recommend going with a data center colocation, or do you prefer to roll your own?

Yes, you get a pretty big discount from the power company for being a large demand customer. My plan is to eventually move everything over to Eastern Washington where power is super cheap.

May I ask what kind of discount you get and how “large demand” you are? You can DM me if you don’t feel like sharing publicly.

I was in a very similar situation and ended up moving my rigs to Great North Data which is a dedicated mining data centre.

The first issue you will have with data centres is that only a very few will take custom GPU rigs. You could find some special rack mounted cases which would give you a lot more flexibility in terms of data centre but would vastly increase your build cost.

There is a reason most data centres don’t take custom GPU rigs and that’s because they are in the main much more difficult to maintain. As such unless you are a large customer (north of 50 kW) you will probably pay a small premium too.

My experience with remote hosting of GPU rigs is it is not worth it. Whilst Great North Data have been amazing in their service and support and I would highly recommend them I have had issues with hardware (caused by transporting them) and software crashes from Zcash mining software that require physical rebooting all of which cause downtime.

My ideal scenario would be to be physically close to the machines to make it easier to solve some of the more major issues. If there is a data centre close by to you that accepts your rigs then that would likely be a good solution. I considered other ‘office type’ space but the power costs here make that a non-option for me and there are obviously other benefits of being in a data centre (tech support, redundant power, cooling, internet connection etc…)

Edited to add I was basing this on your original post of circa 30 rigs in which case the above applies. If you are looking at 200 rigs then that’s a very different beast in my opinion (and also a lot of setup!)

Thanks, Ill have to check them out.

Mind laying out some of those considerations, just off the top of your head? I’m meeting with my financial guy tomorow, and we’re primarily building a cost model for a 30-50 rig operation, but will probably very shortly discuss what a ‘shoot the moon’ operation would look like. Would be good to have your input to validate some of our assumptions here.

I’m noticing that, across the board, ‘industrial’ electricity rates are almost half the cost of residential electricity rates. Does anyone have experience with actually getting a power company to get you industrial rates?

You can safely assume that not all your hardware will work all of the time :wink: which is very significant when mining margins are slim. Also factor in your build out costs which will likely be substantial on a large number of rigs - they take time to setup and the debugging of machines is a real time hog finding dodgy risers etc… on a scale of hundreds of rigs this is probably a full time job.

Great North data are a Canadian company so if you were considering them (and you should as they are excellent) you’d have to factor in the cross border issues of getting your equipment there.

right now i just have one pc with 2 GPUs on it (a R9 390x and an RX470) … i just started mining less than a couple of months ago … but been thinking of building a farm … would start with 10 rigs with 6 GPUs and upgrade them in 10s every month or so maybe … looking at building 50 rigs if i can find funding :))

Hi @garethtdavies can you recommend any other companies along with Great North Data?

I actually don’t know of any other companies that will allow you to host your own custom rigs (for good reason as remote GPU mining is fraught with issues).

There are options if you don’t have the rigs and then there are some US hosts that will build and host the rigs e.g. https://www.oregonmines.com or BlockTree Technologies. Having said that I imagine they would have issues sourcing rigs at the moment.

Oh ok thank you. I’ve reached out to several that I’ve found on this forum but it appears many are completely booked or unresponsive.

If anyone is a host, or knows of one accepting rigs, please let me know.

Please don’t hijack threads this is clearly a thread about hosting physical miners, it says so in the title, and not cloud services.

It’s because you are spamming. Are you going to post in every thread about mining (which form the majority) your cloud services? If everyone did what you do the forum would be nothing but links to cloud services.

What issues? Rigs just run, they restart themselves if they detect a problem, they provide status reports to a central monitoring system, and generally just let me know when they cant fix something themselves (like a failed fan on a GPU).

I don’t disagree when things work it’s easy :slight_smile: The issue for a facility if they allow custom GPU rigs suddenly they are dealing with disparate hardware which does take time to debug if things go wrong and it soon becomes cost prohibitive. There’s a reason nearly all hosts won’t take custom GPU rigs, don’t just take my word for it!