One rig with 56 gpus?

NicodemusNicodemus Member Posts: 16
Hi.

I've seen new expanders that can convert 1 PCIe 1x to 4 PCIe 1x, and another expander that converts 1 PCIe 16x to 8 PCIe 1x. Thinking about the Asrock H110 BTC+ motherboard, that have 1 PCIe 16x and 12 PCIe 1x, we can end up with 56 gpus installed on a single rig.

Theoretically, it is phisically possible. But I want to know if there may be any other problem to do this, like a limitation on management, bios, or OS related, that would not allow this huge amount of gpus on a single equipment.

Thank you!

Comments

  • headshot155headshot155 Member Posts: 118 ✭✭
    I've have a H110 BTC+ board and I had a dream of creating a 13 GPU rig, can tell you from experience it is a complete nightmare with more than 9 GPUs. Yes windows can take more than that but it starts to become much less stable, also with AMD cards wattman grinds to a halt with more than 10 gpus and becomes virtually unusable. With my 12 cards it was taking me almost an hour to simply switch from graphics to compute mode in wattman. Wattman is essential to keep electricity costs and heat low and performance high. It'd fricking awful with 56!!!

    I have ended up limiting my rigs to no more than 8 gpus using windows 10 - thats the best number where you get a cost efficient, stable rig. I've not had a great experience mixing different types of GPUs either, a batch of identical ones for each rig is best - mixing GPUs tends to degrade the hashpower of the rig as a whole for some reason.

    I cant say what it would be like with Linux or Ethos but I do know overclocking and undervolting are harder while it takes Nvidia and AMD much longer to update the drivers.
  • NicodemusNicodemus Member Posts: 16

    I've have a H110 BTC+ board and I had a dream of creating a 13 GPU rig, can tell you from experience it is a complete nightmare with more than 9 GPUs. (...)

    That's kinda scary to hear. I am building a machine with 16gpus these days. It seems I'll have quite bit of problems. =/

  • sutheksuthek Member Posts: 271 ✭✭
    edited January 11
    Short answer, for your specific H110+ board, your max is 13 cards. (due to chipset)
    I have tried more. It does not work.

    Long answer,
    You have a few things to consider.
    • Mobo chipset.
    • OS
    • Drivers
    • Electrical consumption.
    you need your motherboard to have enough chipset pathways to have more cards


    For example, I have that same H110 BTC+ board with 13 cards.
    I tried plugging in one of those 4-1 splitters with 14 cards. Bios wouldn't post. Nadda.
    I removed the 14th card but the bios was still frozen.
    I had to reset the bios to resolve the issue and get my 13 cards running again.

    Assuming you have a motherboard with an awesome chipset, the next limitation you'll have is OS.
    As mentioned above, windows sucks. How do you solve this? Don't use windows.
    I suggest going to simplemining.net and making an account. It's the best mining software I've ever found. up and running with 13 cards in less than an hour. (Linux based but you need zero experience with linux)

    And the next limitation is driver support.
    I can't recall what the max is but you can't have 56 generic AMD cards hooked up.
    I think the only cards that can be at that total is the mining specific Nvidia cards.

    Ultimately, the bigger problem is electricity fed from the city. How the heck are you going to power 56 cards? You'll need to get a breaker panel custom done with QUITE A FEW circuits so that you can power all of that.
    I think you'd need at least 200a service to your house... and you'd likely need to remove most electrical things from your house. (oven... lights... electrical outlets for misc non-mining things... disconnect them all at the breaker). Your house will look like a grow op once you're done with all the electrical.

    Even with 13 cards, I find I need to run extension cords everywhere to get things going. (so I'm in multiple 15a circuits.)
    If you want something that can be stand alone without making a mess in your house, I suggest having 8 cards per rig.
    If your house has 20a circuits, maybe you'd be ok with 12-13 card rigs.





  • NicodemusNicodemus Member Posts: 16
    Well, I already have 11 gpus and were going to buy another 5 till middle year to complete the rig. I guess I'll stay with 13 and then invest in a new rig. Maybe a 8 gpu one... Thanks for the input.
  • sutheksuthek Member Posts: 271 ✭✭
    do you have all 11 plugged into one outlet on the wall? Is it hot? I assume it's 20amp?
    You don't need to stick to 8 if you think you can get 11 running ok.
  • NicodemusNicodemus Member Posts: 16
    edited January 15
    suthek said:

    do you have all 11 plugged into one outlet on the wall? Is it hot? I assume it's 20amp?
    You don't need to stick to 8 if you think you can get 11 running ok.

    No, you misunderstood. This rig will have 13 gpus... 11 for now, and 2 more in a near future. If I'm gonna set up a completely new rig, I'll maybe consider a 8gpu one instead of a 13 gpu. It will depend on all the dificulties I'd have with this 13 gpu one. =)
  • sutheksuthek Member Posts: 271 ✭✭
    Nicodemus said:

    11 for now, and 2 more in a near future.

    I think you misunderstood my point.
    if 11 is working ok, then your next rig should be able to be 11.

    But I recommend having that plugged into a 20a circuit... or multiple circuits.

  • Luck0rLuck0r Member Posts: 19
    suthek said:

    Nicodemus said:

    11 for now, and 2 more in a near future.

    I think you misunderstood my point.
    if 11 is working ok, then your next rig should be able to be 11.

    But I recommend having that plugged into a 20a circuit... or multiple circuits.

    Depends where He lives i guess. In Europe we have 230 V and 10, 13 or 16 Amps.

    10 Amps ist usually Minimum so He would be fine with His Rig.
  • NicodemusNicodemus Member Posts: 16
    Luck0r said:

    suthek said:

    Nicodemus said:

    11 for now, and 2 more in a near future.

    I think you misunderstood my point.
    if 11 is working ok, then your next rig should be able to be 11.

    But I recommend having that plugged into a 20a circuit... or multiple circuits.

    Depends where He lives i guess. In Europe we have 230 V and 10, 13 or 16 Amps.

    10 Amps ist usually Minimum so He would be fine with His Rig.
    You are both right. To put a new 11/13 rig to work on my house, I'd probably have to change eletric cables to support this huge amount of energy. So, in the end, it will depend on some factors.

    But this is a future concern. Now I'm more into putting my first rig to work with these 11 gpus.
  • chromachroma Member Posts: 8
    edited January 15
    Good luck buying even one GPU at MSRP this month. XD Even EVGA's corporate web store can't sell me one of their own GPU products. As near as I can tell, the gaming and VR world is on hiatus for a while, "NO VACANCY," at least if you intend to build yourself a new system for 2018. That's a bummer.
  • Luck0rLuck0r Member Posts: 19
    In Europe Cards are still available, but not a lot.
  • sutheksuthek Member Posts: 271 ✭✭
    Nicodemus said:

    11 cards

    Check your breaker panel to see if you have 15a or 20a brakers.
    most common in north america is 15a.

    If it's 15a, i think 10-11 cards will be pushing it. You'd want NOTHING ELSE on that breaker.

    If it's 20a, i think you could set up a 13 card rig single mining on it.
    Both of my mining rigs are about 2050w constant draw from the outlets. (using wemo insight and kill-a-watt meters to confirm exact draw.)

    that would cook a 15a circuit.... but is just a bit under the limit of a 20a breaker.


  • NicodemusNicodemus Member Posts: 16
    suthek said:

    Nicodemus said:

    11 cards

    Check your breaker panel to see if you have 15a or 20a brakers.
    most common in north america is 15a.

    If it's 15a, i think 10-11 cards will be pushing it. You'd want NOTHING ELSE on that breaker.

    If it's 20a, i think you could set up a 13 card rig single mining on it.
    Both of my mining rigs are about 2050w constant draw from the outlets. (using wemo insight and kill-a-watt meters to confirm exact draw.)

    that would cook a 15a circuit.... but is just a bit under the limit of a 20a breaker.


    I've checked that. The room that the rig will be on have a 20 dedicated breaker. Only things that will be on the circuit as well is a couple of ceiling lamps.

    From what I've heard, the main thing is the eletric cable that supports or not the amount of amperage I'll need. I live in Brazil, and here we have 1,5mm, 2,5mm, 4mm, 6mm (and so) cables. The "base cable" for a room is always a 2,5mm, which supports 21a without it getting hot. Our voltage here is 220v, so, in theory, that cable supports 4.620w (21a x 220v). That's what I've read, same thing that an engineer told me.

    So... I guess I can support two decent rigs with that configuration, without getting the cable hot.
  • chromachroma Member Posts: 8
    edited January 16
    Luck0r said:

    In Europe Cards are still available, but not a lot.

    That's true, they do, actually. EVGA's EU store has GTX1080s, and I could buy an EVGA 1070 FTW2 for £606 from a seller on amazon.co.uk. Still, that's US$275 more than EVGA's own domestic MSRP, as it appears our currency is now worth 20% less under the Trump occupation's Treason against the west than it was a year ago and the conversion does not seem favorable at all. I have to wonder how many intended for domestic use got exported on the grey market.

    That is quite a few hours' wage to throw away over a local supply issue. For now, I'll support the Ethereum ecosystem with my GTX970 and hope that retail-priced inventory comes along in another week or so to provide an opportunity to upgrade for VR and for faster mining.
Sign In or Register to comment.