So a while back I upgraded my old 10-GPU system to newer hardware. It was running a Supermicro X9DRX+-F motherboard which provided 10x PCIe 3.0 x8 slots. It was great for PCIe connectivity and a great project to get it all working. but i wanted to update it to more modern hardware with faster and more power efficient hardware.

So i upgraded it to an AMD EPYC platform, but without getting into an incredibly expensive and proprietary server ecosystem, I would be stuck with only 7x PCIe slots as is with the standard ATX spec. So that's what I did.

Asrock Rack EPYCD8 motherboard
AMD EPYC (Rome) 7402P 24-core/48-thread CPU
64GB (4x16GB) DDR4 3200MHz Registered ECC
8x EVGA RTX 2070 GPUs

So how to put all 8 GPUs on only 7 slots? Bifurcation! This motherboard supports bifurcation natively on all four(4) x16 slots, but I would need a riser to actually split the slot into 2 slots to accept two GPUs. did some digging and came across a user C_Payne on [H]ardforum, and eventually found his webstore where he sells custome risers just for this purpose.

finally got the bifurcation card and got it all setup. working great and it all went as smoothly as I could have hoped for. just plug in the riser to the cards, hook up the 8-pin power, and change the slot settings in the BIOS to x8x8. the biggest issue was waiting 6 weeks for delivery from Germany (I'm in the US), likely delayed due to COVID restrictions or policies.

this system: https://www.gpugrid.net/show_host_detail.php?hostid=543446

pics: https://imgur.com/a/LrreKks

this is the riser board that I bought: https://peine-braun.net/shop/index.php?route=product/product&path=59&product_id=81

obviously this requires a motherboard that supports PCIe bifurcation with support exposed in the BIOS. you can't do this on most consumer motherboards, but it seems to be well supported on Asrock motherboards (even consumer) and some higher end HEDT boards. and very well supported on Server and Workstation motherboards. you could in theory use a riser board with a PLX chip to do the same thing without needing bifurcation support on the mb (C_Payne even sells some), but they are MUCH more expensive.

the advantage here is that it's cheaper and uses less power than a PLX based solution, and i still get PCIe 3.0 x8 bandwidth to each GPU, so no slowdown with GPUGRID. i think i should be able to get all the way to 11 GPUs if i wanted to. but it'll likely stay at 8 for the foreseeable future due to power constraints at this location.
____________

Impressive!

Gives new meaning to gathering around the fire (GPUs) at christmas...

My PC room is 90 degrees (F) at christmas (summer). What room temps do you see ambient?

it varies. i usually leave the window to the room open a bit to let some fresh air in, and its below freezing at night now. i dont think the room ever gets over 75-80F, with the window closed. and much much cooler with it open. probably down to the 50s or 60sF.

the GPUs stay around 50-60C though while running GPUGRID

I have all GPUs power limited to 150W each, with an overclock to offset most of what i lost from power limiting. the whole system pulls 1500-1600W running full tilt.
____________

Asrock Rack EPYCD8 motherboard

Supermicro has very little ATX sized motherboards. Mostly custom footprint solutions for 1U or 2U servers.

Asrock Rack OTOH seems to specialize in common motherboard form factors that fit in standard PC cases very easily.

when I was running the supermicro board. the only thing I cared about at the time was PCIe connectivity.

I wasn't too interested in CPU processing as I wasn't using it. but now, since I've started doing Universe@home on the CPU, i wanted something more capable and power efficient.

the platform change moved to a much more power efficient CPU (and much better IPC, more cores), more power efficient and faster RAM, and the motherboard likely uses less power also.

the only thing that's not better is my wallet lol.
____________

Supermicro has very little ATX sized motherboards. Mostly custom footprint solutions for 1U or 2U servers.

Asrock Rack OTOH seems to specialize in common motherboard form factors that fit in standard PC cases very easily.

When I've looked at the Supermicro mobo page and used the filters, it produced very little ATX or EATX form factor boards. I spend quite a bit of time in fact googling and searching on the SM site. Not many results.

But then I know I have terrible insight into keyword searches that produce nothing for me typically.

SM doesnt seem to have external PCIe power input on any of their boards, where that seems to be a common feature on the Asrock boards. Very important if you're going to have a lot of cards pulling power from the MB like this.

I was interested to see how well the Bifurcation performed. So below is a capture of tasks from the system highlighted in this thread: https://www.gpugrid.net/show_host_detail.php?hostid=543446

Tasks are retained by Gpugrid statistics page for 7 days. All GPUs have 7 days of tasks.
All tasks are MDAD. (ADRIA and GERARD tasks have been exhausted for a while)


The results show the performance is excellent!

check the pics. every single GPU is on a riser. and no GPU has less than PCIe 3.0 x8 bandwidth.
three (3) GPUs on a 3.0x16 link
five (5) GPUs on a 3.0x8 link

only 2 GPUs are running on a bifurcated slot. one single PCIe 3.0 x16 slot split into two (2) PCIe 3.0 x8 slots. they are sharing the slot but no bandwidth goes to waste. all other GPUs are single strung to its own slot on the motherboard via riser ribbon cable.

I'll have to look closer at which cards are where, because the way the nvidia driver enumerates the cards isnt the same as the way BOINC enumerates them. especially hard to figure out which is which when they are all identical like this.

it could be down to WU variation, as well as clock speed variation. I have them all overclocked with the same offsets, but some run hotter than others or have poorer silicon quality and hence some ruin at lower clock speeds than others. i's just easier for me to apply the same exact settings to every single card rather than trying to tweak and fine tune each one.

also keep in mind, if you try to compare these cards to other RTX 2070, I have these cards all power limited to 150W, down from the 175W stock.
____________

Yes, I have studies the pics.
I should have been more specific, my mistake. I should have said the "Riser connected to the Bifurcated slot"

To match your GPUs to the Boinc Device, try this:
nvidia-smi --query-gpu=gpu_name,gpu_bus_id,vbios_version --format=csv

This will output your devices attached and their Bus ID.

slot designation can be gleaned from
dmidecode -t slot

The Bus ID and designation can then be matched to what is in you coproc_info.xml file and then will cross reference to the Boinc Device number.

Another tool to help with identifying the PCIe slots is:
lspci | grep VGA

Yes, I have studies the pics.
I should have been more specific, my mistake. I should have said the "Riser connected to the Bifurcated slot"

To match your GPUs to the Boinc Device, try this:
nvidia-smi --query-gpu=gpu_name,gpu_bus_id,vbios_version --format=csv

This will output your devices attached and their Bus ID.

slot designation can be gleaned from
dmidecode -t slot

The Bus ID and designation can then be matched to what is in you coproc_info.xml file and then will cross reference to the Boinc Device number.

Another tool to help with identifying the PCIe slots is:
lspci | grep VGA

According to the busID and then matching busID to the physical slots by manipulating fan speeds, the 2 GPUs on the bifurcated slot should be BOINC device 3 and 4.
____________

According to the busID and then matching busID to the physical slots by manipulating fan speeds, the 2 GPUs on the bifurcated slot should be BOINC device 3 and 4.

According to the busID and then matching busID to the physical slots by manipulating fan speeds, the 2 GPUs on the bifurcated slot should be BOINC device 3 and 4.

At a guess, Asrock seem to have ordered their PCIe slots priortising the bus order with the x16 slots first then the x8 slots:
PCIe1 - dev 0
PCIe3 - dev 1
PCIe5 - dev 2
PCIe7 - dev 3 & 4
PCIe2 - dev 5
PCIe4 - dev 6
PCIe6 - dev 7

Does this line up with what you have seen?

nope, not at all. they're all over the place.

Really interested in this setup.

What ribbon risers are you using?

How can you check if your motherboard/BIOS supports Bifurcation? While I could easily just get the same board you have, but I am trying to see how cheap I can build one of these using other platforms such as the Intel x99 platform since those boards can be had for much cheaper and CPUs are much cheaper than anything EPYC as well. I don't care about high core count on the CPU, just as long as it's got at least 8 cores so it can keep the GPUs busy.

Really interested in this setup.

What ribbon risers are you using?

How can you check if your motherboard/BIOS supports Bifurcation? While I could easily just get the same board you have, but I am trying to see how cheap I can build one of these using other platforms such as the Intel x99 platform since those boards can be had for much cheaper and CPUs are much cheaper than anything EPYC as well. I don't care about high core count on the CPU, just as long as it's got at least 8 cores so it can keep the GPUs busy.

I said 8 cores, not 8 threads. Thanks.

Really wish their was an easier way to determine if the board/bios supported bifurcation or not. Think I may just end up getting the same board you have, but going with the 7251 8-core EPYC CPU to help keep the costs low.

Thanks for replying.