Hey guys..

So I want to build a computer that will be solely for GPUGrid computing.. That means I want to squeeze as many GPUs into the case as possible..

This is where I need help.. What motherboard should I get that can hold as many gpus as possible?

How many GPUs are we looking at here?

I'm trying to figure out what's feasible/reasonable... and I want to build this over time. I know I can get 460's for fairly reasonable prices off ebay so if I could get a whole pack of them in a case then that'd be pretty cool..

Thoughts?

HI
Well if you don't mind going down the amd bulldozer route the new Sapphire
Pure Black looks interesting,will take six single slot cards.
looks pretty too:)

So I want to build a computer that will be solely for GPUGrid computing.. That means I want to squeeze as many GPUs into the case as possible..

A standard ATX case has only 7 slots at the rear, also the motherboards have 7 PCIe slots at maximum. You should also consider to maximize performance, the motherboard should have as many PCIe x16 slots as possible, because slower PCIe slots hinder the performance of the GPU. Also, a CPU core per GPU is needed to maximize performance.

This is where I need help.. What motherboard should I get that can hold as many gpus as possible?

I would suggest the Asus P6T7 WS SuperComputer motherboard with a 6-core i7 CPU (i7-970, i7-980, i7-980x, i7-990x), as it has 7 PCIe 2.0 x16 slots. Sapphire Pure Black has only two x16, two x8 and two x4 slots (while all of them are capable of taking an x16 card, but they will run slower). When a slot 2011 mb will come out with 7 PCIe 3.0 x16 slots, it will supercede the P6T7 WS SC.

How many GPUs are we looking at here?

At maximum 7 GPUs can be put in a single motherboard, but it will need a very good water cooling (possibly 4 of them). It will also need two good 1200W power supply (as it is advised for longevity and efficiency reasons not to utilize a power supply over 50-70% of it's nominal wattage in long term).

I'm trying to figure out what's feasible/reasonable... and I want to build this over time. I know I can get 460's for fairly reasonable prices off ebay so if I could get a whole pack of them in a case then that'd be pretty cool..

For GPUGRid it's better to have a top end CC2.0 card (GTX 570, 580, 590, 470, 480), than two CC2.1 cards (as only the 2/3rd of their shaders will be used by GPUGrid). I suggest you to wait a couple of months, because nVidia is going to release a new series of GPUs (i.e. Kepler).

For GPUGrid, top cards tend to be double slot width, so well spaced out slots are desirable. A motherboard with 7 single spaced slots can support no more double spaced GPU's than a board with 4 double spaced slots!

Express lane count is also very important; the more lanes you have the more cards it can 'fully' accommodate. We know that performance drops when fewer PCIE lanes are available to the GPU. Typically the more slots used, the lower the PCIE lane availability. Often boards move from one PCIE x16 lane to two PCIE x8 lanes and then to one PCIE x8, one or two PCIE x4 slots and the fourth slot is often PCIE x1. The performance of a high end GPU in a PCIE x4 slot is significantly reduced, and would be terrible in a PCIE x1 slot. I found that even a low end GPU's performance suffered significantly in a PCIE X1 slot.

Until recently the best Motherboards have been LGA 1366, with boards for AMD CPU's being a cheaper alternative. Even now i7-980X (and similar 32nm CPU) systems fair slightly better than the i7-3960X for CPU crunching (clock for clock). So until now the only benefit of these overly expensive systems is the reduction in power, and knowing that they were somewhat future-proofed...

With AMD's latest GPU being fully PCIE3.0 compliant this has changed the picture somewhat. While we cannot use these here (yet), they offer potential. Within a few months (Feb to Apr) NVidia's GeForce 600 series will start turning up, and when they do the high end GPU's in this range will all support PCIE3.0.

Note that a PCIE3.0 X8 slot hosting a fully PCIE3.0 compliant GPU is as fast as a PCIE2.0 X16 slot. PCIE3.0 is also backward compatible with PCIE2, so existing GPU's work. IIRC 2011 boards only have 42 PCIE channels, but that is enough for one GTX 600 @ PCIE3.0 X16 and 3 GTX 600's running at PCIE3.0 X8.

I have no intention to use numerous GPU's in a single board, so I bought and presently use an i7-2600 in an MSI motherboard that has two PCIE3.0 slots. Obviously not an option if you want more than 2 GPU's. If and when both are used they drop to PCIE3.0 X8 (but again, as good as two PCIE2.0 X16).
The present card in that system is not benefiting from PCIE3.0 X16, unless it's 18.5% faster due to the overhead reduction (new PCIE3.0 algorithm) - I was just future proofing, and at the time I didn't want to spend the cash speculatively on an immature 2011 based system.

So, for future-proofing, and slightly lower running costs, you could opt for a 2011 plenum board (if you have the cash to build such a rig). If not then a less expensive 1366 based rig, with higher running costs, and if 1 or 2 high end GPU's is all you want then something based on the SB LGA1155's would do.

Anyone thinking about the forthcoming GF600 series should consider a PCIE3.0 board, and if they want an LGA2011 they should check the PCIE3.0 slot specifications. The better boards seem to be one PCIE3 X16 + three PCIE3.0 X8.

Personally, I would not consider LGA2011 until Intel's 22nm cores turn up and fall significantly in price. $1000 for a system without any GPU's is too much spent on the wrong things.

The 'Sapphire Pure Black' for AMD CPU's is only PCIE2. Not sure if there even are any PCIE3 motherboards for AMD CPU's yet?

The Asus P6T7 can only facilitate three x16 GPU's or six x8 GPU's on PCIE2.0 channels and it's single slot width, so you could only have 4 big GPU's.

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

That is an interesting question which has to be answered directly with another question: how extreme do you want to get? A couple points to consider:

- running costs are going to be very high
- it's going to be very loud
- forget a case, if possible at all, it's not helping cooling
- with water cooling you can use boards with single slot spacing
- there are flexible PCIe riser cables, which let you use boards with single slot spacing and GPUs with their default coolers
- you can use GTX590 to alleviate the "slot pressure"
- using 7+ GPUs (double GPU cards count double here) may require a custom bios from the manufacturer
- BOINC may get troubles recognizing many GPUs
- you may need 2 PSUs, probably rather 80+ Platinum than Gold

So you see, there's no straight answer to "what's possible".. except "a lot" :D

MrS

---

Scanning for our furry friends since Jan 2002

Anything over ~$2000 deserves the bespoke touch.

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Wow guys this is great information, thank you very much.

It had occured to me that the most slots coming out of a ATX case was 7 and so I assumed that would be the max, but with modern cards being 2 slots wide or even 3 in some cases I didn't know how feasible it would be to try to get all 7 working or if 7 thin cards would be as good as one fat beast..

I didn't even consider the slot speeds/lanes.. Never even occured to me..

I have a GTX 580 with a water block running in one of my current crunchers and it works nicely and I figured that I might be able to get enough of those (or something similar) to actually get 7 running cause they will fit in a single slot at that point, but hell the link Skgiven posted now I'm thinking that's not necessary..

The home brew external case looks like a fun project and exactly the kind of nerdporn I can get behind.. =) Completely nerdy and will likely cause my wife to go into fits and spasms but it looks fun to me!

The only question I would have then is with the PCIE risers; They look to be just ribbon cable that you plug into the board, then up to the card.. but they appear to only use 1/3 of the slot? How does that work?

You can get full PCIE raisers,


You could also use every other slots on the board directly (so long as you reinforce the back of the motherboard).

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Don't forget , the more GPUs , the more POWER.
I plan 2 GTX590 (this gives 4 GPUs) on an ASUS Sabertooth 990Fx and an AMD fx8150 -> Need a 1200W PSU !

---

Lubuntu 16.04.1 LTS x64

Or a second PSU.. There are nice little boards over at frozencpu that allow you to have one PSU that turns on the second one..

I plan 2 GTX590

Just in case you didn't already know: wait for Kepler, which should be just 1 - 2 months away. Either it's way better than the current ones (one certainly more efficient), or it at least drives drives the prices of current cards down. and you should be able to get some good 2nd hand cards from people who upgrade.

MrS

---

Scanning for our furry friends since Jan 2002

I plan 2 GTX590

Just in case you didn't already know: wait for Kepler, which should be just 1 - 2 months away. Either it's way better than the current ones (one certainly more efficient), or it at least drives drives the prices of current cards down. and you should be able to get some good 2nd hand cards from people who upgrade.

MrS

You are right but as I just bought a GTX590 a few days ago, I want to wait cause this is expensive (more than 700€ for a Gigabyte).
For the moment I want to mount the 2 GTX460 I already have on the Sabertooth mobo I already purchased with the PSU and CPU and DDR and watercooling for the CPU...

---

Lubuntu 16.04.1 LTS x64

You can get full PCIE raisers,


You could also use every other slots on the board directly (so long as you reinforce the back of the motherboard).

What kind of "gotchas" are there with using risers like this?

And where can I find them? I have been browsing around and going to my local electronics shops and can't find ribbon risers like this..

The obvious gotcha would be length - you can get several different lengths.

Try a search for 'PCIE riser', rather than PCIE raiser (uk en).

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Or a second PSU.. There are nice little boards over at frozencpu that allow you to have one PSU that turns on the second one..

It's very easy to turn on the second PSU, just wire the green cables of the PSUs (they will be grounded together by other cables like the PCIe power cables). You have to choose your second PSU carefully: it has to be regulating the 12V only, and it should use DC-DC converters for making 5V and 3.3V from the 12V. If it's not this type (regulating the 5V and the 12V, and making 3.3V with DC-DC converters), then it should have some load (for example a HDD) on the 5V rail.

I made some research from wikipedia, a cheap vendor and cpubenchmark. Just in case it can help to make a decisition on what Socket and CPU buy.
---------------------------------------- Socket ------ GHz ------ € ------ CPU Mark---- CPU Mark/€
Intel Core i7-950 ---------------------> 1366 ---> 3,2 -----> 317 -------> 6715 ---> 21 -------> 130 W
Intel Core i7-950 ---------------------> 1366 ---> 3,06 ---> 286 -------> 6422 ---> 22 -------> 130 W
Intel Core i5-661 ---------------------> 1156 ---> 3,33 ---> 242,95 ---> 3293 ---> 14 -------> 87 W
Intel Core i5-760 ---------------------> 1156 ---> 2,8 -----> 193,67 ---> 4597 ---> 24 -------> 95 W
Intel Core i5-660 ---------------------> 1156 ---> 3,33 ---> 184,95 ---> 3175 ---> 17 -------> 73 W
Intel Core i5-650 ---------------------> 1156 ---> 3,2 -----> 172,94 ---> 3211 ---> 19 -------> 73 W
Intel Core i3-540 ---------------------> 1156 ---> 3,06 ---> 139,15 ---> 2850 ---> 20 -------> 73 W
Intel Core 2 Duo E7600 --------------> 775 ---> 3,06 ---> 129 -------> 2119 ---> 16
Intel Core i3-560 ---------------------> 1156 ---> 3,33 ----> 121,95 ---> 3148 ---> 26 -------> 73 W
Intel Pentium Dual-Core G6950 --> 1156 ---> 2,8 ----> 100,95 ---> 2039 -----> 20 -------> 73 W
Intel Pentium Dual-Core E6800 ----> 775 ---> 3,33 ---> 96,5 -----> 2373 -----> 25 -------> 65 W
Intel Core i3-550 ---------------------> 1156 ---> 3,2 -----> 96,08 ----> 3100 ---> 32 -------> 73 W
Intel Core 2 Duo E7500 --------------> 775 ---> 2,93 ---> 90 -------> 1988 -----> 22

Thaks very much about all the info to Retvari Zoltan and Skgiven.

It took me a while find a good comparer of the motherboards and go throught all the characteristics. Finally I found this on MSI

The motherboard I chose is Z68A-GD55. Quite cheap.
• 2 PCI Express gen3 x16 slots
• Seems well spaced for 2 GPU
• 32GB Max four unbuffered DIMM of 1.5 Volt DDR3 1066/1333/1600*/2133*(OC) DRAM,
• Military Class II. Supponse it will last longuer if it can handle higher temperarutes.
• Supports Intel® Sandy Bridge processors in LGA1155 package (i3/i5/i7)

Hope this helps.

That board seems like a solid choice. However, it's nothing special. The PCIe lanes come off the CPU, which means if you use 2 GPUs it's going to be 2 x 8x, is it is with all socket 1155 boards. And the current Sandy Bridge CPUs don't support PCIe 3, as far as I know. That's only going to be possible with Ivy Bridge (soon to come).

However, the bandwidth will be fine for GPU-Grid :)

MrS

---

Scanning for our furry friends since Jan 2002

I'm not sure if you are aware of that none of the CPUs on your list will fit in the motherboard you've chosen.
From the Socket 1155 motherboard series I would suggest the ASUS Maximus IV Extreme-Z motherboard, it's not cheap, but it has 4 PCIe 2.0 x16 connectors, 2 of them can work at x16 speed at the same time.
But if price per performance is important, the best choice is the ASUS P7P55 WS Supercomputer motherboard. It's a Socket 1156 motherboard, so you will need an older Core i7 (i5 or i3) CPU for it (from your list).

I'm planning to build two computers with that motherboard and Pentium i5 2300. with 2 GPUs each.

From my experience I prefer to have 2 computers with the same configuration. Just because when something breaks it's quite easy to find and fix. I spent too much time before with LINUX drivers and old hardware in the other computers I had before.

On the other hand normally prices are exponential. Not only in CPUs, but in the rest of the hardware, so from my point of view I think it's better to have x2 lower end computer than 1 super high.

To be able to handle >4 GPU the processor and the memory has to be i7 or Xeon. With that price I could build a third computer.