Is there a limitation on how many GPUs that can be run from one system?

As an example, given dual 6 core Xeons and 48gb of ram, lets say you had two GPUs running in the system and another 4 GPUs running in an expansion chassis.

Anybody ever tried that?

I've been getting the idea (from blog articles etc.) that somehow there is a limit of 4 GPUs max regardless of the capabilities of the rest of the system. I don't know whether its a driver limitation or PCIe bus limitation. But I'm getting the impression that there is a limit.

Is this true?

Operator

---

Good timing as I've been wondering about similar things.

I've never heard of an expansion chassis but I would consider trying one depending on what the gurus here say about it. For now, do you have a link to an expansion chassis in an online catalogue? I'm curious about price, specs, etc. And for purposes of discussion it's always nice to have an example to point to.

One of the admins sort of promised some specs on the 4 GPU systems they run in the project lab. I would still love to see those specs and possibly some photos.

---

BOINC <<--- credit whores, pedants, alien hunters

I believe Nvidia had an 8 card limit in GeForce drivers, although this may have been removed from later drivers. I have run an expansion chassis with 4 x GTS450SP and another two in the machine itself without any issues. I believe BOINC also had a limit (8 or 16) and we asked for it to be changed, some time back now. Not sure if its actually been done.

You'll have to run a 64bit OS as you run out of address space with that many cards.

---

BOINC blog

Found an expansion chassis. Now I see. Hmmm, bottleneck in the cables and it looks to me like the slots will run at x8 if all occupied.

---

BOINC <<--- credit whores, pedants, alien hunters

I was originally considering something like the Dell C410x chassis but it's complete overkill.

The downsides as I see it are, everything is air-cooled, and noisy,16 slots that each take a special card carrier, but the ability to only connect 4 cards using one PCIe iPass connection to the host.

I guess this means that up to 4 hosts can connect to the expansion chassis but that's not what I had in mind. I don't know if you could populate 8 GPU slots and connect all of that through 2 iPass cables to the same host machine.

The smaller expansion chassis from people like Cyclone Microsystems and Dynapower Netstor are pretty spendy to start with even before you mod it for water cooling to keep the noise down and improve the ability to bump the clocks,

I see a lot of folks here with GTX 690s as the maximum solution (even the current top host has 3 GTX 690s but I bet they're all internal to the system chassis).

So that's why I wondered if it was doable at all or if anybody had successfully tried to create a 'mini Titan'.

Operator

---

I've run Tyan FT48-B7025 units with 16 Tesla C1070s, and also seen them configured with 8x K10s. It's not cost-effective, in terms of $/GPU, because the more exotic host system has a significant price premium.

The C410xs are.. disappointing. Expensive, bad PCIe topology and custom sleds.

Dagorath - I did promise specs on our lab systems, didn't ?. We build machines with the following spec:

4xGTX680
Asus P9X79WS motherboard
Xeon E5 1620
16GB RAM
1500W Silverstone PSU
2TB hard disk.

We've our own custom case.

MJH

Aye, they're pricey and I don't like the bottleneck inherent in the cable. The PSU in the one I linked to didn't have enough capacity in the PSU for the GPUs I am considering.

I'm all for jamming as many GTX690 as I can onto a single mobo but was told in another thread here recently 690s have driver and heat problems. There wasn't much elaboration on those problems (perhaps because I didn't ask for elaboration) but the way I see it if one person can make it work then so can I. If it's just a matter of time and effort then that's not a problem. I guess the question is.... are the reported problems insurmountable or merely difficult and when it's finally working does it work reasonably well or is it something you forever wish you had not done?

I'm looking favorably on the 660Ti too thanks to skgiven's advice. Good performance at reasonable price.

At newegg.ca I found this ASUS P6T7 WS Supercomputer mobo with 4 PCI-E x16 slots spaced such that they would accept 4 double height cards. $470, LGA-1366, Intel X58. The specs at newegg say

6 x PCIe 2.0 x16 (triple x16 or six x8)
1 x PCIe 2.0 x16

To me that means I can put 4 cards in the 4 blue slots and they'll all run at x16, is that right?

I would much rather have Gen. 3 PCI-E even at the additional expense, for future considerations, but the word is GPUgrid tasks don't need that much bandwidth and do just fine on Gen. 2 so maybe I would compromise on that point. I am considering starting out with this mobo and one card then add say one more card per month/paycheque.

For CPU probably a low cost Xeon like this one with 8MB L2 cache. I like big cache for multitasking.

For power supply.... 4 X GTX690 needs at least 2800 watts so I am seriously considering 2 of these 1500 watt EVGA SuperNOVA NEX1500 gold certified model. They have a 10 year warranty. One review says this model can be configured via onboard DIP switches to run when not connected to a mobo. So the plan would be to buy the mobo + CPU + 1 GTX690 + 1 PSU for now and the PSU has enough capacity for a second GTX690 which I'll purchase later. On purchase of the third 690 (or perhaps one of the rumored Titans) I would buy another of the same PSU and configure it to run without the mobo connection. It will also power the fourth GTX690/TiTan/whatever.

I haven't looked hard but the only 2800 watt PSUs I can find are these 2 models by Cisco and at $1675 they're too rich for me. Two of the 1500 watt models mentioned above are only $920 and the pair deliver 200 watts more.

Buying 2 X 1500 watt PSU has the advantage of being modular. If one fails I will still have the CPU and 2 GPUs running. There is also the advantage of splitting the cost over 2 payments. And I can't find a 3,000 watt PSU anywhere!

Another PSU option is to have 1 regular PSU to power the mobo and 1 card plus another that puts out only the +12 required for the other cards. A big watt +12 only PSU is less complicated than a normal PSU so it's gotta be cheaper. Also it would probably be very simple and cheap to build one.

---

BOINC <<--- credit whores, pedants, alien hunters

Dagorath - I did promise specs on our lab systems, didn't ?. We build machines with the following spec:

4xGTX680
Asus P9X79WS motherboard
Xeon E5 1620
16GB RAM
1500W Silverstone PSU
2TB hard disk.

Thanks, MJH!

I'm interested mainly in that motherboard. Newegg.ca stocks them and here it is for $377, $90 cheaper than the mobo I mentioned in my previous post. The thing about it that gives me concern is the PCI-E specs. It's PCI-E Gen 3 but the slots run at x8 if they're all occupied. The mobo I mentioned in my previous post has 4 slots that run at x16 even when all occupied but they're only Gen 2 not Gen 3. Which mobo would give the best performance on GPUgrid tasks? My hunch is x16 is more important than Gen 3 but I really don't know for sure.

We've our own custom case.

The best case is no case, IMHO, and I won't waste money on another one. A cardboard box is good enough for me.

---

BOINC <<--- credit whores, pedants, alien hunters

With Windows is possible to have "only" 4 GPU, for example 4xGTX680 or 2xGTX690 (dual GPU).
Another example:
[3] NVIDIA GeForce GTX 690 (2047MB) driver: 310.33

This is actually one GTX 690 and another GPU (GTX670 or something).
Boinc reports each individual GPU. A GTX690 has two GPU's so Boinc reports two GTX690's! It's doesn't report the card count, just the GPU count, and it has to call it something, so GTX690 it is. Boinc also reports the first (or biggest/most powerful GPU) and then the number of GPU's. So if you have a GTX680 in there, it will report it as another GTX690.

---

Member of Boinc Italy.

PCI3 3.0 is double the bandwidth of 2.0. So x16/2.0 == x8/3.0. So no difference in bandwidth.

---

XtremeSystems.org - #1 Team in GPUGrid

I'm interested mainly in that motherboard. Newegg.ca stocks them and here it is for $377, $90 cheaper than the mobo I mentioned in my previous post. The thing about it that gives me concern is the PCI-E specs. It's PCI-E Gen 3 but the slots run at x8 if they're all occupied. The mobo I mentioned in my previous post has 4 slots that run at x16 even when all occupied but they're only Gen 2 not Gen 3. Which mobo would give the best performance on GPUgrid tasks? My hunch is x16 is more important than Gen 3 but I really don't know for sure.

Dagorath - For GPUGRID tasks, PCIe speed isn't tremendously important, except in the occasional experiment where we add customisations. In that case, having a fast CPU is just as important, if not more. Also, for best performance, run Linux.

For GPUGRID tasks alone, check out the Asus P8Z77-WS board. It'll give the same performance at a lower price point. Similarly, 16GB is overkill - 1GB per GPU is adequate.

MJH

Thanks for all the info guys, Gattorantolo, werdwerdus, MJH. It helps us GPU newbies a lot.

---

BOINC <<--- credit whores, pedants, alien hunters

With Windows is possible to have "only" 4 GPU, for example 4xGTX680 or 2xGTX690 (dual GPU).

This seems the best answer to the OP question.
BUT is there a limit to the amount of gpu's per machine on GPUGrid ??
I ask because I can't get a third GTX 690 to be recognized correctly.
Looks like it will be going into another system because it is just wasting power draw sitting in a slot doing sweet fa.
Willing to provide all information required if someone willing to help me solve this issue - if it is solvable.
Thanks

Have you ever checked out the top host list?
There is a host with 6 GTX 690's (it's actually 3 cards) under Win7, and another with 7 GTX 580's under Linux.
If Windows recognizes the 3rd card, and BOINC is not, you should put a line into cc_config.xml under the <options> section:
<use_all_gpus>1</use_all_gpus>
If you don't have a cc_config.xml, you should create one. See this post on how to do that.

Cheers - yep running xml files.
currently running 4 GPUGrid wu's simultanious.
I'm sure it is a windows problem getting 3rd card to start playing.
While relatively new to GPUGrid I've been in top 20 daily now for past few months - minus maintenace days (like last few days)
The 3rd 690 won't go wasted but I would prefer that it stayed in this machine as part of a trio of 690's.
My rig we discuss in this thread is No 56 after 3 months
http://www.gpugrid.net/show_host_detail.php?hostid=154980
So trying to upgrade to improve performance on the grid.
Any help welcomed.
Thanks all

What you're going to want to do is investigate "PCIe lanes."

Different processors support different numbers of lanes.

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

You can find the number of lanes supported by various CPUs in the Intel documentation. I never found it for AMD.

So, just because your motherboard may be a bad mother in its own right, that doesn't mean that every CPU you might install in it will support enough "lanes" to the PCIe bus to let you get away with installing three 690s and a 680.

This was all new to me, I read it with interest, and since I wasn't in the market I promptly forgot the details.

I'm afraid that I have just exhausted all of my knowledge. The CPU is the ultimate limiter, although obviously a lousy motherboard would also prevent it from working even if you had the right CPU.

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

Do you think Windows Server 2008 HPC would support more than 7 if the CPU supports it?

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

Do you think Windows Server 2008 HPC would support more than 7 if the CPU supports it?

I really did tell everything I know and I'm not even sure that information is exhaustively correct.

It seems like I've seen more than 7 cards running on a system, so someone has figured-out something; but I also know that someone else was trying to write a custom BIOS to get 4 GTX 690s to run on a consumer board and found-out that it wasn't possible due to the CPU hardware limitations.

I can see that you've solved this problem.
What have you done to fix it?

I can see that you've solved this problem.
What have you done to fix it?

It was a driver issue.
Uninstalled everything Nvidia, then ran drive-sweeper.
When re-booted and installed latest Nvidia drivers suite everything was recognized and worked in harmony.
I also updated the SR-2 to new Bios recently released (thanks EVGA) but that did not make any difference to the problem I had.
running 2 x Xeon 5650 cpus

This score form yesterday just can't be correct can it ??
25,483,750 from this one host.
Have no idea what the glitch was, but today seems more normal.