Is there a limitation on how many GPUs that can be run from one system?

As an example, given dual 6 core Xeons and 48gb of ram, lets say you had two GPUs running in the system and another 4 GPUs running in an expansion chassis.

Anybody ever tried that?

I've been getting the idea (from blog articles etc.) that somehow there is a limit of 4 GPUs max regardless of the capabilities of the rest of the system. I don't know whether its a driver limitation or PCIe bus limitation. But I'm getting the impression that there is a limit.

Is this true?

Operator
____________

Good timing as I've been wondering about similar things.

I've never heard of an expansion chassis but I would consider trying one depending on what the gurus here say about it. For now, do you have a link to an expansion chassis in an online catalogue? I'm curious about price, specs, etc. And for purposes of discussion it's always nice to have an example to point to.

One of the admins sort of promised some specs on the 4 GPU systems they run in the project lab. I would still love to see those specs and possibly some photos.
____________
BOINC <<--- credit whores, pedants, alien hunters

Found an expansion chassis. Now I see. Hmmm, bottleneck in the cables and it looks to me like the slots will run at x8 if all occupied.

____________
BOINC <<--- credit whores, pedants, alien hunters

I was originally considering something like the Dell C410x chassis but it's complete overkill.

The downsides as I see it are, everything is air-cooled, and noisy,16 slots that each take a special card carrier, but the ability to only connect 4 cards using one PCIe iPass connection to the host.

I guess this means that up to 4 hosts can connect to the expansion chassis but that's not what I had in mind. I don't know if you could populate 8 GPU slots and connect all of that through 2 iPass cables to the same host machine.

The smaller expansion chassis from people like Cyclone Microsystems and Dynapower Netstor are pretty spendy to start with even before you mod it for water cooling to keep the noise down and improve the ability to bump the clocks,

I see a lot of folks here with GTX 690s as the maximum solution (even the current top host has 3 GTX 690s but I bet they're all internal to the system chassis).

So that's why I wondered if it was doable at all or if anybody had successfully tried to create a 'mini Titan'.

Operator
____________

Aye, they're pricey and I don't like the bottleneck inherent in the cable. The PSU in the one I linked to didn't have enough capacity in the PSU for the GPUs I am considering.

I'm all for jamming as many GTX690 as I can onto a single mobo but was told in another thread here recently 690s have driver and heat problems. There wasn't much elaboration on those problems (perhaps because I didn't ask for elaboration) but the way I see it if one person can make it work then so can I. If it's just a matter of time and effort then that's not a problem. I guess the question is.... are the reported problems insurmountable or merely difficult and when it's finally working does it work reasonably well or is it something you forever wish you had not done?

I'm looking favorably on the 660Ti too thanks to skgiven's advice. Good performance at reasonable price.

At newegg.ca I found this ASUS P6T7 WS Supercomputer mobo with 4 PCI-E x16 slots spaced such that they would accept 4 double height cards. $470, LGA-1366, Intel X58. The specs at newegg say

6 x PCIe 2.0 x16 (triple x16 or six x8)
1 x PCIe 2.0 x16

Dagorath - I did promise specs on our lab systems, didn't ?. We build machines with the following spec:

4xGTX680
Asus P9X79WS motherboard
Xeon E5 1620
16GB RAM
1500W Silverstone PSU
2TB hard disk.

We've our own custom case.

With Windows is possible to have "only" 4 GPU, for example 4xGTX680 or 2xGTX690 (dual GPU).
Another example:
[3] NVIDIA GeForce GTX 690 (2047MB) driver: 310.33

This is actually one GTX 690 and another GPU (GTX670 or something).
Boinc reports each individual GPU. A GTX690 has two GPU's so Boinc reports two GTX690's! It's doesn't report the card count, just the GPU count, and it has to call it something, so GTX690 it is. Boinc also reports the first (or biggest/most powerful GPU) and then the number of GPU's. So if you have a GTX680 in there, it will report it as another GTX690.
____________
Member of Boinc Italy.

PCI3 3.0 is double the bandwidth of 2.0. So x16/2.0 == x8/3.0. So no difference in bandwidth.
____________
XtremeSystems.org - #1 Team in GPUGrid

Thanks for all the info guys, Gattorantolo, werdwerdus, MJH. It helps us GPU newbies a lot.

____________
BOINC <<--- credit whores, pedants, alien hunters

With Windows is possible to have "only" 4 GPU, for example 4xGTX680 or 2xGTX690 (dual GPU).

Have you ever checked out the top host list?
There is a host with 6 GTX 690's (it's actually 3 cards) under Win7, and another with 7 GTX 580's under Linux.
If Windows recognizes the 3rd card, and BOINC is not, you should put a line into cc_config.xml under the <options> section:
<use_all_gpus>1</use_all_gpus>
If you don't have a cc_config.xml, you should create one. See this post on how to do that.

Cheers - yep running xml files.
currently running 4 GPUGrid wu's simultanious.
I'm sure it is a windows problem getting 3rd card to start playing.
While relatively new to GPUGrid I've been in top 20 daily now for past few months - minus maintenace days (like last few days)
The 3rd 690 won't go wasted but I would prefer that it stayed in this machine as part of a trio of 690's.
My rig we discuss in this thread is No 56 after 3 months
http://www.gpugrid.net/show_host_detail.php?hostid=154980
So trying to upgrade to improve performance on the grid.
Any help welcomed.
Thanks all

What you're going to want to do is investigate "PCIe lanes."

Different processors support different numbers of lanes.

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

You can find the number of lanes supported by various CPUs in the Intel documentation. I never found it for AMD.

So, just because your motherboard may be a bad mother in its own right, that doesn't mean that every CPU you might install in it will support enough "lanes" to the PCIe bus to let you get away with installing three 690s and a 680.

This was all new to me, I read it with interest, and since I wasn't in the market I promptly forgot the details.

I'm afraid that I have just exhausted all of my knowledge. The CPU is the ultimate limiter, although obviously a lousy motherboard would also prevent it from working even if you had the right CPU.

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

You can get 7 discrete GPU chips (e.g. three 690s and a 680) to work under Windows 7 provided that your CPU, that's with a "C" like CAT, has the "connections" to support that many PCIe "lanes."

Do you think Windows Server 2008 HPC would support more than 7 if the CPU supports it?

I can see that you've solved this problem.
What have you done to fix it?

I can see that you've solved this problem.
What have you done to fix it?

that happens when you merge computers on the GPUGrid site
____________
XtremeSystems.org - #1 Team in GPUGrid

that happens when you merge computers on the GPUGrid site

I've been semi-corrected elsewhere, so just wanted to correct the information I gave here in an effort not to confuse anyone who might read this thread.

The resource limitation with "lanes" and the like with CPUs is only a factor when trying to run four GTX 690s.

Apparently the GTX 590 is not such a resource hog and can be run four at a time.

That would mean it IS possible to run four, double-GPU cards in one system, just not four GTX 690s which are resource hogs. I have no idea what the "extra" burden a GTX 690 places on a system that a GTX 590 does not. DMA channels? Interrupts?

I don't know and don't know where to look.

I hope my previous comment hasn't mislead anyone.

I've been semi-corrected elsewhere, so just wanted to correct the information I gave here in an effort not to confuse anyone who might read this thread.

The resource limitation with "lanes" and the like with CPUs is only a factor when trying to run four GTX 690s.

Apparently the GTX 590 is not such a resource hog and can be run four at a time.

That would mean it IS possible to run four, double-GPU cards in one system, just not four GTX 690s which are resource hogs. I have no idea what the "extra" burden a GTX 690 places on a system that a GTX 590 does not. DMA channels? Interrupts?

I don't know and don't know where to look.

I hope my previous comment hasn't mislead anyone.

We build machines with the following spec:

4xGTX680
Asus P9X79WS motherboard
Xeon E5 1620
16GB RAM
1500W Silverstone PSU
2TB hard disk.

We've our own custom case.

MJH

We build machines with the following spec:

4xGTX680
Asus P9X79WS motherboard
Xeon E5 1620
16GB RAM
1500W Silverstone PSU
2TB hard disk.

We've our own custom case.

MJH

Question: How do you (or anyone) get the bottom card to fit? I have several different models of these "4x dual wide GPU boards", and they all have the same flaw: They put a lot of headers under the space of the 4th double-wide GPU. And so you can't push the GPU down into the PCIe slot without crushing the cables plugged into those headers.

Ah. That explains it. Thanks!
____________
Reno, NV
Team: SETI.USA