Hi all,
I've been looking into building a GPUGRID specific rig since my only Nividia card went on me a year ago and I had to replace with an AMD. I've been doing some analysis on number of lanes and PCIe 2.0 vs 3.0 to optimize the built output per $. What it looks like I'm seeing is that going with PCIe 3.0 isn't worth the additional cost for newer boards and chips and that lane speed isn't ultimately significant. Can someone confirm this? I'm looking into an MSI gd-70 which runs 4x PCIe 2.0 at x16 *2 or x8 *4. Is that the best layout for a GPU cruncher? From what I've seen it appears to be more cost effective than going with something like an ASUS Rampage with 3 x16 slots running 3.0 for almost 4 times the cost.

If I do go the 2.0 route do I have to worry about certain cards not being fully utilized or bottlenecking on the lanes? I was thinking of filling it with 4 GTX 560Ti cards but wanted to double check the x8 configuration with 2.0 PCIe would be fine when compared to other options.

Thanks!

I would also like suggest for a dedicated rig that two smaller machines may be cheaper, more efficient and in the long run easier to manage than a larger one. Don't forget to take into account your yearly running costs which may end up costing as much as your hardware!
Would recommend using an online psu calculator (such as 'eXtreme's') and use 65-75% of the 'minimum recommended PSU wattage' value as a rough power usage estimate for crunching.
Also note the higher end Nvidia cards are rumored to be replaced shortly to the 7 series which may bring down prices.

I would also add that if you want to run multiple GPUs on a mainboard, you need to consider how the PCIe slots are spaced. GPUs which occupy more slots might not fit, also cooling them properly is important. Running the cards 24/7 will generate a lot of heat, so the cards need to have enough spacing for proper air cooling. If that won't work, you'll need to consider liquid coolers for GPUs.

Decide if you want your system to host 2 GPU's or more, and build to that design.

Unless you are going to use 3 or more high-end GPU's, more than 2 PCIE slots isn't of any benefit and is a poor/irrelevant design.

Consider the expense difference between a mid-high end PSU (that can support 2 high end GPU's) against the extreme-end PSUs (that can support 4 high end GPU's); around $75 vs $300+.

I wouldn't invest in old kit for a new system. If you have a 4 PCIE2 x8 motherboard then fine, build on it, but don't buy a new PCIE2 motherboard for 4 mid-range previous generation GPU's.

You would be better off with 2 mid range to high end GPU's in a new system:

GTX680 (£370)
I43R11-NATHAN_dhfr36_6-11-32-RND9487_1 4442761 12 May 2013 | 3:05:32 UTC 12 May 2013 | 12:43:51 UTC Completed and validated 13,641.54 13,612.11 70,800.00
A GTX670 @ £290 seems better value to me.
Two GTX670's would require Four 6pin PICE power connectors, while the 680's need at least 1 8pin connector.

GTX560Ti (£150)
I42R18-NATHAN_dhfr36_5-15-32-RND7485_0 4427556 5 May 2013 | 11:25:59 UTC 5 May 2013 | 19:23:29 UTC Completed and validated 28,503.59 4,085.87 70,800.00

4 GTX560Ti's would require Eight 6pin PICE power connectors; a very high end modular PSU.

If mid range cards are more within your budget the GTX650Ti Boost, and GTX660 (£165) looks like a good price for it's performance. The GTX660Ti sits between the 670 and 660.

Even a GTX650Ti is £115 and can match a GTX560Ti for performance while costing less to buy and less to run:

I73R20-NATHAN_dhfr36_6-12-32-RND2263_0 4443459 12 May 2013 | 6:47:31 UTC 12 May 2013 | 14:48:49 UTC Completed and validated 28,497.36 28,410.83 70,800.00

These only require One 6pin PCIE power connector each.

PCIE3 isn't much faster than PCIE2 for here, but it is future proofed and allows you to use a 3rd generation Intel CPU, which use less electric, as do the GeForce 600 GPU's.

Remember - Only get what you can afford!

How would it be advantageous to build on newer MOBO architectures where the boards range from $300-$400 which handle 2-4 GPUs?

4 slot motherboards start ~$300, but you can get a two-slot PCIE3 motherboard for less than $100 (which was my point)! There are many to choose from, and some even have 3 PCIE slots (albeit the third slot is only PCIE2).
MSI have several LGA1155 Z77A-G45 models to choose from with more than one PCIE3 slot. Gigabyte, Asus, AsRock, and Intel also have numerous Z77 models.

excellent, thank you everyone. I picked up two GTX 660 Ti (2GB) cards and will hopefully have them up and running by the end of the week.

~Jack

GoodFodder built a machine with 2 650 Ti's and an Intel G2020 dual core CPU.

This CPU only supports PCIe 2.0 no matter the motherboard. I'm curious if there is a bottleneck with this type of machine setup for PCIe 2.0 at dual x8 mode. If not, would there potentially be a bottleneck with 2 680s?

I have found it inconvenient to do much crunching on my main desktop machine, so I'm also thinking about building a dedicated machine on a budget. It seems that 2 660s would be cheaper and perhaps gain a better RAC than one 680?

It looks like the Intel Z87 boards will be available soon. In addition to the new Haswell processors.

Here is the line of Asus boards: http://benchmarkreviews.com/index.php?option=com_content&task=view&id=22905&Itemid=47

I haven't found any information regarding additional PCIe 3.0 support yet.

Soon i can give a answer how a 570 runs in a pcie 1.0 x4 slot, cos i got my next one on ebay :D when it runs good i think its not from importance what sort of pcie ports you buy with your new board :)
Even now, "two" 570 are running perfect in a pcie 1.0 x16 slot witch means it has the speed of a pcie2.0 x8 or pcie3.0 x4.
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Ok tested, there is no difference between the speed of any kind of pcie slots. I know there are faster cards out there then 570s. But these are running fine even in pcie 1.0 x4 :) gpugrid needs fast gpus, but the rest of the computer you can get from garbage, so the project is a good recycler ^^
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Ok tested, there is no difference between the speed of any kind of pcie slots. I know there are faster cards out there then 570s. But these are running fine even in pcie 1.0 x4 :) gpugrid needs fast gpus, but the rest of the computer you can get from garbage, so the project is a good recycler ^^

Looks like 0.7% difference, if i've interpreted your results correctly. A GTX570 is a big and powerful enough card, and would have taxed the PCIE more under previous apps. It's good that it doesn't with the present app, except for those of us who spent the extra on PCIE3 setups based on previous app performances.

It's hard to know how much PCIE bandwidth is being used by a GF500 series GPU. While we can't go entirely by CPU usage, it does suggest a much more limited PCIE usage by the GF500 and GF400 cards. Last time I looked a GTX470 only needed ~8% of a CPU core/thread.
On the other hand a full CPU core/thread is needed for the GF600 (and presumably GF700) cards. So these cards are more likely to be impacted by a PCIE reduction due to high CPU usage, as well as being the new top performers.

Off topic: my house got hit directly by lightning a night ago. Luckily I have a lightning rod but the energy still goes into the ground and feeds back up through the phone line even though the DSL has a surge protector inline. Took out the DSL modem (fried), the switch the modem was on (5 ports good, 3 bad), one of the motherboards on a machine connected to that switch, maybe 1 GPU on a different box too. Spent half the day yesterday getting the network diagnosed (4 switches and 11 machines) and getting the DSL and routers configured and working again. Third lightning hit in the 20+ years I've lived here, the house is the highest one in the immediate vicinity, probably the reason...

Off topic: my house got hit directly by lightning a night ago. Luckily I have a lightning rod but the energy still goes into the ground and feeds back up through the phone line even though the DSL has a surge protector inline. Took out the DSL modem (fried), the switch the modem was on (5 ports good, 3 bad), one of the motherboards on a machine connected to that switch, maybe 1 GPU on a different box too. Spent half the day yesterday getting the network diagnosed (4 switches and 11 machines) and getting the DSL and routers configured and working again. Third lightning hit in the 20+ years I've lived here, the house is the highest one in the immediate vicinity, probably the reason...

Did you lose any computers or GPU's? Must have been pretty loud, eh? I've been watching it on CNN & FOX, you guys have been getting hammered pretty hard with tornados and T-storms back east plus a heat wave too. This time of the year we worry about forest fires, some real dimwits out there camping.

Off topic: my house got hit directly by lightning a night ago. Luckily I have a lightning rod but the energy still goes into the ground and feeds back up through the phone line even though the DSL has a surge protector inline. Took out the DSL modem (fried), the switch the modem was on (5 ports good, 3 bad), one of the motherboards on a machine connected to that switch, maybe 1 GPU on a different box too. Spent half the day yesterday getting the network diagnosed (4 switches and 11 machines) and getting the DSL and routers configured and working again. Third lightning hit in the 20+ years I've lived here, the house is the highest one in the immediate vicinity, probably the reason...