nVidia can already disable shader clusters as they whish. They could sell a GF100 with only 32 shaders active, if this made any sense. If I remember correctly the first cards to employ these techniques were ATIs X800 Pro and nVidias 6800GS. On newer cards they may also have more flexibility regarding the ROPS, memory controllers etc., so I think that's quite good already.

What do you mean by "lack of limitations for a GPU card"? I think it does have severe limitations: the maximum length is defined by ATX cases (we're already pushing the limits here), the height is also limited by the regular cases and power draw and cooling are severely limited. Sure, you could just ask people to plug even more 6 and 8 pin PCIe connectors into the cards, but somehow you also have to remove all that heat.

I could see two or four 40nm GPUs on one board,

2 Chips definitely (HD5970), but 4? What kind of chips would you use? Since we don't know enough about Fermi I'd prefer to discuss this based on ATI chips. As I said even going for 3 Cypress (HD58x0) is almost impossible, or at least unfeasible. You could use 2 x 2 Juniper (HD 57x0), which are exactly half a Cypress. Performance in Crossfire is surprisingly good compared to the full configuration, but you need an additional 30 - 40 W for the CF tandems compared to the single chip high end cards (see next page). That would turn a 300 W HD5970 into a 360 - 380 W card, if you used 4 Junipers. It had to be noticably cheaper to justify that.

Could chips be designed to include spaces, like fire-breakers?

They could, but the manufacturing cost is the same regardless of whether you use the area for transistors or whether you leave it blank. So you reduce power density somewhat and reduce the number of chips per wafer considerably. And power density at the chip level is not the problem of GPUs, as the chips are large enough to transfer the heat to the cooler. Their main problem is that the form factor forbids the use of larger fans, so removing the heat from the heatsink is the actual problem.

BTW: thanks for discussing :)

MrS

---

Scanning for our furry friends since Jan 2002

@liveonc:

We'd need at least one PCIe 2.0 16x connection routed to the external box. That would mean it would have to be very close to the PC, but that's no show stopper.

What I could imagine: use a common PCB for 4 (or "x" if you prefer) GPU chips, or separate PCBs for each of them (cheaper, more flexible but you'd have to connect them). Make them approximately quadratic and cover each of them with a large heatsink. There'd probably have to be 2 or 3 heatpipes for the GPU core, whereas memory and power cicuitry could be cooled by the heatsink itself. Then put a 120 or 140 mm on top of each of these "GPU blocks". Leave the sides of the device free enough so that the hot air can flow out. You'd end up with a rather flat device which covers some surface, so ideally you'd place it like a tower parallel to your.. ehm, PC-Tower. It could also be snapped upon the front or backside, so you could easily use 2 of these moduls for one PC and both PCIe links would only have to cover ~10 cm to the motherboard I/O panel. For BOINC this would absolutely rock, whereas for games we might need faster and / or wider PCIe. One couild also replace the individual fans with a single really large one.. but then there'd be a large dead spot beneath its axis. And if only selected GPUs are loaded one couldn't speed fans up selectively. BTW: the fans would have easily accessible dust filters.

Oh wait.. did I just write all of this? Let me call my patent attorney first ;)

MrS

---

Scanning for our furry friends since Jan 2002

e-SATA has been around for a long time. e-PCIe should be the next thing, why not? Nvidia wanted to go nuts, let them, if space, 300W, & cooling, no longer is a hindrance, Nvidia could go into the external GPU market, PSU market, & VGA cooling market. Plus they'd get their fermi.

BTW, do you know if a serial repeater is possible, in case the distance from the PC case to the external GPU has to be a bit further away, & would there be any sense in making x1, x4, x8, & x16 PCIe cables, if fx a PCIe 2.0 speed is 5.0 GHz, while the first version operates at 2.5 GHz, if fx the card is PCIe 1.0?

This is what I'm getting at, but much more simple, & only meant to house a single fermi using x16 PCIE 2.0 with an oversized GPU cooler & dedicated PSU: http://www.magma.com/expressbox7x8.html

---

Here's another one: http://www.pcper.com/comments.php?nid=8222 but it's as I "suspect", much like the Asus XG Station: http://www.engadget.com/2007/01/08/hands-on-with-the-asus-xg-station-external-gpu, & if it is, it's limitted to x1 PCIe.

Magma has the right idea with their ExpressBox7x8: http://www.magma.com/expressbox7x8.html the iPass cable used looks as if it's at least 5m in length. But different from Magma, I was thinking of just 1 Fermi, in it's own box, with it's own PSU, on it's back to the bottom of the box (like a motherboard lies in a PC casing), with a giant heatsink & fan. If nVidia wants their Monster Fermi to use 800W it can do it outside with a moster GPU cooler & dedicated PSU. If they want enthusiasts to be able to afford multiple cards, they could make it possible (& attractive), if there is a way to stack one Fermi Box on top of the other to get that external SLI.

Everybody's talking as if nVidia went to far, but if they don't have to think about space limitation, power consumption, or heat dissipation, they can make their monster (& get away with it). I remember that all the critics of Intel LOL when Intel's Original Pentium overheated, only to have Intel come back, (with 40 belly dancers), when the placed a heat sink & fan on top of their Pentium.

---

Read the ads with links at the bottoms of threads - Some of them offer rack-mounted GPU boxes (mostly Nvidia type with a CPU in the box also, but often with 4 GPUs; I've even seen one where one Xeon CPU is expected to control up to 8 GPUs (GTX295 type if you prefer). Rack-mounted systems should be able to get taller than a stack with no frame to support the higher units.

No Fermi boards yet, though.

Disclaimer: I do not know as much about chip design as I'd like, and I'm holding off any commentary on Fermi until it's actually in reviewers' and end users' hands. Anything else is pure gossip and conjecture as far as I'm concerned.

That said, much has been made about Intel's vs AMD's vs nVidia's design decisions.

AMD vs Intel's approach in the CPU world intrigue me. For Intel, they were content to package two chips onto a single die to make their initial quad core offerings. It worked quite well.

What prevents GPU makers from doing the same? Two GT200b(c?) dies on a single GPU package. It'd be a GTX295 that wouldn't require doubling up the vRAM. Make it at 40nm. It seems to make so much sense (at least to me), but nobody is doing it and there has to be a reason. Does anybody know what that reason is?

---

C2Q, GTX 660ti

What prevents GPU makers from doing the same? Two GT200b(c?) dies on a single GPU package. It'd be a GTX295 that wouldn't require doubling up the vRAM. Make it at 40nm. It seems to make so much sense (at least to me), but nobody is doing it and there has to be a reason. Does anybody know what that reason is?

Even at 40nm two GT200b cores would be huge and most likely wouldn't yield well. Huge dies and poor yields = no profit. In addition GPUs are very different from CPUs. The shaders are basically cores. For instance in Collatz and MW a 1600 shader 5870 does twice the work of a 800 shader 4870.

@Robert: the problems with the current rack mounted devices are numerous

- they're meant for professionals and thus feature expensive Teslas
- I don't have a 19" rack at home (and am not planning to buy one)
- the air flow is front to back, with the units designed to be stacked
-> the cooling can't be quiet this way
- a separate Xeon to drive them? I want to crunch with GPUs so that I don't have to buy expensive CPUs!

@liveonc: that cable in the Magma-box looks really long, so it seems like PCIe extension is no big deal :)

@Jeremy: you have to feed data to these chips. 256 bit memory interfaces are fine for single chips, where the entire space around them can be used. 386 and 512 Bit quickly get inconvenient and expensive. In order to place two GT200 class chips closer together you'd have to essentially create a 768 Bit memory interface. The longer and the more curved the wires to the memory chips are, the more problematic it becomes to clock them high. If your wire layout (and some other stuff) is not good you can not even clock the memory as high as the chips themselves are specified.
So without any serious redesign you create quite some trouble by placing two of these complex chips closer together - for zero benefit. You could get some benefit, if there was some high bandwidth, low latency communication interface between the chips. Similar to what AMD planned to include in Cypress as Sideport, but had to scrap to keep the chip "small". But that's a serious redesign. And I suppose they'd rather route this communication interface over a longer distance than to cram the memory chips into an even smaller area.
A single cypress pushes the limits of what can be done with 256 Bit GDDR5 quite hard already: HD4890 was not memory bandwidth limited. HD5870 increased raw power by a factor of 2 and memory bandwidth only by a factor of 1.3 and as a result could use even faster memory. Yet they had to be very careful about the bus to the memory chips, its timing and error detection and correction to reach even these speeds.

MrS

---

Scanning for our furry friends since Jan 2002

Disclaimer: I do not know as much about chip design as I'd like, and I'm holding off any commentary on Fermi until it's actually in reviewers' and end users' hands. Anything else is pure gossip and conjecture as far as I'm concerned.

That said, much has been made about Intel's vs AMD's vs nVidia's design decisions.

AMD vs Intel's approach in the CPU world intrigue me. For Intel, they were content to package two chips onto a single die to make their initial quad core offerings. It worked quite well.

What prevents GPU makers from doing the same? Two GT200b(c?) dies on a single GPU package. It'd be a GTX295 that wouldn't require doubling up the vRAM. Make it at 40nm. It seems to make so much sense (at least to me), but nobody is doing it and there has to be a reason. Does anybody know what that reason is?

I have years of experience in some of the early stages of chip design, mostly logic simulation, but not the stages that would address putting multiple chips in one package. I'd expect two such GPU chips in the same package to generate too much heat unless the chips were redesigned first to generate perhaps only half as much heat, or the clock rate for the pair of chips was reduced about 50% leaving them doing about as much work as just one chip but at a more normal clock rate.

Also, I suspect that the chips are not already designed to let two of them share the same memory, even if both chips are in the same package, so you'd need about twice as many pins for the package to allow it to use two separate sets of memory.

@Robert: the problems with the current rack mounted devices are numerous

- they're meant for professionals and thus feature expensive Teslas
- I don't have a 19" rack at home (and am not planning to buy one)
- the air flow is front to back, with the units designed to be stacked
-> the cooling can't be quiet this way
- a separate Xeon to drive them? I want to crunch with GPUs so that I don't have to buy expensive CPUs!

MrS

The best I can tell, the current application designs don't work with just a GPU portion of the program; they need a CPU portion to allow them to reach BOINC and then the server. The more GPUs you use, the faster a CPU you need to do this interfacing for them. Feel free to ask the rack mounted equipment companies to allow use of a fast but cheap CPU if you can find one, though. Also, it looks worth checking which BOINC versions allow the CPU interface programs for interfacing to separate GPUs to run on separate CPU cores.

However, I'd expect you to know more than me about just how fast this interfacing needs to be in order to avoid slowing down the GPUs. Putting both the GPUs and the CPU on the same board, or at least in the same cabinet, tends to make the interface faster than putting them in separate cabinets.

The rack mounted devices I've looked at put the GPUs and the CPU on the same board, and therefore don't use Teslas. I haven't checked whether they also use enough memory to drive up the price to Tesla levels.

I agree with your other objections, though.

Jeremy, if the heat generated and the package size weren't problems, you could take your idea even further by putting the video memory chips in the same package as the GPU, and therefore speed up the GPU to memory interface.

As it is, consider how much more you'd need to slow down the GPU clocks to avoid overheating the memory chips with this approach.

However, I'd expect you to know more than me about just how fast this interfacing needs to be in order to avoid slowing down the GPUs. Putting both the GPUs and the CPU on the same board, or at least in the same cabinet, tends to make the interface faster than putting them in separate cabinets.

But Intel has already taken the first step on nVidia's foot, by not allowing them to build boards for the Core i7

nVidia already wanted to promote Supercomputing via Fermi. If they start selling external GPU kits, even an ITX with a PCIe should "theoretically", be able to use a monster GPU. Taking it even one step further, if both nVidia & Ati start selling external GPU kits with oversized GPU's unable to fit in any traditional casing, they could start making GPU boards & offer to sell the GPU chip & memory
separately. PSU, GPU Cooling, & RAM manufacturers, would be delighted, wouldn't they? But of course, Intel probably will object...

---

But Intel has already taken the first step on nVidia's foot, by not allowing them to build boards for the Core i7

What are you talking about?

---

Thanks - Steve

Only Intel makes chipsets for LGA1156 Socket & LGA1366 Socket, heard a "rumor", that it wasn't because nVidia couldn't.

---

I was confused because you said boards.
The 1366 socket chips integrated the functionality of the nvidia chipset into the cpu directly and I think nvidia announced they were getting out of the chipset market entirely (not making them for amd motherboards either) before 1156 was launched.

---

Thanks - Steve

Only Intel makes chipsets for LGA1156 Socket & LGA1366 Socket, heard a "rumor", that it wasn't because nVidia couldn't.

Shouldn't matter unless the CPU and GPU share the same socket, the same socket type is not critical otherwise. Intel's moving in the direction of putting both the GPU and the CPU on the same chip, and therefore into the same socket, but I've seen no sign that Nvidia has designed a CPU to put on the same chip as the GPU. A few problems with Intel's approach: I've seen no sign of a BOINC version that can make any use of a GPU from Intel yet. No monster Intel GPUs yet, and I expect both heat problems and VRAM pin count problems from trying to put both a monster GPU and any of Intel's current CPU designs into the same package. I suppose that the GPU could share the same memory as the CPU, IF you're willing to give up the usual speed of the GPU to VRAM interface, but is that fast enough to bother making any? Alternatively, you could wait a few years for chip shrinkage and power requirements shrinkage to reach the point that a CPU, a monster GPU, and the entire VRAM all fit into one package without heat or size problems.

As for Nvidia giving up the chipset market, isn't that most of their product line now? Why would they be willing to risk making such a large change unless the alternative was bankruptcy? Or do you, for example, mean the part of the chipset market for chips not related to graphics?

Robert,

I've seen no sign of a BOINC version that can make any use of a GPU from Intel yet.

Al current Intel GPUs are low-performance, fixed-function units, quite useless for computation.

As for Nvidia giving up the chipset market, isn't that most of their product line now? Why would they be willing to risk making such a large change unless the alternative was bankruptcy?

Nvidia do not have a license for QPI, only the FSB used by older processors (and Atom).

MJH

I was confused because you said boards.
The 1366 socket chips integrated the functionality of the nvidia chipset into the cpu directly and I think nvidia announced they were getting out of the chipset market entirely (not making them for amd motherboards either) before 1156 was launched.

Looks to me more like Nvidia has decided that they'll have to fight some legal battles with Intel, and start some new chip designs at early stages, then wait until these designs are finished, before they can offer any products compatible with any new type of board slots designed to make use of the new features of the Intel 5 and Intel 7 CPUs, but for now will continue making products for computers using the older FSB interface. See here for a report that looks somewhat biased, but at least agrees that Nvidia is not planning a full exit from the chipset market:

http://www.brightsideofnews.com/news/2009/10/9/nvidia-temporarily-stops-development-of-chipsets-to-intels-i5-and-i7.aspx

Intel appears to have designed the new QPI interface to do two things:

1. Give speedups by separating the main memory interface from the peripherals interface. This means, for example, that any benchmark testing that uses programs that use the memory a lot, but not also any peripherals, may have trouble seeing much difference between FSB and QPI interface results.

2. I'd be surprised if they aren't also trying to restrict the ability or either Nvidia or AMD/ATI to enter the graphics card market for machines using new features of the Intel 5 and Intel 7 by refusing then QPI licenses.

http://www.intel.com/technology/quickpath/index.htm?iid=tech_arch_nextgen+body_quickpath_bullet

AMD has developed a feature called Hyper-threading for their newer CPUs; I expect them to try to restrict the ability of either Nvidia or Intel to enter the graphics card market for any computers using that feature by also denying licenses.

I suspect that as a result, Nvidia will have problems offering products for any new machines designed to make full use of the new Intel or AMD features, and for now will have to concentrate on offering products for older types of motherboard slots. Somewhat later, they can offer products that put a CPU on the same board as the GPU, if they can find a CPU company wanting to increase sales of a suitable CPU but not offering any competing graphics products; hopefully one that already has a BOINC version available. I'd expect some of these products to be designed for cases that don't try to fit the ATX guidelines.

This is likely to give crunchers problems choosing computers that give the highest performance for both GPU projects and CPU projects, and are still produced in large enough quantities to avoid driving up the price, for at least some time. Not as many problems for crunchers interested mainly in GPU projects, though.

I've found an article about efforts to design a dual-Fermi board, although with no clear sign that a significant number of them will ever be offered for sale or that it won't overheat with the normal spacing between cards.

http://www.brightsideofnews.com/news/2010/2/26/asus-to-introduce-dual-fermi-on-computex-2010.aspx

2. I'd be surprised if they aren't also trying to restrict the ability or either Nvidia or AMD/ATI to enter the graphics card market for machines using new features of the Intel 5 and Intel 7 by refusing then QPI licenses.

Intel partnered with NVidia and then cut them out, copied from the MS playbook. When Intel settled the lawsuit with AMD, part of the settlement was cross-licensing of technologies so AMD should be OK in this regard. Various governments still have litigation against Intel so it might be in their best interest to play a bit nicer than they're accustomed to.