Retvari,

Yes, I had some trouble with that word you used... "better". Thanks for defining it. And thanks to you and skgiven for filling me in on the details concerning what's been going on in the marketplace regarding nVIDIA's supplying GPUs for supercomputers. I knew they were supplying them but I didn't see the rest of the puzzle until now. It all starts to make so much more sense now and once again I say "if you don't know history you can't truly appreciate where you are today and will likely have trouble choosing a wise path to your future".

So now a question.... I read recently that nVIDIA purposely disabled 64bit floating point ops to protect their high end models. Now that I realize what we're getting on these consumer cards is chips leftover from sales for supercomputers I am beginning to think that maybe what really happened is that these chips were ones that the 64 bit ops did not work on (manufacturing defects due to contaminants and what not) or did not work reliably which meant they did not meet the contracted specs for the supercomputers so they've taken those and made sure the 64bit does not work at all and marketed them as video cards. Similar to when part of the intended cache on an i7 doesn't work reliably they can sometimes excise/disable that portion while leaving the remainder functional. Then they sell it as a model with smaller cache at a lower price. Is that what's happened with the 64 bit ops on nVIDIA cards?

---

BOINC <<--- credit whores, pedants, alien hunters

Is that what's happened with the 64 bit ops on nVIDIA cards?

In short: no.

The number of consumer GPUs are still a lot higher than Quadro, Tesla and supercomputers (basicaly custom Teslas) combined. They don't have that many defective chips. And with GPUs it's easy to just disable a defective SMX, instead of trying to disable certain features seletively.

In the Fermi generation they did cut FP performance from 1/2 the SP performance down to 1/8 on consumer GF100 and GF110 chips (the flagships). However, this was purely to protect sales of the more expensive cards.

With Kepler it's different: all "small" chips, up to GK104 (GTX680) physically have only 1/12 the SP performance in DP. This makes them smaller and cheaper to produce. Only the flagship GK110 has 1/3 the SP performance in DP. These are the only Teslas which matter for serious number crunching. We'll see if they'll cut this again for GF Titan.

MrS

---

Scanning for our furry friends since Jan 2002

An example of the lack of OpenCL maturity is the over reliance on PCIE bandwidth and systems memory rates. This isn't such an issue with CUDA.

POEM@Home is special, the performance characteristic seen there is firstly a result of the specific code being run. What part of this can be attributed to OpenCL in general is not clear. E.g. look at Milkyway: they didn't loose any performance (and almost don't need CPU support) when transitioning from CAL to OpenCL in HD6000 and older cards. The reason: simple code.

However, there was/is some function call / library being used which was optimized in CAL. It's still being used on the older cards (requires some fancy tricks). However, HD7000 GPUs can't use the optimized CAL routine and loose about 30% performance jsut due to this single function. And nothing has changed in this regard since about a year. That's what maturity and optimization mean for GP-GPU.

MrS

---

Scanning for our furry friends since Jan 2002

Is that what's happened with the 64 bit ops on nVIDIA cards?

The GTX580 and the 20x0 series Teslas used the same GF100/GF110 silicon. The products were differentiated, in part, by the amount of DP logic enabled. For the Kepler generation, there are separate designs for GTX680 and the Kx0 series silicon -GK114 and GK110 respectively. The former is distinguished by having a simpler SM design and only a few DP units. It will be interesting to see what features tip up in the GK110-using Geforce Titan. I expect DP performance will be dialled down, in part to maintain some differentiation against the Tesla K20c and also to allow reuse of partially defective dies.

Anyhow, the GPUGRID application has minimal need for DPFP arithmetic and - furthermore - was developed in a period before GPUs had any DP capability at all.

MJH

Neverone to pass up the opportunity for a little brinksmanship or perhaps just the opportunity to reminisce, my first one was a Heath kit a friend and I soldered together with an iron I used as a child to do woodburning, that and solder the size plumber's use. No temperature control on the iron, I took it to uncle who ground the tip down to a decent size on his bench grinder. We didn't have a clue in the beginning and I wish I could say we learned fast but we didn't. It yielded an early version of the original Apple Wozniak and friends built in... ummm...whose garage was it... Jobs'? Built it, took 2 months to fix all the garbage soldering and debug it but finally it worked. After that several different models of Radio Shack's 6809 based Color Computer, the first with 16K RAM and a cassette tape recorder for storage and the last with 1 MB RAM I built myself and an HD interface a friend designed and had built in a small board jobber in Toronto. He earned an article in PC mag for that, it was a right piece of work. That gave me 25 MB storage and was a huge step up from the 4 drive floppy array I had been using. It used OS/9 operating system (Tandy's OS/9 not a Mac thing), not as nice as CPM but multi-tasking and multi-user. Friends running 8088/86 systems were amazed. And it had a bus connector and parallel port we used and for which we built tons of gizmos, everything from home security systems to engine analyzers. All with no IRQ lines on the 6809, lol.

Wow, you're old too! Sounds like a nice piece of engineering you did there.

Win2.x and 3.1 was useless to me since my little CoCo NEVER crashed and did everything Win on a '286 could do including run a FidoNet node, Maximus BBS plus BinkleyTerm. Then the bomb went off... Gates publicly declared the '286 was braindead, IBM called in their option on OS/2, the rooftop party in Redmond, OS/2 being the first stable multitasking GUI OS to run on a PC and it evolving fairly quickly into a 16 bit OS while Win did nothing but stay 8 bit and crash a lot. Ran OS/2 on a '386 I OC'd and built a water cooling system for through the '486 years then a brief dalliance with Win98 on my first Pentium which made me puke repeatedly after rock solid OS/2 and genuine 16 bitness, CPM and OS/9 so on to Linux which I've never regretted for a minute.

I ran a 4 line ProBoard BBS for years , even before Al Gore invented the internet. OS/2 based of course as that was the x86 multitasking OS that was stable. I really did like OS/2.

Gates was the last man standing, no competition. And that is why Windows is such a piece of sh*t. What other examples do you know of?

A hardware example: x86 CPUs. Remember when Intel would release CPUs at 5 MHz speed bumps and charge a whopping $600 (think of that in today's dollars) for the latest greatest? The only thing that kept them honest at all was AMD, who due to licensing agreements was able to copy the 286/386/486 and bring out faster, cheaper versions of them all. Later Intel dropped some of the licensing and meanwhile brought out the very good P3. AMD then countered with the Athlon at about the same time Intel brought out the P4. The P4 was designed for high clock speeds to allow Intel to apply it's old incremental "small speed increase ad nauseum" strategy. Unfortunately for Intel, the Athlon was a much better processor than the P4 and Intel had to scramble hard to try to make up the ground (not to mention a lot of dirty tactics and FUD). They did of course but it took them years to do it. If AMD hadn't been there we'd probably still be using P4 based processors with small speed bumps every year. Competition drives technology...

Beyond

Oh yah, I'm older than dirt. I remember back when dirt first came out, you know dirt was clean back then.

A 4 line BBS was impressive in those days and would have been impossible on one machine without OS/2. As far as I remember the only guys running that many lines on DOS or Win were using one machine per line. At its zenith mine was fed via satellite downlink on sideband through a C band TV receiver. Uplink was dialup through the local fido hub. There were maybe 75 of us doing the sat downlink thing in North America (it wasn't available in Europe or the southern hemisphere) and everybody except me and one other OS/2 user needed a separate machine just to handle the data off the sat. They used Chuck Forsburg's gz to fire the data out the parallel port and over to the machine the BBS ran on. Ethernet just wasn't popular amongst the hobbyists back then. For me it all ran on 1 machine stable as a rock under OS/2. But OS/2 was Big Blue and everybody loved to hate Big Blue so they wouldn't buy in.

Competition drives technology...

That was a good reminisce about Intel vs. AMD, thanks. Competition does drive technology and I wish I could follow up with another reminisce on precisely that topic but I can't so I'll tell a story about how competition drives people nuts.

Recall I was running the CoCo (Color Computer) about the same time friends were running 8088/86 PCs. Mine ran at ~2.8 MHz, their PCs ran at... ummm... what was it... 8 MHz? 10 MHz? Anyway, the point is that the 6809 executes an instruction every 1/4 clock cycle (characteristic of all Motorola chips of that era, perhaps even modern ones) so my cheap little CoCo could pretty much keep up with their 8088/86 machines which execute every an instruction every cycle. (Yes, I'm over-simplifying here as some instuctions require more than 1 cycle or 1/4 cycle). Then one of those 500 MHz bump ups came out and they all forked out the big money for the faster chip and called me to bring my CoCo over so they could show me who was boss dog. Little did they know I had stumbled upon and installed an Hitachi variant of the 6809 that ran at ~3.2 MHz as opposed to the stock Motorola chip at 2.8 MHz and had soldered in a new oscillator to boost that up to ~3.5 MHz. Lol, their jaws dropped when we ran our crude little test suite for my humble little CoCo still kept up with their expensive PCs. Then they started to argue amongst themselves and accuse their ringleader of feeding them BS about the effectiveness of their expensive upgrades to the faster CPU. Oh they were were going to write Intel and tell them off and one said they had obviously gotten some bogus Chinese knockoffs and yada yada yada. I didn't tell them about my upgrade for a week, just let them stew in their own juices because you know, timing is everything. Then I told them and they settled down. They hated it but they settled down, hehehe, the calm before the storm and I let that ride for a week. Then I told them *my* upgrade had cost me about $12 which was the truth and then their jaws hit the floor again as their upgrades had cost.... oh I forget exactly but it was $200 for sure, maybe more. Back then $200 was a lot of money so out came that unfinished letter to Intel and more yada yada yada and steam blowing. Yah, competition drives people nuts, lol.

---

BOINC <<--- credit whores, pedants, alien hunters

Is that what's happened with the 64 bit ops on nVIDIA cards?

In short: no.

Thanks. I snipped the details for brevity but rest assured they help me fill in the puzzle. Much appreciated.

---

BOINC <<--- credit whores, pedants, alien hunters

...competition drives technological advancement.

While I don't doubt this statement, I think that better and better (GPU-based) supercomputers will be built whether there is or isn't competition in the (GP)GPU industry. Moreover the GPU based products made for gaming purposes become less and less important (i.e. profitable) along the way.

With more than one competitor GPUs will obviously progress far faster than in a monopolistic scenario.

Once again: I don't doubt that. What I was trying to say is that nowadays the need for better supercomputers and the potential of the semiconductor industry drives more the progress than competition.

We've seen technology stagnate more than once when competition was lacking. I could name a few if you like. Been building and upgrading PCs for a long, long time. Started with the Apple, then Zilog Z80 based CPM machines and then the good old 8088 and 8086 CPUs...

Your sentences insinuating that there was stagnation in computing technology since the invention of the microprocessor because of lacking competition. I can't recall such times despite I'm engaged in home and personal computing since 1983. As far as I can recall, there was bigger competition in the PC industry before the Intel Pentium processor came out. Just to name a few PC CPU manufacturers from that time: NEC, IBM, Cyrix, SGS Thomson, and the others: Motorola, Zilog, MOS Technology, Ti, VIA. Since then they were 'expelled' from this market.
Nowadays the importance of the PC is decreasing, since computing became more and more mobile, so PC is the past (including gaming GPUs), smartphones and cloud computing (including supercomputers) are the present and the near future, maybe the smartwatches are the future, and nobody knows what will be in 10 years.

i think this will help you settle your psu issue and
http://images10.newegg.com/BizIntell/tool/psucalc/index.html?name=Power-Supply-Wattage-Calculator
CPU: Intel Core I7(lga2011)
Motherboard: High-end Desktop Motherboard
Video Card: NVIDIA GeForce GTX680 x2
Memory: 4GB DDR3 x4
Optical Drive: Combo x1
Hard Drive: 10,000RPM 3.5" HDD x1

Total wattage: 809 Watts

and with oc i wouldnt go less than the 850 which in my opinion is already pushing it. I would go to a 900 watt and you could build that same computer yourself for $3500 so if you were budgeting 5k you would have much room for improvement.

Also a closed loop liquid cooler usually never has problems, most on the market except a few are made by asetek and just rebadged as another brand. (AMD, Antec, ASUS, Corsair, ELSA, NZXT, Thermaltake, and Zalman) which is cheeper to just go through the manufacturer yourself.

and if you stick to air cooling try using Open Hardware Monitor to ramp up your nvidia fans to keep them cool. I use it to keep my GTX570 at full load under 70c at all times. when i didnt enable manual fan control and let the card do its thing it would peak at 92c under full load. and depending on how loud i want my computer i can choose to keep it at 57c full load with ambient air temp of 32c.

For my hardware I get "Our recommended PSU Wattage: 431 W", which is not actually factoring in the modest OC on CPU and GPU. Crunching POEM the machine consumes 205 W, measured at the wall. Running GPU-Grid (quite taxing) and 7 Einsteins on the CPU I reach 250 W, give or take 10 W.

Sure, a 430 W would be sufficient for my setup (actually I'm using a 500 W Enermax 80+ Gold). But as you see they're calculating very generously, as in my case the maximum PSU usage would be about 60%.

MrS

---

Scanning for our furry friends since Jan 2002

I'm a former GPUGrid cruncher who just wanted to add my voice to those supporting OpenCL use here.

I got rid of my nVidia cards due to many of the newer AMDs having such good electrical efficiency. I've been supporting GPUGrid through donate@home, but making bitcoins isn't quite the same as doing scientific research, and only a limited number of bitcoins can be made, so donate's appeal diminishes over time. Maybe some of the fellowships funded by donate could be used to port GPUGrid to OpenCL before the bitcoins run out altogether?

I just think GPUGrid's missing out, and wish the project would do something about it. It was my original card crunching project, and if there stops being any reasonable way for me to crunch for it, even indirectly, I'm gonna be bummed. Not bummed enough to spend $1000 replacing my cards with nVidias, but bummed nonetheless.

I'm a former GPUGrid cruncher who just wanted to add my voice to those supporting OpenCL use here.

I got rid of my nVidia cards due to many of the newer AMDs having such good electrical efficiency.

If you have an HD 7000 series card, then you are in luck on Folding@home. Their new Core_17 (currently in beta testing) does better on AMD than Nvidia, and improvements are still being made. Not only is the PPD better, but the CPU usage is comparably low, which is unusual for OpenCL. Note however that the higher-end cards do much better than the low-end cards, due to their quick-return bonus (QRB).

You can wait for the release, or try out the betas by setting a flag in the FAHClient to get them. CUDA has long reigned supreme over there, and so it is quite a change. Note however that you have to log in to their forums to see the beta testing section, which is at the bottom of the page. And you will need to check the Wiki to see how to set the beta flag. Given the amount of development required to get OpenCL to work properly (they have been at it for years), that will get you results far faster than waiting for GPUGrid to do it.

I'm a former GPUGrid cruncher who just wanted to add my voice to those supporting OpenCL use here.

I got rid of my nVidia cards due to many of the newer AMDs having such good electrical efficiency.

If you have an HD 7000 series card, then you are in luck on Folding@home. Their new Core_17 (currently in beta testing) does better on AMD than Nvidia, and improvements are still being made. Not only is the PPD better, but the CPU usage is comparably low, which is unusual for OpenCL. Note however that the higher-end cards do much better than the low-end cards, due to their quick-return bonus (QRB).

You can wait for the release, or try out the betas by setting a flag in the FAHClient to get them. CUDA has long reigned supreme over there, and so it is quite a change. Note however that you have to log in to their forums to see the beta testing section, which is at the bottom of the page. And you will need to check the Wiki to see how to set the beta flag. Given the amount of development required to get OpenCL to work properly (they have been at it for years), that will get you results far faster than waiting for GPUGrid to do it.

Hi, Jim1348:

I cannot find any mention of a likely implementation date for the new Core_17 at Folding@home. Have I missed the implementation date or has it not yet been given us?

Regards,

John

John,

Folding does not give out implementation dates, as a matter of policy. But they first started a closed beta (only to registered beta testers who had a key) a couple of months ago, and now have moved to open beta, meaning anyone can do it. But still only the registered beta testers get help if it crashes. I am not a registered beta tester, but it has worked fine on my HD 7770 for several days now and I don't see many problems in the forums (except for the predictable ones from overclocking; don't do it, especially in a beta test).

Next, they move to the "advanced" stage, meaning it is out of beta and anyone can get it if they set an "advanced" flag (and get help if they need it). Finally, the do a full release, and anyone gets it, at least if they have an AMD card; I don't know if they will give it to the Nvidia crowd, who may stay on the present Core_15 for some time longer.

They are still making changes and speed improvements:
http://folding.typepad.com/news/2013/03/sneak-peak-at-openmm-51-about-2x-increase-in-ppd-for-gpu-core-17.html
So I don't know when it will get out of beta, but I would expect in a month or two. And then another month or two of Advanced. If that sounds like fun to you, give it a try; otherwise just wait for the formal release, which I assume will be by the latter part of the summer.

Last month Intel started supporting OpenCL/GL 1.2 on it's 3rd and 4th generation CPU's, and their Xeon Pi. They even have a Beta dev kit for Linux. It's my understanding however that if you are using a discrete GPU in a desktop you won't be able to use the integrated GPU's OpenCL functionality (might be board/bios dependent), but it's available in most new laptops.
AMD's HD7000 series are OpenCL1.2 capable and have been supported from their release.
To date NVidia is stuck on OpenCL 1.1, even though the GTX600 series cards are supposedly OpenCL1.2 capable. They haven't bothered to support 1.2 with drivers. I was hoping that the arival of the Titan would prompt NVidia to start supporting 1.2 but it hasn't happened so far. Perhaps the official HD 7990's will encourage NVidia to support OpenCL1.2, given AMD's embarrassingly superior OpenCL compute capabilities.

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

To date NVidia is stuck on OpenCL 1.1, even though the GTX600 series cards are supposedly OpenCL1.2 capable. They haven't bothered to support 1.2 with drivers. I was hoping that the arival of the Titan would prompt NVidia to start supporting 1.2 but it hasn't happened so far. Perhaps the official HD 7990's will encourage NVidia to support OpenCL1.2, given AMD's embarrassingly superior OpenCL compute capabilities.

Conversely, AMD has made great strides in OpenCL for their 7xxx series cards. On the WCG OpenCL app even the lowly HD 7770 is twice as fast as as an HD 5850 that uses twice the power, so 4x greater efficiency. At the Open_CL Einstein, the GTX 660 is faster than the 660 TI. In fact the 560 TI is faster than the 660 TI. Seems strange?

Edit: I won't even mention that at Einstein the Titan runs at only 75% of the speed of the HD 7970, which is well under 1/2 the price. Oops, mentioned it...

Conversely, AMD has made great strides in OpenCL for their 7xxx series cards. On the WCG OpenCL app even the lowly HD 7770 is twice as fast as as an HD 5850 that uses twice the power, so 4x greater efficiency. At the Open_CL Einstein, the GTX 660 is faster than the 660 TI. In fact the 560 TI is faster than the 660 TI. Seems strange?

Edit: I won't even mention that at Einstein the Titan runs at only 75% of the speed of the HD 7970, which is well under 1/2 the price. Oops, mentioned it...

Pls correct me if I'm wrong:
AMD cards are stable, run more advanced software, are faster, use less energy and are cheaper.

Not so long ago a mod posted here, the reason why GPUGRID does not use AMD is time of the programmers and a resource problem.
OK, we need to accept that.
But maybe it's time to think about alternatives that reflect the reality.
First nVidia 7xx card are to be shipped, AMD's HD8xxx is under way. I think in a month or so we will see first results that makes a choice easier.

AMD's have been looked at several times in the past. The project wasn't able to use them for research in the past, but did offer an alternative funding use for these GPU's. Not everybody's cup of tea, but ultimately it was good for the project. I expect new code will be tested on new AMDs. If the project can use AMD GPU's it will, but if not it won't. They are the best people to determine what the project can and can't do, and if they need help, assistance or advice, they have access to it.

If the time is right for the project I would suggest diversifying into other areas of GPU research to accommodate existing or near future AMD GPU resources.

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Pls correct me if I'm wrong:
AMD cards are stable, run more advanced software, are faster, use less energy and are cheaper.

While I like AMD GPUs and especially the "new" GCN architecture, I think this goes too far.

The AMDs generally provide a bit more raw horse power at a given price point, but not dramatically so. I also don't see a general power consumption advantage sine nVidia introduced Kepler. And stable? Sure, given the right software task.

But saying "run more advanced software" makes it pretty difficult. They support higher OpenCL versions, sure. This is likely not an issue of hardware capability but rather nVidia not allocating ressources to OpenCL drivers. This won't change the fact that currently nVidias can not run the "more advanced" OpenCL code.. but I'm pretty sure that anything implemented there could just as well be written in CUDA. So can the GPUs run that ode or not? Really depends on your definition of "that code".

So it really boils down to OpenCL versus CUDA. CUDA is clearly more stable and the much more advanced development platform. And nVidias GPUs perform rather well using CUDA. Switch to OpenCL, however, and things don't look as rosy any more. There's still a lot of work on the table: drivers, SDKs, libraries etc. But they're doing their homework and progress is intense. It seems like regarding performance optimizations nVidia has fallen behind AMD here so far that it starts to hurt and we're beginning to see the differences mentioned above. Some of that may be hardware (e.g. the super-scalar shaders are almost impossible to use all the time), some of it "just" a lack of software optimization. In the latter case the difference is currently real.. but for how long?

MrS

---

Scanning for our furry friends since Jan 2002

While I like AMD GPUs and especially the "new" GCN architecture, I think this goes too far.

MrS

Latest discussions at Einstein made it necessary te recheck my sight of things.
The performance chart I was referring showes the performance with cuda32 apps. So I was compairing latest version openCL against (outdated) cuda, which does not reflect the actual reality.
When Einstein switches over to cuda 5x things might look very different and the high-score table will look different.

Mea culpa. I apologize for that.

Alex