Have just built a new system with a GTX660 with the intention of running GPUGRID (encouraged by another system with a GTX660 that is working well). So far all 4 WUs that I have got start processing but fail after between 2 and 10 minutes. I think they have all had the "Driver has recovered after stopped working" message in Windows which I guess is an indication that the GPU hardware has failed. Both the old and new cards are factory overclocked (old one 1006 (1072 boost), new one 1033 (1098 boost). Memory on both is 1502.3 (which I think is not overclocked). GPU Core clocks reported by GPU-Z when running are 1162.7 (new) and 1123.5 (old) respectively. The new 660 is fine running PrimeGrid (two concurrent - pegged at 99% busy according to GPU-Z). Environmentals on both systems seem fine (temp about 60 degrees).

I tried reducing the clocks on the new card to the same level as the old one (1006 (1072 boost)). Still failed (but did run for nearly 10 mins - longer than the others).

So does this look like I just need to keep backing off the factory overclock, or is there anything else to look at? Here are the two systems:

old (working) system: 145220
new (failing) system: 155065

Thanks..

Thanks Mark,
All my systems have both ATI and NVIDIA cards and this hasn't caused me issues in the past (at least not like this!) so far, and all of the drivers are explicitly downloaded and installed. One difference is that the working system has 310.90 while the failing system has 314.07. I might try the older driver on the new system and see if that helps.

I'd suggest drivers. I notice the failing system has an ATI as well as Nvidia card. From memory you had to install ATI driver first followed by Nvidia. Also get drivers from Nvidia.com or GeForce.com and do clean install, don't rely on windows to install drivers.

The system is Intel chipset.

Just tried with card set to reference clocks (980/1033). No help - WU crashed in less than 2 minutes.

Will try to install different drivers and see whether that helps...

Driver resets are a known issue in windows that few people know about. There is a simple regedit that should cure the driver timeouts. I have used it on all of my ATI cards and it has worked perfectly. I don't see any reason why it shouldn't work on Nvidia cards too. I posted it once before but I don't know if anyone here tried it. I can post it again if you'd like to try it. The regedit will not do anything else to your system if it doesn't work.

Well it's looking like hardware. I swapped the two cards and the original (that works fine in the old system) is working fine in the new system, while the new card (that was failing in the new system) is also failing in the old system (errored the GPUGRID WU that was in process within a minute or two).

...Now to try to explain this to try to organise a replacement...

Thanks to all for your help.

Driver resets are a known issue in windows that few people know about. There is a simple regedit that should cure the driver timeouts. I have used it on all of my ATI cards and it has worked perfectly. I don't see any reason why it shouldn't work on Nvidia cards too. I posted it once before but I don't know if anyone here tried it. I can post it again if you'd like to try it. The regedit will not do anything else to your system if it doesn't work.

Well it's looking like hardware. I swapped the two cards and the original (that works fine in the old system) is working fine in the new system, while the new card (that was failing in the new system) is also failing in the old system (errored the GPUGRID WU that was in process within a minute or two).

...Now to try to explain this to try to organise a replacement...

Thanks to all for your help.

Thanks - I'll certainly try to negotiate something like that (this card is even failing when I set it back to reference clocks!). The reason I got the card was not because of its overclock, but because it has good cooling

Interestingly the card has actually started working now, but I think this is related to the type of WU: all of the failing WUs were NOELIA_klebe which generate TDP % well into the 90's according to GPU-Z). wThe WU that is working OK (so far) is a NOELIA-1MG which GPU-Z reports as 98% GPU busy but TDP % is only in the mid-60's. Also looks like this will take twice as long to run as the _klebe_'s - no way to get the 24 hour bonus here! For comparison, TDP % is around mid-70's running a pair of PRIMEGRID PPS Sieves concurrently.

@nanoprobe - I'll try the timer delay reqistry setting - thanks. Only some of the failures have been accompanied by the "Stopped Responding" message, however - the rest have just quietly stopped.

Is this a machine you are dedicating to GPUGRID? You may have a hardware issue, but seeing as Windows 7 is the least efficient OS for the project and you are having potential issues with the software, I would encourage you to try Linux. This would help narrow down if it is a software or hardware issue, and if it succeeds, you will have a very efficient dedicated machine for GPUGRID.

Yes the memory load was 1% (also on the WU running now on the "good" 660 - it's been running for almost 24 hours and is saying it is 59% complete). Looks like you have seen or heard of this before?!

The WU eventually failed after mor than 1000,000 seconds with:

The WU eventually failed after mor than 1000,000 seconds

Whoops my mistake - should have been 100,000 secs. Sorry..

Hi MrS,
I ran the test from the free version of 3DMark 11. It ran fine with results in the midrange of those with same processor and graphics card. Also ran FurMark. It also seemed to run fine. It actually ran with TDP approaching 110%, and you could see that the card had reduced both core clock and voltage. Temps stabilised at 68 degrees with factory fan settings. From memory this would appear to be stressing the card more that GPUGrid was doing when the WUs failed.

Just read the thread "Current Noelia WUs" and that explains some of what I am seeing (log-running Noelia_MGs) but of course my 660s are 2GB. Does explain the fact that the Noelia_MGs took twice as ling to complete on my (1 GB) 560 ti.

I tried uninstalling the drivers (again) and reinstalled (310.90). Fetched another WU - got a Noelia_klebe. It actually ran for 15 minutes before crashing (maybe a bit of FurMark helped burn it in :-) ). Environmentals looked OK (certainly much less that FurMark) - GPU 93%, TDP 90%, Mem Controller 34%, Temp 65 degrees.

Then I turned off long tasks and got a short WU. It also crashed, just shy of 10 minutes 145220

I'm just about out of ideas here..

Driver resets are a known issue in windows that few people know about. There is a simple regedit that should cure the driver timeouts. I have used it on all of my ATI cards and it has worked perfectly. I don't see any reason why it shouldn't work on Nvidia cards too. I posted it once before but I don't know if anyone here tried it. I can post it again if you'd like to try it. The regedit will not do anything else to your system if it doesn't work.

Sure, post it, and thanks in advance.

I'm trying another tack here - wonder if there is some bad memory on the card? After the short WU failed, I underclocked the memory a bit. The next short WU worked OK! OK there were a couple after that did not, but they have failed on other machines also. I've lowered the memory clock a bit more and will see how it goes. I've found a few memory testing tools which I will try also - if there's a memory problem, hopefully I can find a way to reliably show it that will allow the seller to verify the problem.

I notice that there are two BIOS versions for this card - one for cards with Hynix memory, the other for cards with Samsung memory. My card appears to be Samsung.

Ran some memory tests (memtestcl and memtestG80) with no errors (but probably need to run for a day or two to do a thorough check). Then set the card to reference GPU core clock (980/1045), and memory underclocked from 6008 to 5494. It has processed a NOELIA klebe successfully, and is now nearly 2 hours into a NOELIA 7MG (that is working properly with 38% memory controller load).

I also installed the 320.49 driver but this doesn't appear to have made any difference one way or the other (WUs were still crashing before I underclocked the memory).

Hm thats normal here. If wus begin to fail overvolt by 0.025V and underclock the mem by 100mhz was the recommend solution here in the forum. And it really works for me since a longer time now.
____________
DSKAG Austria Research Team: http://www.research.dskag.at

Driver resets are a known issue in windows that few people know about. There is a simple regedit that should cure the driver timeouts. I have used it on all of my ATI cards and it has worked perfectly. I don't see any reason why it shouldn't work on Nvidia cards too. I posted it once before but I don't know if anyone here tried it. I can post it again if you'd like to try it. The regedit will not do anything else to your system if it doesn't work.

Sure, post it, and thanks in advance.

I run 320.18 and 320.49 with the GTX660's and the errors are Noelia's or a short run from Santi. Almost none Nathan's error out.
Some WU's have a MCU load from 1% and took 50 hours on my rig, but without error. I guess its luck, if the Noelia's finish or error. I don't make all the fuss with the drivers as other projects WU's seem to do well with them.
____________
Greetings from TJ