There is an ongoing problem with CUDA tasks and the Clock Rates being dropped in Nvidia I've written it up and the problem is with Vista and Win7 both.

What happens, is the clock rate gets dropped to conserve power/heat etc. and never returns to high speed again. This always happens with DUAL Nvidia cards installed and seems only magic prevents it from happening on it's own most of the time. Doesn't matter what power settings are set, performance mode seems to help, but not totally correct it. Snoozing or Suspending tasks is a 95% guarantee the clocks with drop and never regain full speed again.

I've informed Nvidia tech support and the forums.


http://forums.nvidia.com/index.php?s=9f29a996e0ac9d6ea44a506f6631f805&showtopic=200414&pid=1237460&st=0&#entry1237460

8-)

Tex1954

As I've posted in reply to your identical post at SETI:

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

@ project devs:

It's been isolated to a CUDA task exit handling problem in the BOINC API library code. Eric Korpela (SETI@home) has a copy of the proposed (and tested) solution for evaluation, and is considering checking it in following testing and evaluation. I can put you in touch with the developer concerned, or pass messages, if you wish.

As I've posted in reply to your identical post at SETI:

This problem has recently been drawn to the attention of the SETI CUDA developers, and is in the process of being reported onwards to other CUDA-enabled BOINC projects. It seems to be related specifically to the release of nVidia drivers which will support the forthcoming v4 release of the CUDA run-time support files. The central BOINC library code isn't yet fully compatible with CUDA v4: the problem has been overcome with test programs, and should go away with the next round of application releases.

Unfortunately, v3 and earlier nVidia drivers aren't available for the very latest generation of nVidia cards, but if you have a card which can run with nVidia driver 266.58, that should avoid the problem.

@ project devs:

It's been isolated to a CUDA task exit handling problem in the BOINC API library code. Eric Korpela (SETI@home) has a copy of the proposed (and tested) solution for evaluation, and is considering checking it in following testing and evaluation. I can put you in touch with the developer concerned, or pass messages, if you wish.

Hi
I have had the same problem with various cards . the new beta 270 driver seems to have fixed the problem. [I hope ]
Cheers
Ross*

---

Thanks for the update. I had a watch on the Nvdia forum but never got an alert.

I have been testing the new Beta 275 drivers and have the same problems with them on 3 different pairs of cards...

Hopefully, we can get this resolved.

Again, thanks for the update!

:)

Tex1954

Has anyone tried out how much the downclocking affects runtimes? I caught my GPU at 405 mhz a while ago while it was crunching but the WU it was on seemed to be progressing at a rather normal pace. Could be it had only just downclocked though, pretty sure I snoozed boinc that day and it caused the downclock, just no idea how long it was before I noticed it.

Anyways, it'd be nice if it would be fixed in the next WHQL release. I need the new drivers to run games like the witcher 2 and dragon age 2 properly. Stupid thing is it stays downclocked even if I start gaming.

Has anyone tried out how much the downclocking affects runtimes? I caught my GPU at 405 mhz a while ago while it was crunching but the WU it was on seemed to be progressing at a rather normal pace. Could be it had only just downclocked though, pretty sure I snoozed boinc that day and it caused the downclock, just no idea how long it was before I noticed it.

Anyways, it'd be nice if it would be fixed in the next WHQL release. I need the new drivers to run games like the witcher 2 and dragon age 2 properly. Stupid thing is it stays downclocked even if I start gaming.

You may read this thread at SETI-forum:

http://setiathome.berkeley.edu/forum_thread.php?id=64243


- or my post at this forum:

http://www.gpugrid.net/forum_thread.php?id=2502&nowrap=true#21188

Does anyone know whether this problem has been observed with Linux CUDA drivers - like, perhaps, the 270.4119 release or the 275.09 Beta, both released May 20, 2011?

Does anyone know whether this problem has been observed with Linux CUDA drivers - like, perhaps, the 270.4119 release or the 275.09 Beta, both released May 20, 2011?

Hi, I'm using the 270.41.19 driver GTX 295 with no problem in Ubuntu Natty 11.04. Greetings.

Downclocking has been observed by several disgruntled GPUGrid members under Linux, including myself, several months ago. Previous drivers definitely caused severe downclocking problems for GT240 and similar cards, and it also messed with some Fermi's, but not so much with the more mature GTX200 cards.
I'm not running any Linux systems right now so I can't re-test with more recent drivers. I expect there has been continuous improvements but that it has not been fully resolved.

Still downclocks occasionally if I manually suspend tasks or snooze boinc with the 275.33 WHQL drivers. This is on win7 64 bit.

Does anyone know whether this problem has been observed with Linux CUDA drivers - like, perhaps, the 270.4119 release or the 275.09 Beta, both released May 20, 2011?

I installed the 270.41.19 Linux drivers a couple hours ago on Fedora 14 with GTX 570. I've been alternately suspending/resuming a GPUgrid task to see if it causes down clocking, as suggested in the OP. So far the clocks have remained constant at:

nvclock=742
memclock=1900
processorclock=1484

Is there anything else that causes down clocking that I could test? Is it more prevalent with certain GPUgrid app(s) than others? If so I'll try again with those apps.

Downclocking appears to be caused by perhaps three things:

A defensive mechanism whereby the card prevents itself being damaged from overheating/overvolting/overclocking (Fermi's only). I think it looks for recoverable errors and when too many are spotted the card is downclocked. This actually prevents a lot of task failures.

It sounds like the downclocking as a result of a cuda task exiting problem might be the GPU trying to protect the system from unwanted thread execution. Any thoughts on this?

Environmental settings - Adaptive power mode is in use and kicks in.

The problem is not really the downclocking, it's that the clocks do not automatically rise again. So no matter what the downclock reason, the problem is that the clocks stay down; they should rise when GPU use increases again. With some earlier drivers I found that even a restart did not resolve the problem on occasions, but this depended on driver, OS and card (3 variables). As there have been several different drivers released since this started it's difficult to assess how and when downclocking occurs. The randomness of the downclock period also make finding the problem more difficult; I have seen everything from a downclock of a few seconds to 405MHz to the clocks being stuck at 50MHz after repeated reboots.

I think Boinc versions would not have any impact. WRT GPUGrid we have now moved away from 6.12, just leaving 6.13 but there is still differences in tasks. The tasks that better utilize the GPU are more likely to tax the GPU to the extent that it overheats/draws too much power (you can use GPUZ to see the utilization of the task). Task type should not however influence the likelihood of the GPU downclocking due to using the Adaptive power mode - not recommended, stick to Full power where possible (not available on XP).
I expect task type would not influence any cuda task exiting problem either.

Amazingly this is still an issue. I uninstalled all Nvidia software and re-installed my old reliable 266.58 drivers, but now it's still downclocking randomly. So why is the driver that once worked without ever downclocking anything now screwed up? I think Nvidia is the new ATI with these horrible drivers.