Devs,

If a NOELIA short run task is running, then:
- when I exit BOINC, or
- I suspend the task
...
it often crashes the NVIDIA driver, and leads to Computation Errors on tasks that are running across all GPUs, causing me to lose work, even from other projects.

This sounds very similar to what was happening when NOELIA tasks were in Beta.
Could you please investigate, and see if you can reproduce the issue?
Again, this is causing me to lose work for other projects. :(

Windows 8 x64, BOINC 7.0.60 Beta, nVidia 314.22 WHQL, GTX 660 Ti (usually runs 2 GPUGrid tasks), GTX 460 (usually runs 2 World Community Grid HCC tasks)

Thanks.

You figured out why some of my machines are crashing. I'll make sure the task doesn't suspend.

-- Craig

... GTX 660 Ti (usually runs 2 GPUGrid tasks)

How do you get a GTX 660 TI to run TWO GPUGrid tasks??

How do you get a GTX 660 TI to run TWO GPUGrid tasks??

You use an app_config.xml file.
Jacob

it often crashes the NVIDIA driver, and leads to Computation Errors on tasks that are running across all GPUs, causing me to lose work, even from other projects---------
I have exactly the same problems, always few times per week/monts nvidia drivers crash-blue screen on my win 8 64bit and I can not get over 620k rac-two months. Before I attacked 650k and climb higher on same HW configurations
I've tried everything and the problem is clearly in favor of NVIDIA and GPUGRID. All the problems started about two months ago when all the people seeking solutions to find why they have problems in the noelia Tasks and cuda 4.2 .. the inability make available counting on TITAN nvidia cards probably just confirms problems with nvidia and GPUGRID. Or they have enough people to count the project..

GPUGrid.net Devs:

This is still a big problem for me.

If NOELIA tasks are suspended (which happens for me a lot because I have several <exclusive_app> settings configured in cc_config.xml).... It often crashes the driver, which crashes the game/program I'm about to run too!

The error is:
Display driver stopped responding and has recovered

You should be able to easily reproduce this by letting a NOELIA task run for a bit, then hit the Suspend button. Do that over and over, 30 times, and see if you get any errors.

My only workaround right now is, if I know I'm going to be suspending the GPU because I want to run a certain application, I have to suspend it manually, to let it crash the driver, before I run the application. You have made it so I cannot use <exclusive_app> or <exclusive_gpu_app> effectively anymore.

PLEASE FIX THIS!
- Jacob Klein

I have this exact same problem. My specs are
Windows 8x64
2 GTX 670
314.07 WHQL driver
Boinc version 7.0.64 x64

My cpu is an Intel i7-3820 overclocked to 4.5 GHz which runs 5 WU's of World Community Grid and the rest of the cores left to power the GPU's.

Today I replaced my GTX 460 with a GTX 660. My first WU is a Noelia, which looks like it will complete in 12 hours; 25% done in three hours. Much better!

I added the app_config.xml file posted here, which BOINC included in its startup.

I am disappointed not to have seen a second WU running yet. Any thoughts?

BOINC log below.


____________

Today I replaced my GTX 460 with a GTX 660. My first WU is a Noelia, which looks like it will complete in 12 hours; 25% done in three hours. Much better!
I added the app_config.xml file posted here, which BOINC included in its startup.
I am disappointed not to have seen a second WU running yet. Any thoughts?

You didn't say what you have in your app config, just "posted here". I don't see it in this thread at least.

Apparently you're trying to run 2 WUs concurrently. If so, they won't make the 24 hour deadline. The new NATHANS are even longer. Are you trying to increase your credit? Even if they run without problem, you will end up with lower credit than running 1X on a GTX 660.

Apparently you're trying to run 2 WUs concurrently. If so, they won't make the 24 hour deadline. The new NATHANS are even longer. Are you trying to increase your credit? Even if they run without problem, you will end up with lower credit than running 1X on a GTX 660.

Ah! That's not what I had understood: that 50% + 50% = 100% but no bonuses... I just wonder why there has been so much kerfuffle here on a 'feature' (2x) that benefits no-one.

tomba,

To answer you question about the app_config, the

tomba,

To answer you question about the app_config, the

<gpu_usage>1</gpu_usage>

statement tells BOINC how many gpu's to use for each task. Currently it is set to use 1 full GPU for each task so only 1 task will run on each gpu at a time. If you set it to

<gpu_usage>0.5</gpu_usage>

that would tell BOINC to use half of a GPU for each task which would allow 2 tasks to run on each GPU.

Be sure to post your test results so we can see if it helped.

I will certainly report back on results!

tomba,

When comparing "before" and "after", to see how much faster the tasks are processed, assuming the tasks have the same amount of work to be done in them, then... it's best to look at the "Run Time" value of the results, after they're done.

Also, this thread is about NOELIA tasks crashing. I'm interested in your results, but I have created a thread that documents the performance testing/results of app_config changes; could you please post to it instead? It's called "GPU Task Performance (vs. CPU core usage, app_config, multiple GPU tasks on 1 GPU, etc.)", and is located here: http://www.gpugrid.net/forum_thread.php?id=3331

I'd like to keep this thread focused on the NOELIA problems, which crash the drivers. It's an ongoing issue, and I'm hoping the admins will please look into it :(

Regards,
Jacob

Developers:

Has anything been done about this?

NOELIA tasks are still, when suspended, sometimes:
- crashing my drivers
- yielding computation errors even for other GPU tasks
- crashing the whole OS with DPC Watchdog Timeout errors

Please fix this obnoxious behavior!
I originally posted this thread over 6 weeks ago.
Where is the response?

Devs,

If a NOELIA short run task is running, then:
- when I exit BOINC, or
- I suspend the task
...
it often crashes the NVIDIA driver, and leads to Computation Errors on tasks that are running across all GPUs, causing me to lose work, even from other projects.

This sounds very similar to what was happening when NOELIA tasks were in Beta.
Could you please investigate, and see if you can reproduce the issue?
Again, this is causing me to lose work for other projects. :(

Windows 8 x64, BOINC 7.0.60 Beta, nVidia 314.22 WHQL, GTX 660 Ti (usually runs 2 GPUGrid tasks), GTX 460 (usually runs 2 World Community Grid HCC tasks)

I will forward it. From a quick forum search it seems to be W7/W8 and driver related. So there might not be much we can do. Are you certain it only happens with Noelias and no other WUs?
Unfortunately you need to keep in mind that at this point there are no dedicated GPUGrid (software) "developers". Our manpower is very limited and the user base very big. This means that with all the hardware/software combinations that work in GPUGrid there are bound to be problems for which we do not have the manpower to fix. We also prefer to dedicate more time on science for obvious reasons.
However since this seems to happen to a few users I will pass it along to MJH who might know something more. Let's hope we find a fix.

Guys,

Is the crash on the suspend or the restart? Do you have "keep application in memory when suspended" set? What if you change to the alternative?

Matt

Starting to get a few funky NOELIA tasks as well. GTX 580 + GTX 670, win7 x64,
boinc 7.0.64

http://www.gpugrid.net/result.php?resultid=6900910
http://www.gpugrid.net/result.php?resultid=6902281
http://www.gpugrid.net/result.php?resultid=6891033

Heck even a Nathan.

http://www.gpugrid.net/result.php?resultid=6899851

Any thoughts?

JugNut,

This thread is focusing on the issue "NOELIA tasks - when suspended or exited, often crash drivers", trying to reproduce it, and trying to fix it.

For your other issues, please consider creating a separate thread.

Thanks,
Jacob

For me driver restart happens everytime with the current Noelias (or better: I have not observed it not happening) but not with other WUs. Win 8 drivers 320.18 and 314.22 (the last 2 WHQLs), "leave apps in memory" active (but I read it doesn't apply to GPUs, as it would be far too risky to leave something dirty or to run out of memory) and a GTX660Ti (Kepler).

There's been some discussion whether disabling the driver watchdog helps. Simply increasing the timer didn't help for me (the screen just kept freezing longer), whereas SK said it did help in his case. Another user said disabling the watchdog altogether would help, but I haven't tried this myself.

MrS

To answer your questions from earlier Jacob, we have not been able to reproduce the error, unfortunately. We only have one box running right now testing Windows 7, and we have not received a NOELIA since those tasks are now dwindling in number. We of course don't doubt that it is real, considering so many people confirming it, we just haven't been able to troubleshoot it yet. Even so, if it is being caused by the driver watchdog or some other Windows bug, we might not be able to do much about it. It will be interesting to see if it still occurs with the watchdog disabled. That should tell us a lot.

Thank you for responding, Nate.

I doubt the issue is a Windows bug or nVidia bug, since it only happens on those NOELIA (klebe) tasks, but I guess you never know. My understanding is that the bug isn't being caused by a driver watchdog; the bug is causing the driver watchdog to be tripped.

Since you think it may be valuable information, to you, to see if it occurs with the watchdog disabled, then... I will invest effort into knowing how to do that with Windows 8, so that I can be prepared to do additional testing when I get one of these units again. http://msdn.microsoft.com/en-us/library/windows/hardware/ff570088%28v=vs.85%29.aspx appears to have very useful information, including registry key settings in some of the child nodes in the tree at the left... that should also hopefully be applicable to Windows 8.

You had mentioned previously, I thought, that you do have a way to test work units before issuing them even on the beta application. If possible, you might take some of those NOELIA (klebe) tasks, and run them (and suspend them several times) locally on that Windows 7 box, in a non-Production environment. That may prove to be beneficial to the user base.

Again, I'll do what I can to provide more information to you.

Thanks,
Jacob

NOELIA tasks are still very much an issue for me.
Suspending them, or closing BOINC, results in graphics driver resets nearly every time.

I wish there was a way to opt not to receive them.
They are that difficult to work with.

The klebe ones give me the most trouble.
Can anything be done?

Windows 8.1 Preview x64
nVidia GeForce 326.01 x64 Beta
GTX 660 Ti
GTX 460

Do not suspend, don't exit BOINC, haha. ;)

Nope I don't know, but I recognize your story/experiences. A suspend will also lead to a driver error at my own system. So that's why I'm touching nothing in BOINC when it's crunching @ GPUGRID. :P

Since I cannot trust any GPUGrid.net units to shutdown gracefully anymore, here has been my workaround:

- Set cc_config.xml to stop computation while certain apps/games are running, using <exclusive_app> entries.
- Before I launch an exclusive app/game, right-click the BOINC Manager tray icon, and select "Snooze GPU".
- Wait 10 seconds. Sometimes all 3 of my GPUs will crash, and I'll get 3 separate TDR errors with flickering Windows, resulting in Windows balloons saying that the GPU driver has crashed and restarted. Sometimes I get none. But waiting 10 seconds is how long I have to wait to find out.
- If there were driver crashes, shutdown any GPU monitoring software, and restart that software (since the values are messed up from the driver crashing)
- Now launch the app/game. Because "Snooze GPU" keeps them snoozed for 1 hour, and because I have <exclusive_app> entries in place, I can be sure that the computation will not resume my app/game is running
- When I'm done with my app/game, BOINC will resume crunching on its own (after the 1-hour timeout has expired from the "Snooze GPU" command).

This is VERY UNFORTUNATE that I have to do this tedious workaround any time I want to use my GPU.

Has GPUGrid.net made ANY PROGRESS into finding out the cause of these driver crashes when GPUGrid.net tasks are suspended or shutdown on Windows??

- Jacob

This is VERY UNFORTUNATE that I have to do this tedious workaround any time I want to use my GPU.

Has GPUGrid.net made ANY PROGRESS into finding out the cause of these driver crashes when GPUGrid.net tasks are suspended or shutdown on Windows??

Hm, good to know.
But it is a Windows GPUGrid problem and not generally a problem all GPU Boinc projects have on Windows, right?

I greatly appreciate the stability my machine has had over the past couple weeks, due to not suspending any NOELIA tasks.

When she gets back, please consider investigating what causes the driver reset and watchdog timeouts, when NOELIA tasks are suspended or BOINC is exited while one is running. I believe some exit logic in the code is not returning quickly enough.

Thank you,
Jacob Klein

His suggested fix is to disable TDR, which I use for games and for other GPU applications. I rely on it. So, no, I didn't try it.

So far as I know, the bug is in the NOELIA tasks.

Strange, that's never happened to me and I've suspended them dozens of times.

Thank you a million times over for setting aside some time to solve this.
I am ecstatic - cannot wait to test your change!

When it's ready, please let us know which version of the app to use, and which task types to look for to test against.

Thank you,
Jacob

Try out 8.02. Give it a damn good suspending.

MJH

Try out 8.02. Give it a damn good suspending.

MJH

Frequency is quite hard to conclusively prove.
I'll admit, though that it feels like it is crashing less when suspending single tasks.
So, I think you're heading in the right direction, but have more work to do.

The main crashes I'm seeing now are when I choose "Snooze GPU" from the system tray; I still *sometimes* get a driver reset when it tries to suspend 2 running NOELIA tasks.

Are there any more critical sections that need specified?

I've got 2 WU's which I couldn't start anymore because as soon as I resume them the nVidia Driver will crash.

http://www.gpugrid.net/workunit.php?wuid=4864368
http://www.gpugrid.net/workunit.php?wuid=4862898

This one also just crashed with "computation error":

http://www.gpugrid.net/workunit.php?wuid=4856494

Actually half of the WU's I tried to do crashed... :(

Hype,

This thread (titled: When suspended or exited, often crash drivers) discusses an issue that occurred when tasks were suspended or BOINC was exited normally, where the drivers would often crash. So far as we all know, recent GPUGrid application versions have actually fixed the issue in this thread.

I assume you have a different error. Could you please read this other thread, http://www.gpugrid.net/forum_thread.php?id=3491, as it describes a currently-open issue that might explain the behavior you are seeing. If your issue is still different, then please open a new thread.

Thanks,
Jacob

Unclear if these are crashing the driver; there is no message saying that it has.

Task http://www.gpugrid.net/result.php?resultid=7393220

Several of these errors, usually with the screen going black for about a second:

10/23/2013 6:21:36 PM | GPUGRID | Task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 exited with zero status but no 'finished' file
10/23/2013 6:21:36 PM | GPUGRID | If this happens repeatedly you may need to reset the project.
10/23/2013 6:21:36 PM | GPUGRID | Restarting task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 using acemdbeta version 814 (cuda42) in slot 1
10/23/2013 6:22:41 PM | GPUGRID | Task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 exited with zero status but no 'finished' file
10/23/2013 6:22:41 PM | GPUGRID | If this happens repeatedly you may need to reset the project.
10/23/2013 6:22:41 PM | GPUGRID | Restarting task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 using acemdbeta version 814 (cuda42) in slot 1

I reset the project as indicated. The task then disappeared from my computer.

Also, a number of the recent NOELIA workunits on this computer have given error -97:

http://www.gpugrid.net/result.php?resultid=7393428
http://www.gpugrid.net/result.php?resultid=7393415
http://www.gpugrid.net/result.php?resultid=7393189

SWAN : FATAL : Cuda driver error 999 in file 'swanlibnv2.cpp' in line 1963.
ACEMD beta version v8.14 (cuda42)


and one SANTI_MAR workunit:

http://www.gpugrid.net/result.php?resultid=7392716

Also error -97.
This one might be from overheating the GPU. No overclocking done.
Short runs (2-3 hours on fastest card) v8.14 (cuda42)

More of the same type of problem, on a different NOELIA task.

http://www.gpugrid.net/result.php?resultid=7393628

I've noticed that the problem occurs most often when I do something
that affects most of the screen, such as opening or closing a program
that uses most of the screen - I see the scrren going black for about
a second.

I aborted this task.

Could be a driver issue, try updating to the latest (beta) driver.
____________
Greetings from TJ

Unclear if these are crashing the driver; there is no message saying that it has.

Task http://www.gpugrid.net/result.php?resultid=7393220

Several of these errors, usually with the screen going black for about a second:

10/23/2013 6:21:36 PM | GPUGRID | Task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 exited with zero status but no 'finished' file
10/23/2013 6:21:36 PM | GPUGRID | If this happens repeatedly you may need to reset the project.

Unclear if these are crashing the driver; there is no message saying that it has.

Task http://www.gpugrid.net/result.php?resultid=7393220

Several of these errors, usually with the screen going black for about a second:

10/23/2013 6:21:36 PM | GPUGRID | Task trypsin_lig_12_2-NOELIA_RCrep_eq-0-1-RND5589_0 exited with zero status but no 'finished' file
10/23/2013 6:21:36 PM | GPUGRID | If this happens repeatedly you may need to reset the project.

I couldn't find the beta test drivers. However, when the 331.58 and 331.65 drivers came out, I installed them. No such crashes under either of these.

Crashes less frequent with the 331.65 driver, but not gone. GPU workunits for other BOINC projects still running properly.

Another -97 error:
http://www.gpugrid.net/result.php?resultid=7410463

Another repeated Windows crashes until I aborted the workunit:
http://www.gpugrid.net/result.php?resultid=7410730

That computer now has GPUGRID on No new tasks. I plan to leave it there until I install some update that looks likely to fix this problem.

My other computer (with a slower GPU, Windows 7, and probably an older driver) still has GPUGRID enabled since it isn't showing showing this problem.

some update that looks likely to fix this problem.

some update that looks likely to fix this problem.

??? explain please ::grabs popcorn::

some update that looks likely to fix this problem.

??? explain please ::grabs popcorn::

I'm waiting for an Nvidia driver release, a BOINC release, or a GPUGRID application release before I enable GPUGRID workunits on that computer again.

some update that looks likely to fix this problem.

??? explain please ::grabs popcorn::

I'm waiting for an Nvidia driver release, a BOINC release, or a GPUGRID application release before I enable GPUGRID workunits on that computer again.

I've now installed BOINC 7.2.28 on the computer with the problem. I tried installing the 331.82 Nvidia driver a few times; it never installed correctly. I'm back to the 331.65 driver.

GPUGRID workunits have run properly on that computer for the last few days, with no more driver crashes.

I've been trying to crash NOELIA tasks on my 660T1 by repeatedly (20 X) suspending and resuming them but I can't. This seems like a Windows bug to me. Suggest installing Linux to fix it or a script that peeks at the names of the GPUgrid tasks in your cache every 60 secs and aborts them if they're NOELIA. Either way would be a path of lesser resistance leading to greater productivity and oneness with the Buddha ;)

____________
BOINC <<--- credit whores, pedants, alien hunters

The 8.14 application version fixed the problem that this thread described. It has been resolved for a while now..

But I couldn't crash them with the previous version either.

____________
BOINC <<--- credit whores, pedants, alien hunters