I am currently running a Long-run task called "901x-SANTI_MARwtcap310-2-32-RND7131_0" using the "cuda55" plan class, using the 8.15 app version, on my Kepler GTX 660 Ti, using new drivers: 334.67 BETA, on Windows 8.1.

I expected Task Manager and Process Explorer to both show that this task would make the "acemd.815-55.exe" process fully utilize a virtual CPU core. However, it is only using a tiny portion (~25%) of the core. Furthermore, this GPU is sometimes not jumping to boost clock while this task is running, and is instead sometimes staying at standard 3D clock; utilization is at 85% with no other tasks running, and I expect the behavior is related to the CPU usage issue.

So... to my questions....
Did something break how SWAN SYNC is automatically selected? Why is the process not using a full virtual core on my Kepler GPU? Is the app broken, or is the driver broken, or is this behavior expected? Are certain GPUGrid tasks intentionally set up to not use a full core on Kepler GPUs?

Thanks,
Jacob

Isn't it just the same issue we have been discussing in the other thread?
http://www.gpugrid.net/forum_thread.php?id=3561&nowrap=true#34709

Unless there is something very different about Win8.1, it is probably just due to the work unit (or portions thereof) being more difficult than the card can handle, so it down clocks. (I assume that affects the CPU too, but have never investigated that to any extent.)

The usual suspects to change are:

Increase power limit
Reduce GPU/Memory clocks
Increase GPU core voltage

I have seen this problem a number of times on GTX 660s and 650 Ti, and have them all running perfectly now by doing the above.

Jim,

This is a different issue altogether. For some reason, tasks on Kepler GPUs are no-longer utilizing the full CPU core.

My questions remain. I welcome answers.

Hello , I only crunch here part time and am not a super tech. I did notice though that you and Tomba (the other user who noticed same ) use the same beta driver. I have not upgraded my 770 or 780 yet(still 331.82) and still have the standard cpu usage on my recent tasks . So perhaps the driver ??

Rob

I am thinking it might be a conflict between the driver and the 8.15 GPUGrid application. I'm going to do some testing tonight, by installing the prior driver version, to prove/verify that.

We still will need someone from GPUGrid to pinpoint the exact nature of the problem (if it is a problem?), though. I can't report an issue to NVIDIA unless someone at GPUGrid confirms that the driver has a problem.

GPUGrid admins? Any input?

---------------------------------
skgiven:

I fully understand that, the way we are used to seeing it working, is that the GTX 660 Ti (a Kepler card) will use a full core (shown as 12.5 CPU in both Task Manager and in Process Explorer), whereas my GTX 460 does not (it uses about 3.25 CPU).

What I'm saying is that, with the 334.67 BETA drivers, the behavior has changed. With 334.67, all of the tasks use 3.25 CPU, even those running on the Kepler.

I've verified that 332.21 WHQL drivers exhibit the "normal" behavior, whereas the 334.67 BETA drivers exhibit this "new" behavior. Can you verify the new behavior using the new drivers?

So, again, my questions remain:
Did something break how SWAN SYNC is automatically selected? Why is the process not using a full virtual core on my Kepler GPU? Is the app broken, or is the driver broken, or is this behavior expected? Are certain GPUGrid tasks intentionally set up to not use a full core on Kepler GPUs?

---------------------------------
MJH?
GPUGrid Admins?
Is something not working as expected?

I had this issue with older drivers on WinXPx64 (and on WinXPx86 too), but only with NOELIA_DIPEPT-0-2 workunits:
Task 7717507, 7718044, 7717962, 7717686, 7718047, 7717487.
The really strange part is that the two workunits following task 7717507 were ok without any intervention: task 7720966, 7722153
All of my hosts processed the subsequent NOELIA_DIPEPT-0-2 workunits normally.

For me, my issue seems related to the new driver version, and not batches of tasks.

Another odd thing I see in the runtimes when I read Tomba`s post on the server board(?) and checked his times. Post driver change cpu runtime is about 75% less but gpu run times pre & post driver change are about the same on similar wus.

It is odd that with 75% less cpu usage the completion time would stay about the same.

JK are you seeing same?

It's going to be hard for me to determine if the task run time is affected. I stop and start tasks many times before they are completed, and I have 2 heterogeneous GPUs, the 660 Ti and the 460. So, the same task usually has a portion of it ran using the 660 Ti, and a portion ran using the 460. So, I will likely be unable to determine if the new behavior of "Tasks are not using a full CPU on Kepler GPUs using 334.67 drivers" is going to affect my task run times.

But the behavior is new/different, and I'm still hoping that skgiven can confirm the behavior, and that someone can answer the questions, especially if it is expected behavior or not.

Thanks for testing, skgiven. I'm glad you reproduced the behavior.

I am still requiring the following questions to be answered:
1) Is it an actual problem, or is it correct behavior?
2) If it is a problem, is the problem in the driver, or is the problem in the application code that determines how the acemd process functions?

Can anyone close to the acemd 8.15 application answer those questions?

Thanks,
Jacob

I know the app hasn't changed, but the driver has. Even if this means a problem has surfaced, we don't know whether the problem is in the app or in the driver.

I'll monitor my results, to at least try to get a feel of whether it affects my GPUGrid performance on my GTX 660 Ti.

I'm hoping an admin can chime in, so that we can determine if it's a bug or not, and if it, where the bug is.

Thanks for the information. I have put out a request for more info from NVIDIA, in their 334.67 driver feedback thread.

My post is here:
https://forums.geforce.com/default/topic/679611/geforce-drivers/official-nvidia-334-67-beta-display-driver-feedback-thread-released-1-27-14-/post/4112315/#4112315

Phenomenal information, skgiven, thanks for sharing. I love the angle of "perhaps it's a fix instead of a bug" :)

This has prompted me to change my app_config.xml cpu_usage values. I had been using 0.5, such that when a GPUGrid task was running on each GPU, and I knew a full core would be used because of the Kepler, the summed values would be 0.5+0.5=1.0 CPU, so BOINC considered a CPU used by the GPU tasks, and did not over-commit my system. But now, with these 334.67 drivers, I'm using a cpu_usage value of 0.2, in order to keep my system full committed. Essentially, this results in my system being able to run an additional CPU task!

I'll chime back in if I hear anything from NVIDIA.

Hello: Possibly it is the same issue that is sucediento in Linux for a month.

The low use (12% + -) of the CPU but with a performance almost equal to when the load was 100% of one CPU per GPU.

see:http://www.gpugrid.net/forum_thread.php?id=3601

Carlesa, did your change in GPUGrid task behavior... coincide with an updated driver version? Here in the Windows world, we only noticed the new behavior when we started using 334.67 BETA drivers.

Sorry for not providing input on this sooner but I've been swamped with other issues.

I have 2 Linux rigs crunching GPUgrid. One has a 670 and a 660Ti in it (shows as two GTX 660Ti on the website) with driver version 331.20 and it is showing 99 - 100% CPU usage on each of 2 tasks. Those are real cores, no HT on that CPU.

The other rig has one GTX 670 with driver 331.38 and it is showing 9 - 11% CPU usage. Again those are real cores, no HT on that CPU. I don't think 331.38 is a beta driver but I could be wrong.

Take this report with a bit of caution because something weird that I don't understand is going on. Due to events I'm not going to bother explaining because they're long and complicated, the rig with the older driver should actually have the older driver so something is fishy here bit I'm not sure what it is, yet. I'm either confused or NVIDIA released a driver and then retracted it or something. I'll get back with more details when I know more but I hope what I have provided so far sheds some light. I would be very happy to learn that we're getting the same production with less CPU time.

____________
BOINC <<--- credit whores, pedants, alien hunters

Carlesa, did your change in GPUGrid task behavior... coincide with an updated driver version? Here in the Windows world, we only noticed the new behavior when we started using 334.67 BETA drivers.

Carlesa, did your change in GPUGrid task behavior... coincide with an updated driver version? Here in the Windows world, we only noticed the new behavior when we started using 334.67 BETA drivers.

Hi, The difference, as commented on the information is in the version of Linux you use Ubuntu 13.10 or 14.04 of the Nvidia driver is the same in both cases see. 331.38.

The purpose of the "SWAN_SYNC" knob is to set the CUDA runtime to use a low-CPU mode. In practice, this hasn't worked very well for quite some time (in terms of driver releases). Looks rather like "correct" behaviour is restored with version 334.

You should see lower CPU load, without diminished GPU performance.

MJH

Thank you for performing these tests, skgiven. I consider them very useful, and look forward to your additional results. It would be difficult for me to perform such tests, since I'd have to disable work on a heterogeneous GPU in order to ensure tasks stay on the Kepler. But I will surely use your results to modify my configuration appropriately.

So far, my configuration has been changed a little. I still run GPUGrid-only on my GTX 660 Ti and my GTX 460. And I run Albert/Einstein/SETI/Beta (A/E/S/B) only, on my GTS 240.

Old config: Set GPUGrid tasks to 0.5 CPU, since when running 2, I for sure was using a core on the Kepler. Set A/E/S/B to 0.5 CPU for most of their apps, since they don't use a full core. Use 100% CPUs. Result: 7 CPU tasks, 2 GPUGrid tasks, 1 A/E/S/B task; CPU slightly overloaded.

Current config: Set GPUGrid tasks to 0.4 CPU, to better reflect values I'm seeing. Set A/E/S/B to 0.3 CPU for most of their apps. Note: 0.4+0.4+0.3=1.1. Use 100% CPUs. Result: 7 CPU tasks, 2 GPUGrid tasks, 1 A/E/S/B task; CPU very slightly underloaded.

Note: There is 1 A/E app that I set to 1 CPU, "Gamma-ray pulsar search #2". And there is 1 S/B app that I set to 1.0 CPU, "AstroPulse v6". Process Monitor shows that those apps actually use a full core, which is why I have them setup to use 1.0 CPU. When one of those tasks run, the result is: 7 CPU tasks, 2 GPUGrid tasks, 1 1-core A/E/S/B task; CPU moderately overloaded.

Regards,
Jacob

I have been running the 334.67beta on WinXP (32-bit) long enough to complete a Long on both a GTX 660 (SANTI_MARwtcap310) and a GTX 650 Ti (SANTI_MAR422cap310). It seems fine on GPU usage, being its usual 97% on the 660 and 98% on the 650 Ti. The run times also seem identical to the previous driver (332.21); that is, they are no faster.

The only real difference is the CPU usage. It is now down to 17% on the 660, and 12% on the 650 Ti (as measured by BoincTasks). That is very nice, since that motherboard is an older P45 with an E8400 Core2 Duo at 3.0 GHz. The next step will be to try to run a single WCG project also on the CPU. We will see about that.

I expect XP is different to W7 because the GPU utilization is higher on XP.

Zoltan, it occurred to me that the difference might actually be the CPU projects you were running; I'm just running WCG task that set the app priority to Idle. Many projects use higher settings, such as Normal.

For those who may be interested I deployed 334.67 BETA drivers to my small machine with two gtx 650ti's.

Agree swan-sync appears to be enabled now as CPU usage dropped to 8% yet the run time for a long task appear to take only 2% longer - a good trade off for the reduced power usage, not to mention my GPU temps dropped a few degrees.

http://www.gpugrid.net/results.php?hostid=166813

I set Boinc to run with an 'above normal' priority using: http://www.efmer.eu/boinc/download.html

Incidently my machine is running headless; on this occasion I was able to install the drivers through an rdp session - though they do not take effect(as the cards cannot be seen). Hence once rebooted I just updated the cards directly through device manager.

Measured the power draw at the wall with the 334.67 beta drivers on XP x86 and its about 8% less (176w) than the previous whql drivers (192w) whilst performance only dropped by about 2% - a good result in my opinion.

Just wanted to chime in that, after repeated pressure by me, ManuelG is going to find more information on what has changed in the driver that has affected Kepler CPU usage. He hopes to have such information within 48 hours.

https://forums.geforce.com/default/topic/679611/geforce-drivers/official-nvidia-334-67-beta-display-driver-feedback-thread-released-1-27-14-/post/4119864/#4119864

ManuelG said he had to file a bug, to have the devs even look at it.
https://forums.geforce.com/default/topic/690370/geforce-drivers/official-nvidia-334-89-whql-display-driver-feedback-thread-released-2-18-14-/post/4127820/#4127820

I tried to stress that it may not be a bug at all, and that we are simply looking for confirmation that something changed, and more information about that change:
https://forums.geforce.com/default/topic/690370/geforce-drivers/official-nvidia-334-89-whql-display-driver-feedback-thread-released-2-18-14-/post/4128112/#4128112

I'll let you guys know when I know more.

Hello: In Windows 8.1 since I installed the latest Nvidia driver (a couple of days) 334.89 behaves like Linux, (for some time now) the CPU load is reduced to 10-20% (varies according to the task type) instead of 100% previously.

The overall performance of the GPU hardly altered and the change improves load / performance / consumption ratio.