PAOLA_3EKO_8LIGANDS very low GPU load

Message boards : Number crunching : PAOLA_3EKO_8LIGANDS very low GPU load
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
klepel

Send message
Joined: 23 Dec 09
Posts: 189
Credit: 4,798,881,008
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26676 - Posted: 23 Aug 2012, 20:57:23 UTC - in response to Message 26674.  

I've seen people advise this action here and at other BOINC forums and it seems to me that this would never work because telling BOINC to use 6 of 8 cores or 7 of 8 cores takes them away from all projects. I would think you would want to set CPU usage at you're GPUgrid account, by taking away cores in BOINC, only the operating system or programs not connected to BOINC can utilize those cores.

I don't think the prefrences in our GPUGRID account allows for enough minipulation of the CPU to make it do what you want.


Does work for me! I set max CPU utilization to 99% on my AMD FX8150, and on the 7 of the 8 cores crunch climateprediction.net WUs and one core makes my GTX670 happy. Nice side effect my system is more stable and does not hamper workflow that much.

Under Windows Work Administrator/Process (ctrl alt delete) I can see that all the cores are used to their maximum by BOINC (13% with about 70,000 to 130,000 KB Memory utilization for CP and 9% and 192,000 KB for this EKO – PAOLA WUS we are talking about (for NATHAN-WUs this is normally 13% as well and about 230,000KB)).
ID: 26676 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
flashawk

Send message
Joined: 18 Jun 12
Posts: 297
Credit: 3,572,627,986
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 26677 - Posted: 23 Aug 2012, 21:38:12 UTC - in response to Message 26676.  

That just don't make no sense at all. There is no way in BOINC to allocate cores to particular work units, that setting is for freeing up CPU power for the OS. If you're wu's use less than .5 CPU power, you wont see issues, anything over that and you have to suspend wu's. I don't know why I'm responding to this, I feel like I'm walking into another one.
ID: 26677 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Snow Crash

Send message
Joined: 4 Apr 09
Posts: 450
Credit: 539,316,349
RAC: 0
Level
Lys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26678 - Posted: 23 Aug 2012, 21:46:26 UTC

I've set my GPUGrid preferences to use 100% CPU for graphics but I *think* this refers to how much CPU to use for displaying a project's screensaver ... I'm going to do a quick taks switch to see ...

GPU = GTX 480 Win7 x64, shaders @1512, mem @1848
CPU = Core i7-980 CPU clocked at 4.050 HT = ON
RAM = 6GB triple channel @ 1500 DDR3
BOINC = 7.0.28 set to use 100% of processor
because I am runing both a GTX670 and a GTX 480 BOINC forces so a full thread dedicated to GPU.
OTHER PROJECTS = 11 threads to WCG


NATE -- GPU @95-96% utilization, MCU @24-25%
PAOLA - GPU @59-61% utilization, MCU @6%
- occasionally GPU jumps to 70+%

I'll let this run for a while to get a good estimate of runtime to compare to previous run that I'll double check but I think it took 14 hrs.

side note a PAOLA too 25 hours on the 660Ti



Thanks - Steve
ID: 26678 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26679 - Posted: 23 Aug 2012, 22:29:27 UTC - in response to Message 26678.  
Last modified: 23 Aug 2012, 22:32:36 UTC

You are correct, these CPU preferences are just for the screen saver and nothing to do with how much CPU is used to support a GPU.

When people set Boinc to use 6/8 threads this means Boinc will use 6 CPU cores/threads to crunch CPU project, and it means the remaining 2 can be used for GPU projects (which are not considered as CPU projects, in this respect).

We need to better isolate this problem.
Is the problem that some tasks just run slow on all cards randomly (a task issue) or just GF600 cards?
Does the number of available CPU cores influence the issue on all cards, or just some?
Is the CPU type of importance?
Do these tasks require increased disk read/write (SSD vs standard SATA HDD), or high memory I/O?
Does changing the Boinc write to disk settings make any difference? (Tools, Computer Preferences, Disk and memory usage, tasks checkpoint every..., ->900sec)
Are CPU tasks causing issues?
It might be worth looking at the bus utilization on different boards (PCIE3/2/1.1).
Boinc versions might be worth noting, and operating system XP/Vista/W7/Linux.

So post up a few more details and we might be able to narrow it down. Also say if Aero is on, or not installed.

My impression is that it's more of an issue with GF600 series. It's as if tasks are running on the 3.1app (which is not compatible with GF600), so perhaps there is some legacy code being used.
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help
ID: 26679 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Snow Crash

Send message
Joined: 4 Apr 09
Posts: 450
Credit: 539,316,349
RAC: 0
Level
Lys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26680 - Posted: 23 Aug 2012, 23:01:13 UTC - in response to Message 26678.  

I've set my GPUGrid preferences to use 100% CPU for graphics but I *think* this refers to how much CPU to use for displaying a project's screensaver ... I'm going to do a quick taks switch to see ...

GPU = GTX 480 Win7 x64, shaders @1512, mem @1848
CPU = Core i7-980 CPU clocked at 4.050 HT = ON
RAM = 6GB triple channel @ 1500 DDR3
BOINC = 7.0.28 set to use 100% of processor
because I am runing both a GTX670 and a GTX 480 BOINC forces so a full thread dedicated to GPU.
OTHER PROJECTS = 11 threads to WCG


NATE -- GPU @95-96% utilization, MCU @24-25%
PAOLA - GPU @59-61% utilization, MCU @6%
- occasionally GPU jumps to 70+%

I'll let this run for a while to get a good estimate of runtime to compare to previous run that I'll double check but I think it took 14 hrs.

side note a PAOLA too 25 hours on the 660Ti



1 hour of processing and it is 4.2% complete
GPU utilization hanging around 70%, GPU mem still at 6%
CUDA = 4.2 app
DRIVER = 301.42
MOBO PCIE2 @8X
All memory stats running nominal differences to NATE
--- (usage, private, pool, paged, non-paged)
Page faults are high (compared to NATE) @ 173k after 1 hour
No hard faults

Let me know if there are any other details I can help with :-)
Thanks - Steve
ID: 26680 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
flashawk

Send message
Joined: 18 Jun 12
Posts: 297
Credit: 3,572,627,986
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 26681 - Posted: 23 Aug 2012, 23:20:24 UTC

I have 3 machines that are identical,

AMD 8150 3.6GHz 8 core
16GB Kingston DDR3 PC1600
Asus M5A97 AM3+
Seagate Constellation Enterprise SATA6 7200RPM 1TB
OCZ 750 watt PSU
Everything water cooled (CPU, Video)

1 rig has 2xGTX670FTW 2GB 1533 shaders GPU usage at 30%
1 rig has 2xGTX560 1GB 336 cuda cores GPU usage at around 40%
1 rig has 1xGTX560Ti 1GB 384 cores GPU usage at 35% and 1xGTX550Ti 192 shaders GPU usage at 55%

All the memory controllers are between 9 and 11%

Windows XP Pro x64 SP2

BTW, all my cards run much cooler, if there's anything I forgot, just ask.

ID: 26681 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Rayzor

Send message
Joined: 19 Jan 11
Posts: 13
Credit: 294,225,579
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 26682 - Posted: 23 Aug 2012, 23:40:53 UTC

From some quick testing on my part.

GTX 480, i3 530, 4GB ram, Windows 7 Ultimate, using the late swan_sync parameter, and with one core free, the GPU usage had fallen to around 40% to 50% usage. BOINC CPU usage was set at 75%.

The other 3 cores were busy with Seti, and 1 core was dedicated to the GPU, and Task Manager showed the CPU was at 100% usage.

After freeing another one of the cores on the i3 530(2 cores plus 2 hyper threaded cores, total of 4 cores), and setting the acmed.win.23 Priority to High, the GPU has remained rock steady, at 93% to 94% usage.

Task Manager CPU usage is now showing appox. 81% usage. This is with 2 cores on Seti, 1 core for GPU, and now 1 core free for the PC.

For me BOINC CPU usage has to be set at 50% or less, in order for high GPU usage.
ID: 26682 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Luke Formosa

Send message
Joined: 11 Jul 12
Posts: 32
Credit: 33,298,777
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwatwatwat
Message 26683 - Posted: 24 Aug 2012, 7:32:40 UTC
Last modified: 24 Aug 2012, 7:35:49 UTC

Regarding the setting I mentioned ("on multiprocessors use at most xxx% of processors" - which I set to 87.5 to use 7 out of 8 cores), that only applies to tasks that use the CPU exclusively (like WCG). Tasks that use the GPU ignore that setting - they simply use as much GPU as possible, and the associated amount of CPU needed. For GPUGrid, that's 1 task per GPU (since each task uses 1 NVIDIA GPU), and hence 0.585 CPU cores per GPU task. For something like Einstein@Home, which uses 0.5GPUs per task, it runs 2 GPU tasks simultaneously, and consumes however much total CPU two Einstein GPU tasks need.

For POEM@Home, each task uses 1CPU+1GPU, so it's 1 cpu core per GPU task (ouch). In each case, GPU projects get to use as much CPU as they need, and it's only the other (non-GPU) projects that BOINC limits to the percentage specified.

Now, as for the Paola_3EKO, my guess is that they're not coded to make full use of the GPU. I would suggest trying to run multiple Paola's on each GPU by making an app_info.xml file. Problem is that since this is a limited run, I don't have any more Paola's to test with. And BOINC discards all tasks in the queue when you introduce an app_info.xml file, so if you're reading this post and you have paola tasks in your queue, you still can't create a custom app_info and try it because you'll lose the tasks currently in the queue...

So looks like this problem will solve itself (because the limited number of Paola 3EKO tasks dispatched will all be completed) before we get the chance to try fixing it.
ID: 26683 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26684 - Posted: 24 Aug 2012, 8:23:13 UTC - in response to Message 26680.  

Page faults are high (compared to NATE) @ 173k after 1 hour


Suggests that something is not being kept in memory that should be, and is repeatedly being read from disk (which would obviously be a Lot slower). Maybe this is a CPU process for the GPU? Having an SSD would mask this to some extent - you would experience the same issue but not as severely.

Having more RAM available or faster RAM might also reduce this somewhat, but it sounds like a systemic issue. Sort of explains why Luke only noticed an increase from 45% to 52% when they stopped running all CPU tasks.

The more GPU's a system has the more this is a problem, and the more CPU projects are running (generally) worsens it. On a 12thread system I wouldn't use more than 10, if supporting 2GPU's. Very much depends on the CPU project too; some CPU projects eat memory (and 6GB isn't enough) while others use 10 to 100MB. Some also have extremely high I/O requirements. Both RAM shortages and high I/O are known to negatively impact upon GPU projects.

If I had 4 GPU's in a system, I probably wouldn't run any CPU projects.
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help
ID: 26684 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile dskagcommunity
Avatar

Send message
Joined: 28 Apr 11
Posts: 463
Credit: 958,266,958
RAC: 31
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26685 - Posted: 24 Aug 2012, 8:27:06 UTC - in response to Message 26673.  

does anyone has default max 20% CPU time for GPU work set on the websiteprofiles too like me until today? perhaps this value is to low for these new units? i set it to 100% and waiting now until i get a new one of this WUs and finished it.


I feel it must be something like this, because there are some users who can compute much faster than the rest (and at speeds we were expecting). Keep us updated dskagcommunity.

If anyone else wants to play with the setting, click on your username up above, then "GPUGRID preferences". "Edit Preferences", and change "Maximum CPU % for graphics..." to 100% (or whatever you prefer).

Still, this might not be it. Wouldn't explain this, though, unless the cards are on different machines with different settings...

These workunits do not use a full CPU core with Kepler GPUs, unlike any previous workunits. It's like the late swan_sync parameter wasn't set to 0. These workunits run twice as fast on my GTX 480s than on my GTX 680s.


Let's see...


so it doesnt had any effect. :/

DSKAG Austria Research Team: http://www.research.dskag.at



ID: 26685 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Luke Formosa

Send message
Joined: 11 Jul 12
Posts: 32
Credit: 33,298,777
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwatwatwat
Message 26686 - Posted: 24 Aug 2012, 9:42:01 UTC - in response to Message 26684.  

Having more RAM available or faster RAM might also reduce this somewhat, but it sounds like a systemic issue. Sort of explains why Luke only noticed an increase from 45% to 52% when they stopped running all CPU tasks.


If it helps, I have 8GB RAM (of which BOINC is allowed to use 90%, or 7GB, when the computer is in use) running at 1600 MHz. I have all of Windows, and the BOINC executable, on a 128GB SSD, but my BOINC data folder is on a 2TB hard drive (both drives use the motherboard's two SATA 6GB/s ports that come from the Z77 chipset). The HDD does 150 MB/s in HDtune and the SSD does 400 MB/s.

I'd tell you my page and hard faults, but I'm not being given Paola tasks - they seem to have run out and we're back to Nathan tasks. Right now with 8 WCG tasks and one GPUgrid nathan running, I'm seeing 0 hard faults per second and 56% physical memory utilisation. Gpugrid made 156,000 page faults in 35 minutes. Just to put that in context, flash player made 25 million page faults in 7 minutes of CPU time (about 3 hours of youtube videos...).
ID: 26686 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Snow Crash

Send message
Joined: 4 Apr 09
Posts: 450
Credit: 539,316,349
RAC: 0
Level
Lys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26688 - Posted: 24 Aug 2012, 21:53:24 UTC

I don't have time do all the stats but on Win7x64 core i7-920 HT ON, 6 GB ram, Boinc 7.25, ALL CPU tasks suspended my GTX 670 runs at about 45%
When I run with only 1core free it drops to about 32%
Thanks - Steve
ID: 26688 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile vitalidze

Send message
Joined: 15 May 09
Posts: 20
Credit: 239,712,351
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26694 - Posted: 25 Aug 2012, 11:12:14 UTC

i5 2500k/8Gb 1333Mgh/Asus p67/GF680GTX Win7 Ultimate 304.48
all in default 38-42% gpu utilization
ID: 26694 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Luke Formosa

Send message
Joined: 11 Jul 12
Posts: 32
Credit: 33,298,777
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwatwatwat
Message 26706 - Posted: 25 Aug 2012, 23:20:07 UTC

Possible solution - see post #2 in this thread:

http://www.gpugrid.net/forum_thread.php?id=3118
ID: 26706 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile nate

Send message
Joined: 6 Jun 11
Posts: 124
Credit: 2,928,865
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 26716 - Posted: 27 Aug 2012, 17:52:28 UTC

Well, I think I've found the general cause, although I can't say I have a solution yet. When I run the workunits on our machines, NOT via BOINC, the simulations use 100% of the CPU. When I run the PAOLA WUs via BOINC, the max I get is ~50% CPU usage. No doubt that's where the 2x slowdown comes from. Now, why that happens, I have to find out. I'll ask the more technical people here and hopefully have answers soon, but if anyone knows why CPU usage is limited to 50% via BOINC, feel free to explain. Is this something common for other GPU based tasks, or specific to us?...
ID: 26716 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile dskagcommunity
Avatar

Send message
Joined: 28 Apr 11
Posts: 463
Credit: 958,266,958
RAC: 31
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26717 - Posted: 27 Aug 2012, 18:09:43 UTC

Perhaps it would be enough when the project use 1 cpu insteed 0,65?
DSKAG Austria Research Team: http://www.research.dskag.at



ID: 26717 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Luke Formosa

Send message
Joined: 11 Jul 12
Posts: 32
Credit: 33,298,777
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwatwatwat
Message 26718 - Posted: 27 Aug 2012, 19:04:51 UTC - in response to Message 26716.  

The CPU utilisation is about 7-10% on my 8-core CPU (so that's 50-75% of one of the cores), but I think you guys made it that way by design because it uses 0.585 or 0.65 CPUs (can't remember because no running any right now).

DistRTGen has about as much CPU utilisation as GPUGrid. POEM@Home uses an entire CPU core for every running GPU task.

GPUGrid's Nathan tasks still use 7-10% CPU but achieve a GPU load of 90-95% on my GTX 670.
ID: 26718 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Snow Crash

Send message
Joined: 4 Apr 09
Posts: 450
Credit: 539,316,349
RAC: 0
Level
Lys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26750 - Posted: 30 Aug 2012, 22:04:24 UTC - in response to Message 26716.  

Well, I think I've found the general cause, although I can't say I have a solution yet. When I run the workunits on our machines, NOT via BOINC, the simulations use 100% of the CPU. When I run the PAOLA WUs via BOINC, the max I get is ~50% CPU usage. No doubt that's where the 2x slowdown comes from. Now, why that happens, I have to find out. I'll ask the more technical people here and hopefully have answers soon, but if anyone knows why CPU usage is limited to 50% via BOINC, feel free to explain. Is this something common for other GPU based tasks, or specific to us?...

Not sure how relevant this is but back quite a while ago we were using the SWAN_SYNC environmental variable to tell ACEMD to fire off a process that used a full CPU to poll the GPU rather than waiting for the GPU to make a call and the inherant latencies that involved ... then we were told that we no longer needed SWAN_SYNC as that was now baked directly into ACEMD. Perhaps that was done through a configuration mechanism in the WU generation process that got missed this time around?
Thanks - Steve
ID: 26750 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Niels Kornoe

Send message
Joined: 29 Aug 08
Posts: 1
Credit: 133,018,242
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 26756 - Posted: 2 Sep 2012, 7:42:01 UTC

Are I'm the only one who abort every "PAOLA_3EKO" task I get?

My GPU goes below 400MHz and the task time goes up to ~ 18H.
That is more than double the time it should take, and I could have done more than 2 "NATHAN_RPS1120801" tasks in that amount of time.
ID: 26756 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Luke Formosa

Send message
Joined: 11 Jul 12
Posts: 32
Credit: 33,298,777
RAC: 0
Level
Val
Scientific publications
watwatwatwatwatwatwatwat
Message 26760 - Posted: 2 Sep 2012, 9:18:35 UTC - in response to Message 26756.  

Are I'm the only one who abort every "PAOLA_3EKO" task I get?

My GPU goes below 400MHz and the task time goes up to ~ 18H.
That is more than double the time it should take, and I could have done more than 2 "NATHAN_RPS1120801" tasks in that amount of time.


You really shouldn't do that. Remember that they rely on you and other volunteers to do the crunching for their research. If everyone aborted certain kinds of tasks, they'd never get any research done. If you're concerned about low utilisation, I suggest using a custom app_info.xml - I posted about it a few posts ago in tis thread.
ID: 26760 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : PAOLA_3EKO_8LIGANDS very low GPU load

©2025 Universitat Pompeu Fabra