Hello everyone,

I'm creating this thread to document my GPUGrid GPU Task performance variances, while testing things such as:
- GPU task with no other tasks
- GPU task with full CPU load
- GPU task with overloaded CPU load
- Multiple GPU tasks on 1 video card

My system (as of right now) is:
Intel Core i7 965 Extreme (quad-core, hyper-threaded, Windows sees 8 processors)
Memory: 6GB
GPU device 0: eVGA GeForce GTX 660 Ti 3GB FTW (primary display)
GPU device 1: eVGA GeForce GTX 460 (not connected to any display)
OS: Windows 8 Pro x64 with Media Center

So far, I have some interesting results to share, and would like to "get the word out". If you'd like to share your results within this thread, feel free.

Regards,
Jacob

I originally did some performance testing in another thread, but wanted the results consolidated into this "GPU Task Performance" thread.

That thread is titled "app_config.xml", and is located here:
http://www.gpugrid.net/forum_thread.php?id=3319

Note: The post within that thread, which contains the app_config values that I recommend using, can be found here:
http://www.gpugrid.net/forum_thread.php?id=3319#29216

Here are the first results (from running only on my GTX 660 Ti), copied from that thread:

========================================================================
Running with no other tasks (every other BOINC task and project was suspended, so the single GPUGrid task was free to use up the whole CPU core):

Task: 6669110
Name: I23R54-NATHAN_dhfr36_3-17-32-RND2572_0
URL: http://www.gpugrid.net/result.php?resultid=6669110
Run time (sec): 19,085.32
CPU time (sec): 19,043.17

========================================================================
Running at <cpu_usage>0.001</cpu_usage>, BOINC set at 100% processors, along with a full load of other GPU/CPU tasks:

Task: 6673077
Name: I11R21-NATHAN_dhfr36_3-18-32-RND5041_0
URL: http://www.gpugrid.net/result.php?resultid=6673077
Run time (sec): 19,488.65
CPU time (sec): 19,300.91

Task: 6674205
Name: I25R97-NATHAN_dhfr36_3-13-32-RND4438_0
URL: http://www.gpugrid.net/result.php?resultid=6674205
Run time (sec): 19,542.35
CPU time (sec): 19,419.97

Task: 6675877
Name: I25R12-NATHAN_dhfr36_3-19-32-RND6426_0
URL: http://www.gpugrid.net/result.php?resultid=6675877
Run time (sec): 19,798.77
CPU time (sec): 19,606.33
========================================================================

CONCLUSION:
So, as expected, there is some minor CPU contention whilst under full load, but not much (Task Run time is maybe ~3% slower). It's not affected much because the ACEMD process actually runs at a higher priority than other BOINC task processes, and therefor, are never starved for CPU, and are likely only minorly starved for contention during CPU process context switching.

Here are some more results, where I focused on the "short" Nathan units:

========================================================================
Running with no other tasks (every other BOINC task and project was suspended, so the single GPUGrid task was free to use up the whole CPU core):

Task: 6678769
Name: I1R110-NATHAN_RPS1_respawn3-10-32-RND4196_2
URL: http://www.gpugrid.net/result.php?resultid=6678769
Run time (sec): 8,735.43
CPU time (sec): 8,710.61

Task: 6678818
Name: I1R42-NATHAN_RPS1_respawn3-12-32-RND1164_1
URL: http://www.gpugrid.net/result.php?resultid=6678818
Run time (sec): 8,714.75
CPU time (sec): 8,695.18

========================================================================
Running at <cpu_usage>0.001</cpu_usage>, BOINC set at 100% processors, along with a full load of other GPU/CPU tasks:

Task: 6678817
Name: I1R436-NATHAN_RPS1_respawn3-13-32-RND2640_1
URL: http://www.gpugrid.net/result.php?resultid=6678817
Run time (sec): 8,949.63
CPU time (sec): 8,897.27

Task: 6679874
Name: I1R414-NATHAN_RPS1_respawn3-7-32-RND6785_1
URL: http://www.gpugrid.net/result.php?resultid=6679874
Run time (sec): 8,828.17
CPU time (sec): 8,786.48

Task: 6679828
Name: I1R152-NATHAN_RPS1_respawn3-5-32-RND8187_0
URL: http://www.gpugrid.net/result.php?resultid=6679828
Run time (sec): 8,891.22
CPU time (sec): 8,827.11
========================================================================

CONCLUSION:
So, again, as expected, there is only slight contention while under full CPU load, because the ACEMD process actually runs at a higher priority than other BOINC task processes, and therefor, are never starved for CPU, and are likely only minorly starved for contention during CPU process context switching.

So, previously, I was only running 1 GPU Task on that GPU (and the GPU Load would usually be around 87%-88%). But I wanted to find out what would happen when I run 2.

So, the following tests will use <gpu_usage>0.5</gpu_usage> ... in my app_config.xml.

Note: The GPU Load goes to ~97% when I do this, and I believe that's a good thing!

========================================================================
Long-run Nathan tasks...
Running at <cpu_usage>0.001</cpu_usage>, <gpu_usage>0.5</gpu_usage>, BOINC set at 100% processors, along with a full load of other GPU/CPU tasks:

Name: I19R1-NATHAN_dhfr36_3-22-32-RND2354_0
URL: http://www.gpugrid.net/result.php?resultid=6684711
Run time (sec): 35,121.51
CPU time (sec): 34,953.33

Name: I6R6-NATHAN_dhfr36_3-18-32-RND0876_0
URL: http://www.gpugrid.net/result.php?resultid=6685136
Run time (sec): 39,932.98
CPU time (sec): 39,549.67

Name: I22R42-NATHAN_dhfr36_3-15-32-RND5482_0
URL: http://www.gpugrid.net/result.php?resultid=6685907
Run time (sec): 35,077.12
CPU time (sec): 34,889.61

Name: I31R89-NATHAN_dhfr36_3-21-32-RND1236_0
URL: http://www.gpugrid.net/result.php?resultid=6687190
Run time (sec): 35,070.94
CPU time (sec): 34,901.26

Name: I8R42-NATHAN_dhfr36_3-22-32-RND2877_1
URL: http://www.gpugrid.net/result.php?resultid=6688517
Run time (sec): 32,339.90
CPU time (sec): 32,082.15

========================================================================
Short-run Nathan tasks...
Running at <cpu_usage>0.001</cpu_usage>, <gpu_usage>0.5</gpu_usage>, BOINC set at 100% processors, along with a full load of other GPU/CPU tasks:

Name: I1R318-NATHAN_RPS1_respawn3-11-32-RND9241_0
URL: http://www.gpugrid.net/result.php?resultid=6684931
Run time (sec): 12,032.03
CPU time (sec): 11,959.47

Name: I1R303-NATHAN_RPS1_respawn3-14-32-RND0610_0
URL: http://www.gpugrid.net/result.php?resultid=6690144
Run time (sec): 14,621.04
CPU time (sec): 10,697.88

========================================================================

CONCLUSIONS:

Long-run Nathan units:
1-at-a-time + full CPU load: ~19,600 run time per task
2-at-a-time + full CPU load: ~35,100 run time per task
Speedup: 1 - (35,100 / (19,600 * 2)) = 10.5% improvement

Short-run Nathan units:
1-at-a-time + full CPU load: ~8,900 run time per task
2-at-a-time + full CPU load: ~13,300 run time per task
Speedup: 1 - (13,300 / 8,900 * 2)) = 25.3% improvement

So far, it looks like running multiple tasks at a time... GETS WORK DONE QUICKER!

Now, admittedly, I am estimating on very few results here, but.. I'll continue using this "2-at-a-time" approach, and will reply here if I find anything different.

This is very good info. However, I need to point out a couple potential down-side issues:

1) even with 2 tasks per GPU via app_config.xml, it does not increase the number of tasks you can download. For example, on my 4 GPU machine, it normally has 4 running, and 4 waiting to run. Running 8 at once means all 8 are running. So now there is a delay between the time a task completes, uploads, reports, a new task is downloaded (big file), and starts running. That *may* wipe out any utilization advantage.

2) The longer run-time with 2 tasks per GPU *may* cause them to miss the credit bonus for early returns.

YMMV
____________
Reno, NV
Team: SETI.USA

Point 1: ideally this would average out after some time, so that the different WUs per GPU finish at different times. Depending on your upload speed this might provide enough overlap to avoid running dry. Having more GPUs & WUs in flight should help with this issue.

After some setup difficulties, I now have two long run tasks running - one on each of my GTX 650 Ti GPUs. GPUGrid runs 24/7 on this AMD A10 based PC and there are always two tasks running with either one or two waiting to run. As each GTX 650 processes at a slightly different rate the number of tasks waiting to run varies. I believe this will maximize output from my PC enabling me to make the maximum contribution to the research.

John,

My research indicates that you might be able to contribute more to the project, if you run 2 tasks on each of your GPUs, assuming the tasks don't result in computation errors.

You might try that, using the app_config.xml file, and see if your overall performance increases. I was able to see gains in GPU Load (seen via a program called GPU-Z), as well as increased throughput (seen by looking at task times, as noted within this thread).

Regards,
Jacob

Hi, Jacob.

I am very inexperienced in writing .xml files and fear losing running tasks through syntax errors.

I would like to take it one step at a time for now and, maybe in a couple of weeks, try your suggestion. I will likely ask for help.....

Thanks for the suggestion.

John

No problem. It's really not that hard, so don't be afraid, and... when you're ready, I encourage you to read this entire thread, which has details and examples:

"app_config.xml" located here:
http://www.gpugrid.net/forum_thread.php?id=3319

- Jacob

Sure, in order to get maximum bonus credits, you'll have to be careful to make sure you complete all your tasks within 24 hours. And, in general, they want results returned quickly.

But, in order to help the project the most, throughput (how fast can you do tasks) is the factor to measure, and the "deadline" is the task's deadline, which usually is a few days I think. If the administrators deem that a task must be done at a certain time, then I hope they are setting task deadline appropriately.

Thanks, Gentlemen:

I will leave this alone for now.....

With falling prices for the GTX 660 Ti, I may add one to my other AMD A10 based PC in September around my birthday.

John

Yes, I still have testing to do. You can/should test too!

It's not easy to cherry-pick certain task type combinations -- I usually just let any task types run together. Maybe once I find even more time to test, I'll attempt doing the specific-combination testing, using custom suspending, and more vigilant monitoring.

As far as "freeing up a core", my research indicates that, at this point, doing so is COMPLETELY UNNECESSARY, at least for me. If you look at the acemd processes in Process Explorer, you'll see that process priority is 6, and the CPU-intensive-thread priority is either 6 or 7. This ensures that the thread and process do not get swapped out of the processor, even when I'm running a full load of other CPU tasks, since those CPU tasks are usually priority 1 or 4. Watching how the CPU time gets divvied up (in Process Explorer, or in Task Manager), also proves it -- you'll see the other processes getting less-than-a-core, but you'll see the acemd process "suffer" much. Plus, as you said, sometimes the GPUGrid tasks don't require much CPU at all (like when a NATHAN Long-run is on my GTX 460), so, reserving a core is sheer waste at that point, at least for my goals. So I won't do it.

I'm not trying to speculate here, and I'm certainly not trying to find reasons not to run multiple tasks on the same GPU. I think it's worth it.

What I'm trying to do is show the results that I have achieved, using my goals (maximize throughput for GPUGrid, without sacrificing any throughput for my other projects), and I encourage others to do the same.

Thanks,
Jacob

Sounds good, thanks for testing.

Note: When running 2-at-a-time, I expect tasks to take slightly-less-than-double what they normally take, which would mean they are being processed faster over-all.

Ah, you bring up a good point, I forgot to mention my clocking experiences with my Keplar architecture eVGA GTX 660 Ti 3GB FTW card...

- It's base clock is 1045 MHz, which I think is the lowest clock it can be while running a 3d application or GPU task.
- When GPU Load is not great (~60-75%), I think it usually upclocks a little (maybe up to 1160 MHz), but because it sees the application as "not demanding a lot", it doesn't try hard to upclock.
- When GPU Load is decent-ish (86%), it auto-upclocks a bit (usually to around 1215 MHz or 1228 MHz I think), with Power Consumption around 96-98% TDP.
- When GPU Load is better-saturated (97%-99%), it usually tries to upclock higher, but reaches a thermal limit. It usually ends up clocked at around 1180-1215 MHz, with a temperature of 84*C-89*C, at a Power Consumption around 96%.
- TIP: At that saturation, if you want, you can usually allow it to auto-upclock just a tad more, by using whatever overclock tools you have (I have eVGA Precision X), and just adjust the "Power Target". By default, I think the driver sets a Power Target of 100%, but what I usually do is adjust it to 140%. This let's it auto-clock higher, until it starts really hitting those thermal limits. My end result: My card usually runs at 1215 MHz, 86*C - 90*C, with Power Consumption around 106% TDP.

So, running at higher GPU Load keeps it clocked high, as high as the thermal limits can let it... which is a good thing, if you care more about GPUGrid throughput than the lifespan of your GPU. :)

Regards,
Jacob

Careful, guys. The GTX650Ti (Johns GPUs) sounds like it's almost the same as a GTX660Ti (Jacobs GPUs), but it's actually about a factor of 2 slower. Currently 70k credit long-runs take John 33k seconds, running 2 of them might require ~60 ks. That's almost one day, so we're getting close to missing the deadline for the credit-bonus here for even longer tasks (some give 150k credits, so should take over twice as long).

You are correct that there is a memory concern, thank you for reminding me about this.
My testing indicates that going beyond your GPU's memory limit will result in an immediate task failure for the task being added.

To test/prove this, I wanted to overload my secondary GPU, the GTX 460 1 GB (which is not connected to any monitor).

Below is my setup and my testing results:

I suspended all tasks, suspended network activity, closed BOINC, made a copy of my data directory (so later I could undo all this testing without losing active work), changed the cc_config to exclude GPUGrid on device 0 (the 660 Ti), changed the GPUGrid app_config <gpu_usage> value to 0.25 (yup, this is a stress test), and I started resuming GPUGrid tasks 1 at a time, while monitoring Memory Usage (Dynamic) in GPU-Z.

Here are the scenarios that I saw (had to reset the data directory for nearly each scenario):

Scenario 1:

• Added a Long-run Nathan dhfr, GPU Mem Usage became 394 MB
  
• Added a Short-run Nathan RPS1, GPU Mem Usage increased to 840 MB, no problems
  
• Added a Long-run Nathan dhfr, GPU Mem Usage spiked up to 1004 MB, and the task immediately failed with Computation Error

Scenario 2:

• Added a Short-run Nathan RPS1, GPU Mem Usage became 898 MB
  
• Added a Long-run Nathan dhfr, GPU Mem Usage spiked up to 998 MB, and the task immediately failed with Computation Error

Scenario 3:

• Added a Long-run Nathan dhfr, GPU Mem Usage became 394 MB
  
• Added a Long-run Nathan dhfr, GPU Mem Usage increased to 789 MB, no problems
  
• Added a Short-run Nathan RPS1, GPU Mem Usage spiked, and the task immediately failed with Computation Error

Scenario 4:

• Added a Long-run Nathan dhfr, GPU Mem Usage became 394 MB
  
• Added a Long-run Nathan dhfr, GPU Mem Usage increased to 789 MB, no problems
  
• Added a Long-run Nathan dhfr, GPU Mem Usage increased to 998 MB, and surprisingly, all 3 tasks appear to be crunching OK

CONCLUSIONS:

• If adding a task will put the memory usage beyond your GPU's memory limit, the task being added will immediately fail with Computation Error.
  
• If you look carefully at those Short-run's being added, it looks like they "detect" the available GPU Ram, and run in a certain mode made to fit within that limit. For instance, in Scenario 1, I think the task detected 610 MB free, thought it was a 512 MB card, and limited itself to (840-394=) 446 MB. Then, in Scenario 2, it saw 1004 MB, thought it was a 1GB card, and limited itself to 898 MB. Then, in Scenario 3, there was only 215 MB free, and it couldn't load at all.
  
• If you look at the Long-run's, they follow a similar pattern. In Scenario 1, it saw a 1GB card, but maybe the tasks were built to only ever need 512 MB cards, so it limits to 394 MB. In Scenario 2, it couldn't load within 106 MB. Scenario 4 is interesting, because for the third task, it saw 215 MB free, and was able to run using (998-789=) 209 MB. So it can scale to 256 MB cards perhaps. Looking back at scenario 1, we see that the third task only had 164 MB free, which wasn't enough.
  
• So... I'm going to have to revisit my settings. I'd like to tell BOINC to run 2 tasks at a time on device 0, but only 1 task at a time on device 1... but I don't think that's an option. Another thing I might do is disable certain GPUGrid applications on device 1, such that I can be sure to only double-up ones that I know would work. But, because GPUGrid mixes "types" within a single "app" (like NOELIA and NATHAN, both within the same application), I don't think that's a valid play either. I may just devote device 1 to another project, whilst device 0 is set for 2 tasks. Still deciding, and am open to any suggestions you may have.
  
• Note, this issue hasn't bitten me yet, because device 1 is currently focused on World Community Grid's (WCG) Help Conquer Cancer (HCC)tasks; so GPUGrid tasks haven't doubled up much on my device 1 at all yet, but they will soon, when WCG HCC runs out of work in < 1 month, and I'll need a plan by then.
  

Hope you find this helpful -- it took 40 minutes to test and write up, and I encourage you to do your own testing too!

Regards,
Jacob

Another thing to keep in mind concerning the 2x strategy. These current Nathans are the shortest running long type WUs we've seen in quite a while (BTW, not complaining at all). The Tonis are longer, and the Noelia and Gianni WUs are a lot longer.

Just a heads up - Some NOELIA tasks are currently crashing when suspended or closed, even when running just 1. Those crashes are not the fault of running 2-at-once, so far as I know.

First, I believe the correct time numbers to use, when comparing, is Run Time (which determines overall throughput of how quickly you can complete tasks, and can be considered overall performance I believe), not CPU Time (which may vary, depending on several factors).

Second, for whatever reason, I thought that the "Swan" setting wasn't used anymore. Do you believe it's still used? If so, is it still used via a System Variable called "Swan_Sync" with a value of 0 or 1? Is it maybe only used on certain types of GPUS (like non-Keplar)? Is there a task type I could easily test turning Swan on and off, to immediately and easily see the resulting CPU Usage variation in Task Manager?

Regards,
Jacob

Yep, I'm seeing similar times when comparing Run Time to CPU Time.

And Task manager agrees with those times...
For me, for the Nathans tasks, Task Manager shows:
- the long-run tasks use a full CPU core all the time while processing
- the short-run tasks only use a portion of the CPU

So, because the CPU usage can very task to task (think different task types within the same app, even), that is why I have set <cpu_usage> to 0.001 for each app, allowing my CPU tasks to pick up any remaining actual CPU slack.

FYI:

Per Toni (a project administrator), because of application and validator problems, I was told to disable running 2-at-a-time. So I have suspended my testing/research, and have put <gpu_usage> to 1, in the app_config.xml file.

http://www.gpugrid.net/forum_thread.php?id=3332#29425

Regards,
Jacob Klein

For the past two years I've run an OCed ASUS GTX 460 1GB; core 850, shader 1700, memory 2000. It's stable (perhaps I should try pushing it further...?). The fan is silent, and temp is around 66C. Current long Nathans take about nine hours.

If I run two WUs concurrently, what % increase in thruput might I see?

Thanks, Tom
____________

tomba, running two tasks on a GTX 460 1GB is a bad idea

running two tasks on a GTX 460 1GB is a bad idea;

FYI:

Per Toni (a project administrator), because of application and validator problems, I was told to disable running 2-at-a-time. So I have suspended my testing/research, and have put <gpu_usage> to 1, in the app_config.xml file.

http://www.gpugrid.net/forum_thread.php?id=3332#29425

Regards,
Jacob Klein

Yesterday I retired my trusty ASUS GTX 460, replacing it with an ASUS GTX 660.

I activated app_config.xml, restarted BOINC, then the 25% in-process NOELIA. Immediately another NOELIA appeared and both crunched together.

The first WU finished this morning (18 hours run time, full bonus), but it was not until it had uploaded that another WU came down to replace it. So I lost more than an hour of crunching.

Previously I had run with minimum work buffer set to 0.01, which gave me a new WU download about 10 minutes before the active WU finished. I just set the buffer to 2.00 days. No reaction, even after a few clicks on BOINC Update.

Any thoughts? Thanks.

It's early days but what I reckon is that the doom & gloom about lost bonuses when doing 2X is unfounded, at least for me. Both my current WUs will complete well within 24 hours.
____________

Hmm...
Do you have any GPU exclusions in your cc_config.xml file?
Also, what version of BOINC are you using? Hopefully v7.0.64.

Behavior I've noticed previously is that... if BOINC has a task that's running, it doesn't consider the GPU "idle", and so it only fetches work when the work left on *that task* is below the "Minimum work buffer". Assuming you want to keep the GPU busy with 2-tasks-at-1-time, I think you'd have to increase the "Minimum work buffer" to be larger than the time necessary to complete a task running 2-at-1-time (note: This is larger than the time necessary to complete a task running 1-at-a-time). But I noticed you said you increased it already, to 2 days! I would have thought 2 days would have been enough, but maybe it needs to be even higher?

I kept my GTX 460 1GB in this machine, alongside my GTX 660 Ti, and so.... because the GTX 460 only has 1GB, I can't do 2-at-a-time unless I exclude that GPU. So I only do 1-at-a-time.

Hmm... Do you have any GPU exclusions in your cc_config.xml file?
Also, what version of BOINC are you using?

You should take that report_results_immediately flag out of there -- BOINC has a feature where the project can request that now, and GPUGrid does request it, so... by you using it, you're only increasing server load on other projects that may not need results immediately.

As far as your problem goes... Don't click update. BOINC evaluates whether it needs to fetch work every minute. So, I recommend increasing the Minimum buffer by a little (by 0.1 or 0.2), then waiting 2 minutes, to see if it got work. If it didn't, keep increasing it a little at a time, until it does what you want.

Actually, you should be able to get it to fetch work if you increase the Minimum buffer to be a bit larger than the current "Remaining (estimated)" time of the current task, but again, you really should set it to be a bit higher than the largest "Remaining (estimated)" time possible for a given task, to ensure you always have 2 tasks to run.

What other projects are you attached to? Are any of them GPU projects that get work? Are you sure you have network activity allowed? (BOINC -> Activity -> Network)? Are you sure you're setting the "Minimum work buffer"?

<cc_config>
<options>
<report_results_immediately>1</report_results_immediately>
</options>
</cc_config>

Personally I'd leave the report line in and try adding this one:
<fetch_on_update>1</fetch_on_update>
See if it makes a difference.

What does it say in the BOINC log when you manually update GPUGrid?

What does it say in the BOINC log when you manually update GPUGrid?

Alright, let's try this:

Set your cc_config to show work_fetch_debug, using something like this:
---------------------------------------------------------------

<cc_config>
<log_flags>
<work_fetch_debug>1</work_fetch_debug>
</log_flags>
<options>
</options>
</cc_config>

---------------------------------------------------------------

Then restart BOINC, and watch Event Viewer to see work fetch run. You'll see it say "[work_fetch] work fetch start" at the beginning of a run.

In the Event Viewer, select all the lines from "Starting BOINC" all the way until the 2nd instance of "[work_fetch] work fetch start", so that we include all the messages from the entire first run.

Then hit "Copy selected", and then paste here in a reply. I'm generally very good at reading these work fetch logs.

Oh, well there's your answer. If it says you've reached a limit on tasks in progress... BOINC did ask for work, but GPUGrid only lets you have so many tasks. I see (by clicking your name and looking at your computers) that you currently have 2 in progress; perhaps they only allow 2. Do you need more than 2?

Oh, well there's your answer. If it says you've reached a limit on tasks in progress... BOINC did ask for work, but GPUGrid only lets you have so many tasks. I see (by clicking your name and looking at your computers) that you currently have 2 in progress; perhaps they only allow 2. Do you need more than 2?

So, you DO need more than 2. You need a 3rd one to be downloaded, a couple minutes before either of the 2 running ones are done... right? :)

I wonder if GPUGrid has things setup to only allow 2-tasks-per-GPU. (For reference, I've had 5 or 6 tasks in progress before, but my system has 3 GPUs).

I'm not sure if I'll be able to help you test/confirm. To do that, I think I'd have to pull some GPUs out. But it sounds like your setup will probably be asking GPUGrid for work a lot, and not getting any, which is not ideal.

Alright, let's try this:

Set your cc_config to show work_fetch_debug, using something like this: etc...

If you read that work fetch log, you'll see that the reason it didn't fetch work there, was because "can't req work: suspended via Manager"

Anyway, for your scenario, you do want a 3rd task. However, I have just confirmed (by pulling 2 of my 3 GPUs out of the system) that GPUGrid does only allow 2 in-progress tasks on a 1-GPU-system.

That being said... when the "completed" task gets reported, if you have a sufficiently high minimum buffer at that point, then it might get a new task.

So... try keeping your "Minimum work buffer" high enough (maybe set to 2 days), but then let it run long enough to complete a task. Don't do anything manual (don't click Update, don't suspend it, just let it go), and watch what it does. With a 2-day-buffer, does it automatically request and get a new task at the same time as reporting the completed one?

Please report your findings. I also am doing this test, since I'm curious.

If you read that work fetch log, you'll see that the reason it didn't fetch work there, was because "can't req work: suspended via Manager"

Anyway, for your scenario, you do want a 3rd task. However, I have just confirmed (by pulling 2 of my 3 GPUs out of the system) that GPUGrid does only allow 2 in-progress tasks on a 1-GPU-system.

That being said... when the "completed" task gets reported, if you have a sufficiently high minimum buffer at that point, then it might get a new task.

If you read that work fetch log, you'll see that the reason it didn't fetch work there, was because "can't req work: suspended via Manager"

Anyway, for your scenario, you do want a 3rd task. However, I have just confirmed (by pulling 2 of my 3 GPUs out of the system) that GPUGrid does only allow 2 in-progress tasks on a 1-GPU-system.

Surely it's not just my scenario? Anyone running 2X wants a third WU before the WU that's about to complete actually completes. Yes?

That being said... when the "completed" task gets reported, if you have a sufficiently high minimum buffer at that point, then it might get a new task.

As I said this morning, I got a new WU when the completed task had been uploaded - reported - missing an hour+ of crunching. That was with a buffer of 0.01. My buffer is now at two days. If BOINC reports at WU-finished time, not just WU uploaded time, there's a chance I'll now see a new WU downloaded as soon as one finishes. The proof of the pudding comes in about 100 minutes, when my most-advanced WU completes.

If, with a two day buffer, I still have to wait for a new WU until the upload is done, then I have to consider going back to 1X.

I hope it works out for you (and for me too!). But, even if it doesn't work, why would you have to go back to 1-at-a-time? Wouldn't you prefer "usually-2-at-a-time" instead of "always-1-at-a-time"? Just trying to understand your logic.

Regards,
Jacob

If it is even only marginally better...
I'd still think that "usually 2-at-a-time but always at least 1-at-a-time" would be always better than "always just 1-at-a-time".

ie: Not having that 3rd task "ready" shouldn't make it any worse than running 1-at-a-time.

:)

Not having that 3rd task "ready" shouldn't make it any worse than running 1-at-a-time.:)

Not having that 3rd task "ready" shouldn't make it any worse than running 1-at-a-time.:)

I guess even you have yet to come to grips with measurable performance improvement..

I guess even you have yet to come to grips with measurable performance improvement..

I have no idea what you meant by that.

I'm hoping your results are positive.

Right. So, when you say <gpu_usage>0.5</gpu_usage> ... all that is doing is telling BOINC how much to "consider allocated" for each task, for purposes of deciding how many tasks to start, therefor allowing you to do 2-at-a-time.

That app_config.xml setting does NOT LIMIT the GPU Usage or Load, in any way.

In fact, what you say is true; when one of the 2-at-a-time tasks gets done, the remaining task now utilizes the GPU Usage as if it was the only one ever running on it (to an extent; I believe I have proof that the task gets started in a certain "mode" based on GPU RAM available, which may possibly have an affect on GPU Load, but hopefully doesn't.)

You can test this by suspending certain tasks while watching GPU Load (though, I caution you, suspending a NOELIA task can crash the drivers and make GPU tasks error out, even tasks on other GPUs or tasks doing work for other projects.)

So... as an example:
If you had 3 GPUGrid tasks:
- Task A: normally gets 63% GPU Load
- Task B: normally gets 84% GPU Load
- Task C: normally gets 79% GPU Load; not downloaded yet
then...
The walkthrough of the scenario is:
You are running Tasks A and B on the same GPU, getting 93% GPU Load... Task A gets done, so uploading Task A. During that upload, only Task B is running, at 84% GPU Load. Then, once you get the Task C downloaded, you can run both B and C together, at 98% GPU Load.

Hope that makes sense. That's the behavior I'm used to seeing on other projects, and would expect to see here too.

Also, I see you consider 91% GPU Load "very busy". I would consider 98% to be "very busy" :) You might be interested in running eVGA Precision X; although it was designed for overclocking, I use it as a very handy tool for showing monitoring history over time, and putting GPU Temps as icons in my system tray.

At 3:05am BOINC reported a failed NOELIA. No WU downloaded to replace it.

It was not until 5:07am, when BOINC reported the completion of the other WU, that two NOELIAs were downloaded.

What's your cache at?

I have concluded my test (where I had only 1 active GPU, which was processing 2-tasks-at-once, with a 1.5 day min buffer, and wanted to see when the 3rd new task gets started). Note, I'm using BOINC v7.1.1 alpha, which includes a major work fetch tweaking as compared to the v7.0.64 public release.

Assuming I'm reading the logs below correctly, here is what I see:
26-May-2013 03:20:29: Computation for Task 2 finished
26-May-2013 03:20:29: At this time, GPUGrid was in "resource backoff", meaning we couldn't ask it for NVIDIA work, because the last time we asked it for NVIDIA work we didn't get any (because of the maximum-2-in-progress-per-GPU server-side rule), and BOINC automatically creates an incrementing backoff timer when that happens. The resource backoff was still effective for 3314.63 secs (~55 minutes)
26-May-2013 03:20:43: Upload of Task 2 results started
26-May-2013 03:32:59: GPUGrid was still in "resource backoff" for 2564.79 secs (~43 minutes)
26-May-2013 03:33:42: Upload of Task 2 results finished
26-May-2013 03:33:44: BOINC reported the GPUGrid task, and piggybacked a request for NVIDIA work (since there were no other contactable projects that supported NVIDIA work, and we still needed some). Note: This piggyback request should also happen on v7.0.64 also, but I cannot guarantee that.
26-May-2013 03:33:46: RPC completed; BOINC did get 1 new task from GPUGrid
26-May-2013 03:33:48: Download of Task 3 started
26-May-2013 03:34:09: Download of Task 3 finished; Task 3 started processing

So, according to this...
There was a ~14 minute "layover" where BOINC was only allowed to run 1 task on the GPU, due to GPUGrid's server-side limitation. But it did gracefully handle the scenario, did eventually get the 3rd task, and started it promptly. It worked as I expected it to work, given the server-side limitation, but it's not optimal, because we should be allowed to keep the GPU continuously fully loaded with 2 tasks. :(

I wonder if we can convince GPUGrid to relax the limit, to max-3-in-progress-per-GPU, instead of max-2-in-progress-per-GPU? In theory, that should close this gap, as it would allow the 3rd task to be downloaded/ready whenever the min-buffer says the client needed it (earlier than Task 2 completion).

Full log snippet:
26-May-2013 03:20:29 [---] [work_fetch] Request work fetch: application exited
26-May-2013 03:20:29 [GPUGRID] Computation for task I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1 finished
26-May-2013 03:20:29 [---] [work_fetch] work fetch start
26-May-2013 03:20:29 [---] [work_fetch] ------- start work fetch state -------
26-May-2013 03:20:29 [---] [work_fetch] target work buffer: 129600.00 + 8640.00 sec
26-May-2013 03:20:29 [---] [work_fetch] --- project states ---
26-May-2013 03:20:29 [GPUGRID] [work_fetch] REC 261498.750 prio -13.919910 can req work
26-May-2013 03:20:29 [---] [work_fetch] --- state for CPU ---
26-May-2013 03:20:29 [---] [work_fetch] shortfall 0.00 nidle 0.00 saturated 149576.75 busy 0.00
26-May-2013 03:20:29 [GPUGRID] [work_fetch] fetch share 0.000 (no apps)
26-May-2013 03:20:29 [---] [work_fetch] --- state for NVIDIA ---
26-May-2013 03:20:29 [---] [work_fetch] shortfall 122118.58 nidle 0.50 saturated 0.00 busy 0.00
26-May-2013 03:20:29 [GPUGRID] [work_fetch] fetch share 0.000 (resource backoff: 3314.63, inc 19200.00)
26-May-2013 03:20:29 [---] [work_fetch] ------- end work fetch state -------
26-May-2013 03:20:29 [---] [work_fetch] No project chosen for work fetch
26-May-2013 03:20:43 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_0
26-May-2013 03:20:43 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_1
26-May-2013 03:20:43 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_2
26-May-2013 03:20:43 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_3
26-May-2013 03:21:06 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_0
26-May-2013 03:21:06 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_7
26-May-2013 03:21:07 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_7
26-May-2013 03:21:07 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_9
26-May-2013 03:21:12 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_3
26-May-2013 03:21:12 [GPUGRID] Started upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_10
26-May-2013 03:21:13 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_10
26-May-2013 03:21:38 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_1
26-May-2013 03:21:38 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_2
26-May-2013 03:32:59 [---] [work_fetch] work fetch start
26-May-2013 03:32:59 [---] [work_fetch] ------- start work fetch state -------
26-May-2013 03:32:59 [---] [work_fetch] target work buffer: 129600.00 + 8640.00 sec
26-May-2013 03:32:59 [---] [work_fetch] --- project states ---
26-May-2013 03:32:59 [GPUGRID] [work_fetch] REC 261379.396 prio -13.917504 can req work
26-May-2013 03:32:59 [---] [work_fetch] --- state for CPU ---
26-May-2013 03:32:59 [---] [work_fetch] shortfall 0.00 nidle 0.00 saturated 148218.82 busy 0.00
26-May-2013 03:32:59 [GPUGRID] [work_fetch] fetch share 0.000 (no apps)
26-May-2013 03:32:59 [---] [work_fetch] --- state for NVIDIA ---
26-May-2013 03:32:59 [---] [work_fetch] shortfall 123046.19 nidle 0.50 saturated 0.00 busy 0.00
26-May-2013 03:32:59 [GPUGRID] [work_fetch] fetch share 0.000 (resource backoff: 2564.79, inc 19200.00)
26-May-2013 03:32:59 [---] [work_fetch] ------- end work fetch state -------
26-May-2013 03:32:59 [---] [work_fetch] No project chosen for work fetch
26-May-2013 03:33:42 [GPUGRID] Finished upload of I2HDQ_5R9-SDOERR_2HDQd-3-4-RND0408_1_9
26-May-2013 03:33:42 [---] [work_fetch] Request work fetch: project finished uploading
26-May-2013 03:33:44 [---] [work_fetch] ------- start work fetch state -------
26-May-2013 03:33:44 [---] [work_fetch] target work buffer: 129600.00 + 8640.00 sec
26-May-2013 03:33:44 [---] [work_fetch] --- project states ---
26-May-2013 03:33:44 [GPUGRID] [work_fetch] REC 261379.396 prio -13.917369 can req work
26-May-2013 03:33:44 [---] [work_fetch] --- state for CPU ---
26-May-2013 03:33:44 [---] [work_fetch] shortfall 0.00 nidle 0.00 saturated 148162.47 busy 0.00
26-May-2013 03:33:44 [GPUGRID] [work_fetch] fetch share 0.000 (no apps)
26-May-2013 03:33:44 [---] [work_fetch] --- state for NVIDIA ---
26-May-2013 03:33:44 [---] [work_fetch] shortfall 123101.89 nidle 0.50 saturated 0.00 busy 0.00
26-May-2013 03:33:44 [GPUGRID] [work_fetch] fetch share 1.000
26-May-2013 03:33:44 [---] [work_fetch] ------- end work fetch state -------
26-May-2013 03:33:44 [GPUGRID] [work_fetch] set_request() for NVIDIA: ninst 1 nused_total 0.500000 nidle_now 0.500000 fetch share 1.000000 req_inst 0.500000 req_secs 123101.894609
26-May-2013 03:33:44 [GPUGRID] [work_fetch] request: CPU (0.00 sec, 0.00 inst) NVIDIA (123101.89 sec, 0.50 inst)
26-May-2013 03:33:44 [GPUGRID] Sending scheduler request: To report completed tasks.
26-May-2013 03:33:44 [GPUGRID] Reporting 1 completed tasks
26-May-2013 03:33:44 [GPUGRID] Requesting new tasks for NVIDIA
26-May-2013 03:33:46 [GPUGRID] Scheduler request completed: got 1 new tasks
26-May-2013 03:33:46 [---] [work_fetch] Request work fetch: RPC complete
26-May-2013 03:33:48 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-LICENSE
26-May-2013 03:33:48 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-COPYRIGHT
26-May-2013 03:33:48 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_1
26-May-2013 03:33:48 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_2
26-May-2013 03:33:49 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-LICENSE
26-May-2013 03:33:49 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-COPYRIGHT
26-May-2013 03:33:49 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_3
26-May-2013 03:33:49 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-pdb_file
26-May-2013 03:33:52 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_1
26-May-2013 03:33:52 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_3
26-May-2013 03:33:52 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-psf_file
26-May-2013 03:33:52 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-par_file
26-May-2013 03:33:55 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_2
26-May-2013 03:33:55 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-conf_file_enc
26-May-2013 03:33:56 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-par_file
26-May-2013 03:33:56 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-conf_file_enc
26-May-2013 03:33:56 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-metainp_file
26-May-2013 03:33:56 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_7
26-May-2013 03:33:57 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-metainp_file
26-May-2013 03:33:57 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_7
26-May-2013 03:33:57 [GPUGRID] Started download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_10
26-May-2013 03:33:58 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-I61R18-NATHAN_dhfr36_5-18-32-RND7448_10
26-May-2013 03:34:06 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-pdb_file
26-May-2013 03:34:09 [GPUGRID] Finished download of I61R18-NATHAN_dhfr36_5-19-psf_file
26-May-2013 03:34:09 [GPUGRID] Starting task I61R18-NATHAN_dhfr36_5-19-32-RND7448_0 using acemdlong version 618 (cuda42) in slot 11

At 3:05am BOINC reported a failed NOELIA. No WU downloaded to replace it.

It was not until 5:07am, when BOINC reported the completion of the other WU, that two NOELIAs were downloaded.

Boinc Manager (Advanced View), Tools, Computing Preferences, network usage tab, minimum work buffer + maximum additional work buffer.

Did you have the work_fetch_debug option on at that time? If so, can you provide the log for the work-fetch sequence where the failed task was reported? It will hopefully be able to tell us why a request for work was not also piggybacked.

Did you have the work_fetch_debug option on at that time? If so, can you provide the log for the work-fetch sequence where the failed task was reported? It will hopefully be able to tell us why a request for work was not also piggybacked.

I have the debug log. The failed task was reported at 03:05:40. I've endlessly scrolled the log around that time, looking for the failure. It doesn't help that I don't know what I'm looking for!!

The lazy way around this is to have another GPU in the system, an ATI, or maybe use cc_config to exclude a second NVidia for this project. While that should keep the work flowing, it's not a proper fix.

The project setting of no more than 2 WU's per GPU won't be changed.

It's not the problem anyway. The problem is that the "resource backoff" remains after a resource becomes free. It needs to be reset/zeroed.