Author |
Message |
|
NATHAN_FA4........... 1.29 credits/sec (99% GPU usage)
TONI_SH2MS3_....... 2.13 credits/sec
GIANNI_KKFREE5... 2.37 credits/sec
IBUCH_2_nmEGFR.. 2.43 credits/sec
IBUCH_PYRT........... 1.35 credits/sec (this is also very low, but it has a very low GPU usage, and high CPU usage by an iteration algorithm)
IBUCH_freePYRT..... 1.73 credits/sec
I think this situation is similar to what we had with the crediting of previous GIANNI_KKFREE workuntis. |
|
|
skgivenVolunteer moderator Volunteer tester
Send message
Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level
Scientific publications
|
Running a long IBUCH_PYRT task now on a GTX470.
Without SWAN_SYNC it was only utilizing about 59% of the GPU.
With SWAN_SYNC in place (and 2 threads freed up) the task used around 63 or 64% (about 4 or 5%) more.
This is quite low compared to the average 85% on this system and the 99% for Nate's tasks, but the tasks are very different.
____________
FAQ's
HOW TO:
- Opt out of Beta Tests
- Ask for Help |
|
|
|
Based on the very high GPU usage of this NATHAN_FA4 workunits, I expected the credits/sec ratio will exceed the GIANNI_KKFREE5's, and the IBUCH_2_nmEGFR's. But - to my disappointment - it is the half of what I would think proportional.
I'm actually running without SWAN_SYNC, because I'm crunching for SIMAP with all of my CPU cores. |
|
|
skgivenVolunteer moderator Volunteer tester
Send message
Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level
Scientific publications
|
The present mix of tasks makes it difficult to decide what to do re SWAN_SYNC on Windows (it's still worth using on Linux with a Fermi). I think the tasks that use more CPU are awarded more credit because they use more CPU (and you can't use that CPU for crunching on other CPU projects). That makes sense, but the delta between the credit awarded it too large in my opinion, and ignores the fact that a GPU running at 99% is doing a lot more than a GPU at 59% or even 85%. I'm happy with the present diversity of task types (the more the merrier) but it's a challenge when some tasks utilize 40% more of the GPU.
____________
FAQ's
HOW TO:
- Opt out of Beta Tests
- Ask for Help |
|
|
|
... I think the tasks that use more CPU are awarded more credit because they use more CPU (and you can't use that CPU for crunching on other CPU projects).
According to this post IBUCH_PYRT tasks do not have the credit compensation you've mentioned. Even so the NATHAN_FA4 tasks have lower credit/time ratio than IBUCH_PYRT tasks, while they have 40% more GPU usage.
Back in the times of CPU only crunching I was know when my tasks were failing during processing by monitoring my hosts' RAC, but nowadays this much diversity in the credit/time ratio between different GPU tasks makes it difficult and time consuming. |
|
|
skgivenVolunteer moderator Volunteer tester
Send message
Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level
Scientific publications
|
Perhaps credit compensation is not being applied to some of the tasks in the way we are used to?
____________
FAQ's
HOW TO:
- Opt out of Beta Tests
- Ask for Help |
|
|
nateSend message
Joined: 6 Jun 11 Posts: 124 Credit: 2,928,865 RAC: 0 Level
Scientific publications
|
My apologies, everyone. The lower credit on my tasks is in part due to a short command I was not aware I needed to include for long tasks. New guy mistake. I'll try to get it fixed as soon as I can. Thanks for pointing this out. |
|
|
|
Sorry Nate,
I'm aborting your tasks because they use 98% GPU and make it difficult to do anything else when they are running.
____________
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline |
|
|
SMTB1963 Send message
Joined: 27 Jun 10 Posts: 38 Credit: 524,420,921 RAC: 0 Level
Scientific publications
|
I was wondering about the points awarded for these new tasks as well. I wouldn't normally worry about it too much, except that our team (XtremeSystems) is having a GPUGRID event this week.
Nate, would you mind posting here when you get it fixed? Thanks!
Best Regards |
|
|
|
Hi Betting Slip,
I see that you cancelled a task. (I1R64-NATHAN_FA4-4-100-RND6005_0)
And I was surprised about the use of the NATHAN work units. My GTX 580 is used at 92 %
but, I also see a high use of memory.
So I think this is what is cousing you problems.
This is a log (from GPUZ) for a NATHAN WU
Date = GMT+2
Date , GPU Core Clock [MHz] , GPU Memory Clock [MHz] , GPU Shader Clock [MHz] , GPU Temperature [°C] , Fan Speed (%) [%] , Fan Speed (RPM) [RPM] , Memory Used [MB] , GPU Load [%] , Memory Controller Load [%] , Video Engine Load [%] , VDDC [V] ,
2011-11-06 02:29:01 , 781.1 , 1002.0 , 1564.0 , 75.0 , 30 , 1650 , 1310 , 91 , 23 , 0 , 1.0630 ,
It is using 1310 MB of memory, so I have 226 MB (1536 - 1310) to show some pixels on the screen :)
comparing with GIANNI_KKFREE5 WU it only uses 532 MB.
Your GeForce GTX 460 has only 1024 MB so I expect that you will have some problems with this WU.
Nate:
Is it correct what am I thinking?
Best regards,
Alejandro
|
|
|
skgivenVolunteer moderator Volunteer tester
Send message
Joined: 23 Apr 09 Posts: 3968 Credit: 1,995,359,260 RAC: 0 Level
Scientific publications
|
Alejandro, 1310MB sounds a bit excessive, but GPUZ will not show how much the task is using, it shows how much everything is using. If you close Boinc then you can work out what the system and everything else is using. Are you using Aero? How much do your other tasks use? I typically see about 492MB DDR5 when running long tasks on a GTX470 on XP/2003. As the amount of GDDR5 used depends on the number of active shaders, less is required for lesser cards.
PS. Your GPU is at 75degC but your fan is only at 30%. I suggest you use one of the recommended programs to increase your fan speed and reduce your GPU temperature. You should be able to run a GTX580 at 55 to 60 degC and run the fan at <60%
____________
FAQ's
HOW TO:
- Opt out of Beta Tests
- Ask for Help |
|
|
nateSend message
Joined: 6 Jun 11 Posts: 124 Credit: 2,928,865 RAC: 0 Level
Scientific publications
|
Hey all,
I decided to just kill that entire set of jobs last friday since I started it only a few days before. Though, any that were in progress or sent out before I cancelled the series would still be completed (the data is still useful). I re-submit the continued jobs today under NATHAN_FA5. I believe I have solved the problems, but let me know ASAP if any problems arise.
As to your specific questions/comments:
BettingSlip: No worries. You guys should never feel compelled to complete a task that is causing you problems. Do what you must and let us know about it so we can try to fix it.
SMTB1963: The points were unnecessarily low and the usage too high on these tasks due to a command that is not automatically active in the long tasks. I was unaware of this, and sent them without this command. :-/
Alejandro, BettingSlip, et al: It is possible you are correct, Alejandro, and I am looking into this now. The system I am running is quite large, in fact the largest we have ever sent to GPUGRID, and the memory usage is substantial. I had no issues running locally with our various 400 and 500 series cards when I set up the system, but obviously there are a large variety of cards I can't test. It is possible that cards with only 1024MB will have little or no free memory with my tasks. I will look at that tomorrow when I get in and try to post guidance as soon as possible.
Once again, I'd like to apologize for these issues. We strive to make crunching for GPUGRID as seamless and painless as possible (we all benefit). I will update our SOP here so we can avoid the low credit/high GPU usage issue in the future, and I will also include a section about memory considerations for large systems.
Nate |
|
|
|
Appreciate that Nate. I don't think it was a memory issue as when I checked your running units where using about 500 MB. more a GPU usage or priority issue. Have seen this before when someone here altered priority to get more efficient use of badly configured systems and ended up with a similar problem or it could just be with using 98% there is nothing left for the system to do other work.
Anyway, whatever it is I'm sure you'll sort it out.
____________
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline |
|
|
nateSend message
Joined: 6 Jun 11 Posts: 124 Credit: 2,928,865 RAC: 0 Level
Scientific publications
|
Hi all,
I wanted to update on the possible memory issues that were being discussed. I have run some tests here and discussed it with the more hardware/software focused members of our team, and memory should not be a problem. In short, the software is capable of adjusting its function to accommodate a wide range of memory capacities. For example, while running my large system on a GTX 580 here, the memory usage is 1138MB out of 1.5GB. But the exact same system run on a GTX 275 uses only 580MB of 1024GB. The code adjusts to the memory available (with a small performance sacrifice for lower memory capacities). Perhaps we'll post more on this later, but based on current empirical results and the (expected) function of the code, it should not cause any unexpected or uncontrolled performance issues on your cards. But as with everything, bugs, errors, and mistakes are possible so keep posting here when you have problems.
Happy crunching
Nate |
|
|
ignasiSend message
Joined: 10 Apr 08 Posts: 254 Credit: 16,836,000 RAC: 0 Level
Scientific publications
|
Unlike then, more recent *PYRT_111019* tasks DO contain credit compensation. However, compared to *nmEGFR* that Zoltan was pointing out, they don't contain a simulation-related perfomance factor that gives ~30% increase. It is such way as PYRT WUs are reproduction jobs of old TRYP experiments. And those, didn't use the perfomance factor... These are the drawbacks of trying to do rigorous science. :)
cheers,
ignasi |
|
|
|
Hi Skgiven, Nate,
I disabled aero, and incremented the fan speed to 40%
NATHAN_FA2 with aero, fan 40%
2011-11-11 19:28:48 , 781.1 , 1002.0 , 1564.0 , 68.0 , 40 , 2040 , 1322 , 92 , 22 , 0 , 1.0630 ,
NATHAN without aero , fan 40%
2011-11-11 19:28:58 , 781.1 , 1002.0 , 1564.0 , 68.0 , 40 , 2040 , 1275 , 91 , 23 , 0 , 1.0630 ,
fan 50%
2011-11-11 20:03:50 , 781.1 , 1002.0 , 1564.0 , 65.0 , 50 , 2430 , 1313 , 93 , 23 , 0 , 1.0630 ,
fan 61%
2011-11-11 20:08:25 , 781.1 , 1002.0 , 1564.0 , 62.0 , 61 , 2790 , 1272 , 93 , 23 , 0 , 1.0630 ,
The temperature goes from 75C (fan on auto mode) to 68C (fan set to 40%). I tried to put the fan at 100% , but it sounds like a vacuum cleaner :)
GIANNI_KKFREE5 with aero and fan on auto mode.
2011-11-11 13:19:30 , 781.1 , 1002.0 , 1564.0 , 65.0 , 26 , 1500 , 746 , 64 , 16 , 0 , 1.0630 ,
Alejandro
|
|
|
SMTB1963 Send message
Joined: 27 Jun 10 Posts: 38 Credit: 524,420,921 RAC: 0 Level
Scientific publications
|
...
SMTB1963: The points were unnecessarily low and the usage too high on these tasks due to a command that is not automatically active in the long tasks. I was unaware of this, and sent them without this command. :-/
...
Well Nate, you definitely fixed the issue! My 570s are cranking the points on your tasks now.
Best |
|
|