Message boards : Graphics cards (GPUs) : Run times
Author | Message |
---|---|
Has the run time been creeping up while I was not looking? I recall run times of about 6:30 on most of my cards (with some plus minus slop) but now It seems I am seeing 7:30 to 8:00 run times. | |
ID: 12554 | Rating: 0 | rate: / Reply Quote | |
There is a new bug in BOINC (6.6.36) which assigns all WU to the same gpu if you have multiple gpus. As they time share it, they take much longer. | |
ID: 12562 | Rating: 0 | rate: / Reply Quote | |
There is a new bug in BOINC (6.6.36) which assigns all WU to the same gpu if you have multiple gpus. As they time share it, they take much longer. GDF, As far as I know that assign all tasks to GPU 0 is a Linux only bug ... of course that is what you run ... :) At the moment I am running mostly 6.6.3x versions but have been pretty happy with 6.10.3 which does not have the two major issues of 6.10.4 and .5 ... 6.10.6 fixed a couple issues but has still left uncorrected some problems with the order in which it processes GPU tasks for some people (introduced in 6.10.4). Also note that I think that I just uncovered a new bug / situation with task ordering on the GPU with multiple projects that, in essence, will cause Resource Share to be ignored. I do not know how far back in versions that this bug extends. For me it is new in that to this point there was no pressure to run multiple projects for the simple reason that effectively there were no projects to run ... Now that we are ramping up more and more projects with GPU capabilities ... well ... Anyway, my suggestions are still 6.5.0, 6.6.36, or 6.10.3; and as I said these are versions I had run extensively or am running now ... | |
ID: 12563 | Rating: 0 | rate: / Reply Quote | |
I just realized that you neatly sidestepped my original question... :) | |
ID: 12565 | Rating: 0 | rate: / Reply Quote | |
There is a new bug in BOINC (6.6.36) which assigns all WU to the same gpu if you have multiple gpus. As they time share it, they take much longer. Aha, that'll be why my GTX295 has started working, albeit slowly | |
ID: 12566 | Rating: 0 | rate: / Reply Quote | |
There is a new bug in BOINC (6.6.36) which assigns all WU to the same gpu if you have multiple gpus. As they time share it, they take much longer. GPU scheduling seems to be fubar'd for Linux in one way or another with pretty much all releases. There is the everything gets assigned to '--device 0' bug in the 6.6.3x series (cause coproc_cmdline() is called post fork()) and the preempt problems with 6.10.x. I'm running 6_6a branch (which is equiv to an unreleased 6.6.39) plus the following patch (r18836 from trunk) which will resolve the '--device 0' issue. It seems pretty solid. --- boinc_core_release_6_6_39/client/app_start.cpp.orig 2009-09-15 11:18:45.000000000 +0100 +++ boinc_core_release_6_6_39/client/app_start.cpp 2009-09-15 11:52:34.000000000 +0100 @@ -104,8 +104,10 @@ } #endif -// for apps that use coprocessors, reserve the instances, -// and append "--device x" to the command line +// For apps that use coprocessors, reserve the instances, +// and append "--device x" to the command line. +// NOTE: on Linux, you must call this before the fork(), not after. +// Otherwise the reservation is a no-op. // static void coproc_cmdline( COPROC* coproc, ACTIVE_TASK* atp, int ninstances, char* cmdline @@ -793,6 +795,13 @@ getcwd(current_dir, sizeof(current_dir)); + sprintf(cmdline, "%s %s", + wup->command_line.c_str(), app_version->cmdline + ); + if (coproc_cuda && app_version->ncudas) { + coproc_cmdline(coproc_cuda, this, app_version->ncudas, cmdline); + } + // Set up core/app shared memory seg if needed // if (!app_client_shm.shm) { @@ -924,10 +933,6 @@ } } #endif - sprintf(cmdline, "%s %s", wup->command_line.c_str(), app_version->cmdline); - if (coproc_cuda && app_version->ncudas) { - coproc_cmdline(coproc_cuda, this, app_version->ncudas, cmdline); - } sprintf(buf, "../../%s", exec_path ); if (g_use_sandbox) { char switcher_path[100]; Send me a PM if you want a link to the RPM's and SRPM for a Fedora 11 build. | |
ID: 12569 | Rating: 0 | rate: / Reply Quote | |
Should agree with you, Paul. | |
ID: 12570 | Rating: 0 | rate: / Reply Quote | |
No, WUs are not longer. They are designed to last 1/4 of a day on a fast card. | |
ID: 12571 | Rating: 0 | rate: / Reply Quote | |
I have updated the recommended client to 6.10.3. | |
ID: 12572 | Rating: 0 | rate: / Reply Quote | |
No, WUs are not longer. They are designed to last 1/4 of a day on a fast card. Looking through my last lot of results the shortest seem to be 7 hours 30 mins and the majority seem to be around 8 hours 30 mins. That was taken using the "approx elapsed time" shown in the wu results. These were run on GTX295 and GTX275 so by no means a slow card, although they do run at standard speeds. One machine (the GTX275) has BOINC 6.6.37 and the other is currently running 6.10.3 under windows. ____________ BOINC blog | |
ID: 12576 | Rating: 0 | rate: / Reply Quote | |
No, WUs are not longer. They are designed to last 1/4 of a day on a fast card. Well, your design is bent ... I used to get timings that were in the range of 6 hours and change on my GTX295 cards... sadly I cannot prove this as the task list is truncated at about the first of September and I am thinking back to much earlier. If your intent is to run for about 1/4 of a day, or 6 hours well, you are over-shooting that on GTX260s, GTX285, and GTX295 cards ... the more common time seems to be up in the 28,000 seconds than down at 21K seconds. This does seem task dependent. I am only pointing this out because it seems strange that before most tasks did come in under 7 hours and now more and more are running up to 9 hours. And you don't seem to be aware of the increase in run times ... A minor point then becomes you are shading the credit grant ... :) But most importantly to me is that you are not aware that you are overrunning your execution time targets ... Off to see the football ... | |
ID: 12577 | Rating: 0 | rate: / Reply Quote | |
Ok, let's say that it is between 1/4 and 1/3 of a day. | |
ID: 12578 | Rating: 0 | rate: / Reply Quote | |
Ok, let's say that it is between 1/4 and 1/3 of a day. Ok ... But I am not sure that you are seeing to point of my question... Are you aware that the time is growing... The only reason I really noticed it was because for a couple months here lately I was not able to pay attention to GPU Grid (notice the lack of posting) and it was a little bit of a shock to see that my run times are almost always over 7 hours now and running as high as 9 where before my run times were real consistenly clustered about 6.5 hours ... Not to put too fine a point on it, but, if this is the case the low end recommendation for hardware needs revision ... | |
ID: 12581 | Rating: 0 | rate: / Reply Quote | |
I had a similar increase in runtime just after I upgraded to the 190 drivers. Completely removing them and reinstalling them fixed it for me! | |
ID: 12583 | Rating: 0 | rate: / Reply Quote | |
The increased time may be due to a bug in the 190.xx drivers, which puts the GPU in 2D mode, more information in this post. | |
ID: 12606 | Rating: 0 | rate: / Reply Quote | |
The increased time may be due to a bug in the 190.xx drivers, which puts the GPU in 2D mode, more information in this post. I could have sworn we were told we needed to update to the 190 series drivers. Did I misunderstand? I mean I think I have all my systems running the 190.62 drivers now ... no 2 are on 190.62 and one is on 190.38 ... The thing is that in that I don't turn my systems off and they run 24/7 I don't see how they get back into 3D mode if the issue is down-shifting to 2D mode... I would think that once it was down it could not, or at least would not, re-adjust up on the next task. That is why I have trouble thinking that this is that kind of problem. I have not done a survey though my quick look seemed to hint that it is more likely task type dependent ... that is, some of the tasks (by task name class) are now running longer than the norms ... | |
ID: 12607 | Rating: 0 | rate: / Reply Quote | |
With the new Linux cuda 2.2 application, my work units are running faster and producing more credit per day. I am using the 190.32 Nvidia drivers. | |
ID: 12649 | Rating: 0 | rate: / Reply Quote | |
I don't think, that sudden switch to 2D mode could be the reason. I'm almost run all the time GPU-Z, to control the core \ mem frequencies, as well, as core temps. All three GTX 260 runs at full load. Additionally, in <stderr_txt> file main details is shown, as well, as core frequency. Here is sample: | |
ID: 12691 | Rating: 0 | rate: / Reply Quote | |
Another consideration is that the amount of CPU time has risen sharply which slows down other projects. | |
ID: 12714 | Rating: 0 | rate: / Reply Quote | |
Another consideration is that the amount of CPU time has risen sharply which slows down other projects. GPU Grid was my project of choice for the GPU however, it appears it consumes a hefty amount of CPU, more than I would expect and I am aware it needs to use some CPU time. I have noticed that v670 of the Linux app uses approx 10% more CPU than it used to with v666. I wonder whether that is by design or an unwelcome side effect? | |
ID: 12715 | Rating: 0 | rate: / Reply Quote | |
it might depend on the WU. The new application uses some more CPU for some runs. | |
ID: 12722 | Rating: 0 | rate: / Reply Quote | |
Should state again, that WU's becomes longer & longer. | |
ID: 12936 | Rating: 0 | rate: / Reply Quote | |
I've seen this as well -- it is quite noticeable with the slower cards I run (9600GT to GTS 250). In fact, with the 'run time creep' which seemed to start over the summer, I found 9400GT cards which could just squeak in under the 120 hour run limit had no chance to complete any longer. I'm running Collatz for that. Should agree with you, Paul. | |
ID: 12973 | Rating: 0 | rate: / Reply Quote | |
Though it is not a "cure" and I have not done it universally yet... I have started to run MW on one of my CUDA equipped machines. With the longer run times here, I hate to say it, but, based on my thumbnail calculations the pay is better with MW ... I have not yet tried Collatz on the CUDA cards yet to see what the time is ... but ... with the time creeping higher and higher ... | |
ID: 12977 | Rating: 0 | rate: / Reply Quote | |
I have updated the recommended client to 6.10.3. I found that, at least for the 64-bit Windows versions, simply upgrading from 6.6.36 to 6.10.3 makes a large increase in the initial expected time to completion for GPUGRID workunits. All I've had has since then ran in much less time than the initial expected value, though, so the initial expected value is gradually coming down. Now down from about 120 hours to about 31 hours for me, compared to the 20 hours they typically take. Also, I've found that at least this 6.10.3 version has a problem with ignoring the memory limit when any memory-demanding CPU project in the queue has enough workunits in the queue to be unable to run them on only the number of CPU cores it can use while obeying the memory limit and still get enough CPU time to finish them all by their deadlines without switching more than that number into high-priority mode immediately. Could you let me know if you've found a 64-bit Windows BOINC version that does not have similar problems and is still recent enough to use with GPUGRID? | |
ID: 13087 | Rating: 0 | rate: / Reply Quote | |
I have updated the recommended client to 6.10.3. You will find that later BOINC versions don't react as badly. A bug was intoroduced back in Feb where the <on_frac> value didn't get updated. 6.10.3 fixed this. The <on_frac> value tells BOINC how much time your computer is on during a day, so it can estimate how long things will take, taking this figure into account. As I don't have a 64 bit windows so I can't comment on not honouring the memory limits. I haven't seen any reports about it on the Boinc_Alpha mailing list. I found 6.10.11 to be stable, for me anyway. The current BOINC version is 6.10.13 which, unless there is a major bug found, will probably become the "release" version soon. ____________ BOINC blog | |
ID: 13088 | Rating: 0 | rate: / Reply Quote | |
Sounds like it's a good time for me to start preparing to check the 64-bit Windows 6.10.11 version for similar problems, then. | |
ID: 13089 | Rating: 0 | rate: / Reply Quote | |
Could you modify the procedure for creating new posts in this thread so that I don't often have to wait until the next day after the last post until I can see the last post without the graphics and ads for the bottom of the thread overlaying it and keeping me from seeing all of the last post? A number of other threads need this as well. | |
ID: 13133 | Rating: 0 | rate: / Reply Quote | |
Since the button to add another post to the thread is visible this time, I've decided to try that and see if it helps. It did help this time. That button isn't always visible in such cases, though. | |
ID: 13134 | Rating: 0 | rate: / Reply Quote | |
Note - I finished uninstalling 6.10.3 and started looking for 6.10.11 to install. Surprise - 6.10.11 is no longer available at any site I've found. You can find all of them here. ____________ BOINC blog | |
ID: 13135 | Rating: 0 | rate: / Reply Quote | |
Note - I finished uninstalling 6.10.3 and started looking for 6.10.11 to install. Surprise - 6.10.11 is no longer available at any site I've found. Thanks. I've just downloaded 6.10.11 and am about to start installing it. | |
ID: 13151 | Rating: 0 | rate: / Reply Quote | |
Have finished installing and found the following: | |
ID: 13154 | Rating: 0 | rate: / Reply Quote | |
The first GPUGRID workunit after that would not start until I adjusted the local preferences to allow GPU use while I use the computer. It then ran about 5 hours, then had a computation error, and have been unable to download another GPUGRID workunit since even though the GPU is still waiting for its next workunit. It keeps asking all the BOINC projects I participate in for GPU workunits, even though none of the others are supposed to have any. Re: The checking for GPU work This is normal behaviour with the BOINC 6.6.xx versions onwards. It doesn't know which projects have GPU apps and which ones have CPU apps, so it asks for each sort. It has backoff mechanisms to it will cut back on the frequency it checks if it doesn't get any for each project. One of recent changes is to allow the projects to tell it not to bother asking for types of work (CPU/ATI or Nvidia) for up to 28 days. As for the error, can you give us a link to the work unit in question and then we can try and see what its doing. ____________ BOINC blog | |
ID: 13158 | Rating: 0 | rate: / Reply Quote | |
The first GPUGRID workunit after that would not start until I adjusted the local preferences to allow GPU use while I use the computer. It then ran about 5 hours, then had a computation error, and have been unable to download another GPUGRID workunit since even though the GPU is still waiting for its next workunit. It keeps asking all the BOINC projects I participate in for GPU workunits, even though none of the others are supposed to have any. This workunit: http://www.gpugrid.net/result.php?resultid=1374844 BOINC finally downloaded and started another workunit, after leaving the GPU idle for several hours. | |
ID: 13166 | Rating: 0 | rate: / Reply Quote | |
Talking about runtimes and me crunching my first NVidia wu for GPUGRID, how long will it take for my XFX 9600GT 512MB to finish the wu? It has run now for almost 9 hours, progress showing 22% now. Other projects are running on the CPU (POEM@Home and WCG). | |
ID: 14105 | Rating: 0 | rate: / Reply Quote | |
Talking about runtimes and me crunching my first NVidia wu for GPUGRID, how long will it take for my XFX 9600GT 512MB to finish the wu? It has run now for almost 9 hours, progress showing 22% now. Other projects are running on the CPU (POEM@Home and WCG). My guess would be around 18-30 hours if it's running continuously. Recent work units have been running between 6 to 9 hours on my GTX280, and judging by this table, your GPU should be about 1/3 the speed of mine. However, my card is factory overclocked, so it's a bit faster than the reference numbers in that table. My last few WUs have been at the high end of the spectrum, so I'm guessing you'll see closer to 30 hours than to 18 hours. ____________ Want to find one of the largest known primes? Try PrimeGrid. Or help cure disease at WCG. | |
ID: 14106 | Rating: 0 | rate: / Reply Quote | |
Thanks for the info Michael. | |
ID: 14110 | Rating: 0 | rate: / Reply Quote | |
Thanks for the info Michael. Just so you know, running other tasks on the CPU has negligable effects on running GPU tasks on virtually all projects. It is nice to now have a selection of projects for GPUs (thoiugh the ATI selection is still weak) so that if one goes down there are others to fill in ... You may be able to also run Colaltz and/or Prime Grid on your card, been a long time since I ran such a low end Nvidia so you will have to try or wait till someone tells me I am all wet ... :) | |
ID: 14130 | Rating: 0 | rate: / Reply Quote | |
I've recently run both Collatz and PrimeGrid successfully on an G105M Nvidia card, which is near the low end of the Nvidia cards now available. | |
ID: 14131 | Rating: 0 | rate: / Reply Quote | |
Message boards : Graphics cards (GPUs) : Run times