"Hanging" WU?

Message boards : Multicore CPUs : "Hanging" WU?
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8708 - Posted: 22 Apr 2009, 3:41:05 UTC

Well, I have one of the new CELLGA_SHORT WUs running for over one day and 5 hours now, still showing 0% done. Is it save to assume the CPU time is wasted and the WU will never complete? ;-) Should I abort it? It's this one - http://www.gpugrid.net/result.php?resultid=555615

pixelicious.at - my little photoblog
ID: 8708 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 8717 - Posted: 22 Apr 2009, 12:58:58 UTC - in response to Message 8708.  
Last modified: 22 Apr 2009, 13:46:28 UTC

Yes, please, abort it. It was expected to run for 2 hours, roughly. Thank for reporting the problem!
ID: 8717 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8725 - Posted: 22 Apr 2009, 14:58:35 UTC - in response to Message 8717.  

Ok, thanks! Aborted it now after 1 day and 16 hours... My nice credits... ;-)

pixelicious.at - my little photoblog
ID: 8725 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8727 - Posted: 22 Apr 2009, 15:49:26 UTC

Maybe it wasn't a problem with the task but with my PS3.
Now I got a TONI_CELLGA task that also seemed to hang because it showed no progress after 20 minutes, but after a reboot it is running fine now...

pixelicious.at - my little photoblog
ID: 8727 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8732 - Posted: 22 Apr 2009, 17:25:07 UTC

Hmmm, after 20 minutes the next task was hanging, this time a TONI_CELLGA_MED - http://www.gpugrid.net/result.php?resultid=565176. I aborted it, and will try another task...

pixelicious.at - my little photoblog
ID: 8732 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist

Send message
Joined: 14 Mar 07
Posts: 1958
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 8817 - Posted: 24 Apr 2009, 7:14:49 UTC - in response to Message 8732.  

If a workunit hangs. You can just try to restart the machine.
The problem is not usually with the wu, but with the fact that your processor is reserved by another application.

gdf
ID: 8817 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8819 - Posted: 24 Apr 2009, 8:29:29 UTC

Thanks Gianni, but I think I just found the problem...

I just started my PS3 again, started BOINC and got a hanging task again. It was running for 2:37 minutes and then the CPU time was hanging with 0% progress.

When I started BOINC it was running CPU benchmarks, but after a few minutes it was running the benchmarks again. The problem is boincmgr showed "suspending computation - running CPU benchmarks", but "top" still showed 154% CPU for cellmd2_5.03_po. Another thing I noticed was the time shown in boincmgr. First it was wrong and when it started the second benchmarks it showed the right time (see screenshot). So I don't know if the task was hanging because it wasn't suspended during benchmarks or because the system time has changed. Weird thing is NTP is turned on, and the system time should always be right...

I think I'll try to delete the whole boinc dir and redownload BOINC and if that doesn't help I'll try a new YDL 6.1 install...



pixelicious.at - my little photoblog
ID: 8819 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JaRski-S60R

Send message
Joined: 31 Dec 07
Posts: 27
Credit: 224,325,256
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 8822 - Posted: 24 Apr 2009, 10:04:27 UTC

Have same problem;

http://www.gpugrid.net/workunit.php?wuid=399536
http://www.gpugrid.net/workunit.php?wuid=405292

lot's of hrs and energy wasted again.
I'm not requesting new WU for a while and only do work for yoyo.
ID: 8822 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 8823 - Posted: 24 Apr 2009, 10:46:39 UTC - in response to Message 8822.  
Last modified: 24 Apr 2009, 11:06:00 UTC

I'm trying to figure out the pattern behind those failures.

These new "short" WUs, called "SHORT" and "MED", are expected to run for <3 hours and grant approximately 300 credits. Another difference with respect to other WUs is that they generate a largish result file (*_4, approx 16M).

The WUs crunch correctly on most machines - hopefully we'll be able to reproduce the problem.
ID: 8823 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Stefan Ledwina
Avatar

Send message
Joined: 16 Jul 07
Posts: 464
Credit: 298,573,998
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwatwatwat
Message 8824 - Posted: 24 Apr 2009, 11:10:24 UTC - in response to Message 8823.  

Thanks for looking into the problem Toni!

I previously deleted the BOINC directory, redownloaded BOINC and got a GRA_US task. Seems this one is running ok now. It`s almost at 10% after 4 hours crunching...


pixelicious.at - my little photoblog
ID: 8824 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 8827 - Posted: 24 Apr 2009, 11:29:59 UTC - in response to Message 8824.  
Last modified: 24 Apr 2009, 12:02:17 UTC

Thanks to you for reporting the symptoms: "runaway" processes may in fact be related to anomalous task suspend/resume, triggered by the CPU benchmarks.

To relieve the problem, I'm postponing some WUs and lowering the FP bound, which will hopefully halt "runaway" jobs before they hog the CPU for days (the changes will take time to propagate, though).
ID: 8827 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JaRski-S60R

Send message
Joined: 31 Dec 07
Posts: 27
Credit: 224,325,256
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 8908 - Posted: 25 Apr 2009, 13:26:40 UTC - in response to Message 8827.  
Last modified: 25 Apr 2009, 13:28:45 UTC

Hi TG,
No benchmark was running during start and the time I cancelled the WU, below I inserted the log from the moment I started till I cancelled it.
Hope you can fix it or at least build in a max running-time for certain WU.
Will follow the forums to see if things get better and I'll be back than.


do 23 apr 2009 18:27:40 MDT|GPUGRID|Starting A8-TONI_CELLGA_MED_4-6-40-RND6539_1
do 23 apr 2009 18:27:40 MDT|GPUGRID|Starting task A8-TONI_CELLGA_MED_4-6-40-RND6539_1 using cellmd2 version 503
do 23 apr 2009 23:27:22 MDT|GPUGRID|Sending scheduler request: Requested by project
do 23 apr 2009 23:27:22 MDT|GPUGRID|(not requesting new work or reporting completed tasks)
do 23 apr 2009 23:27:27 MDT|GPUGRID|Scheduler RPC succeeded [server version 607]
do 23 apr 2009 23:27:27 MDT|GPUGRID|Deferring communication for 31 sec
do 23 apr 2009 23:27:27 MDT|GPUGRID|Reason: requested by project
vr 24 apr 2009 04:27:28 MDT|GPUGRID|Sending scheduler request: Requested by project
vr 24 apr 2009 04:27:28 MDT|GPUGRID|(not requesting new work or reporting completed tasks)
vr 24 apr 2009 04:27:34 MDT|GPUGRID|Scheduler RPC succeeded [server version 607]
vr 24 apr 2009 04:27:34 MDT|GPUGRID|Deferring communication for 31 sec
vr 24 apr 2009 04:27:34 MDT|GPUGRID|Reason: requested by project
vr 24 apr 2009 09:27:34 MDT|GPUGRID|Sending scheduler request: Requested by project
vr 24 apr 2009 09:27:34 MDT|GPUGRID|(not requesting new work or reporting completed tasks)
vr 24 apr 2009 09:27:44 MDT|GPUGRID|Scheduler RPC succeeded [server version 607]
vr 24 apr 2009 09:27:44 MDT|GPUGRID|Deferring communication for 31 sec
vr 24 apr 2009 09:27:44 MDT|GPUGRID|Reason: requested by project
vr 24 apr 2009 11:05:05 MDT|GPUGRID|Deferring communication for 1 min 0 sec
vr 24 apr 2009 11:05:05 MDT|GPUGRID|Reason: Unrecoverable error for result A8-TONI_CELLGA_MED_4-6-40-RND6539_1 (aborted by user)
vr 24 apr 2009 11:05:05 MDT|GPUGRID|Computation for task A8-TONI_CELLGA_MED_4-6-40-RND6539_1 finished
vr 24 apr 2009 11:05:05 MDT|yoyo@home|Resuming task ogr_090407202003_15_0 using crunch version 211
ID: 8908 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Quasar

Send message
Joined: 17 Dec 08
Posts: 6
Credit: 1,943,937
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 8938 - Posted: 25 Apr 2009, 20:50:04 UTC

Could it be that this problem occurs when more than one project are using the PS3 (maybe it's related to task-switching)? I've been crunching on my PS3 for a couple of months without problems but had this problem yesterday once I attached BOINC to yoyo@home. I aborted the WU and got another one but this one seems to be hanging too. Here are the links to the WUs:

http://www.gpugrid.net/workunit.php?wuid=407696
http://www.gpugrid.net/workunit.php?wuid=407774
ID: 8938 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 8987 - Posted: 27 Apr 2009, 12:15:53 UTC - in response to Message 8938.  
Last modified: 27 Apr 2009, 12:17:46 UTC

Yes, we believe that task switching is an issue: the accelerated processors are not properly "freed" somehow upon process termination, probably a shortcoming of the platform. :-( Do just PS3GRID hang, or also those from other projects?
ID: 8987 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Quasar

Send message
Joined: 17 Dec 08
Posts: 6
Credit: 1,943,937
RAC: 0
Level
Ala
Scientific publications
watwatwatwatwat
Message 9037 - Posted: 27 Apr 2009, 22:17:27 UTC

I think all the WUs, from PS3Grid and others, hang, at least that's what happened with me. I was looking around and found a newer BOINC client optimized for the PS3 (http://www.dotsch.de/boinc/boinc6219_10.linux-ps3.tar.gz), unfortunately it's a command-line version and I tried to get it to work with BOINC Manager but it didn't work. Maybe an update to the current PS3Grid BOINC client (whether using the one mentioned above or otherwise) would fix the problem?
ID: 9037 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
jboese

Send message
Joined: 30 Jul 08
Posts: 21
Credit: 31,229
RAC: 0
Level

Scientific publications
wat
Message 9098 - Posted: 29 Apr 2009, 5:32:47 UTC - in response to Message 8987.  
Last modified: 29 Apr 2009, 5:33:50 UTC

Just to be helpful to another BOINCer at least on my machine the hanging WU seem to be specific to gpugrid. If you set the resource share for yoyo to say 3E+38 (a huge number) and gpugrid to 1 then your ps3 will happily only crush yoyo WU and will not hang. The problem only seems to occur when the PS3 starts working on a gpugrid WU. I am not sure but the problem also seems more pronounced with the memory stick version (what I run) as compared to YDL but I think it occurs with both at times.
ID: 9098 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
sam

Send message
Joined: 30 Apr 09
Posts: 15
Credit: 228,425
RAC: 0
Level

Scientific publications
wat
Message 9156 - Posted: 30 Apr 2009, 20:02:01 UTC

just joined the gpugrid; first wu is hung 0.0%

Thu 30 Apr 2009 04:48:38 PM EDT|GPUGRID|Starting task 5-TONI_CELLGA_MED_9-0-2-RND0374_0 using cellmd2 version 503

posted because the admin mentioned trying to discern a pattern. I have not tried another ps3 based project. I just setup the machine with ydl6.1, basically vanilla.



ID: 9156 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
sam

Send message
Joined: 30 Apr 09
Posts: 15
Credit: 228,425
RAC: 0
Level

Scientific publications
wat
Message 9160 - Posted: 30 Apr 2009, 20:33:23 UTC

I guess I picked a bad day to join. I booted; no worky. aborted. started another, no worky. then I see a message suggesting I joined the project with the wrong link, the one from the website. so I tried again, new wu: Thu 30 Apr 2009 05:20:10 PM EDT|GPUGRID|Starting task 638000-IBUCH_GRAUS-1-100-RND2278_1 using cellmd2 version 503

no worky. 0.0% I just suspended the wu as I will my effort for today.

I am open recommendations regarding this project and/or ydl6.1; I am new to both.

thanks.
ID: 9160 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 9174 - Posted: 1 May 2009, 11:37:14 UTC - in response to Message 9160.  

Hi sam - the progress bar does not show the actual progression for PS3 WUs. All WUs last circa 12 hours each.
ID: 9174 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JaRski-S60R

Send message
Joined: 31 Dec 07
Posts: 27
Credit: 224,325,256
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 9189 - Posted: 2 May 2009, 8:10:59 UTC - in response to Message 9174.  
Last modified: 2 May 2009, 8:14:24 UTC

1 WU since a week i crunched for gpugrid and hung, +16hrs and 0%

http://www.gpugrid.net/result.php?resultid=598295
ID: 9189 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Multicore CPUs : "Hanging" WU?

©2026 Universitat Pompeu Fabra