Simultaneously starting MCs

Message boards : Multicore CPUs : Simultaneously starting MCs
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
STARBASEn
Avatar

Send message
Joined: 17 Feb 09
Posts: 91
Credit: 1,603,303,394
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwat
Message 49634 - Posted: 6 Jun 2018, 21:16:59 UTC

Started two QC Wu's simultaneously using app version 3.20 and one failed with this stderr report:

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
13:45:21 (5569): wrapper (7.7.26016): starting
13:45:21 (5569): wrapper (7.7.26016): starting
13:45:21 (5569): wrapper: running /bin/bash (-c "flock /var/lib/boinc/projects/www.gpugrid.net/miniconda.lock ./miniconda-installer.sh -b -u -p /var/lib/boinc/projects/www.gpugrid.net/miniconda")
flock: failed to execute ./miniconda-installer.sh: Text file busy
13:45:30 (5569): /bin/bash exited; CPU time 0.000265
13:45:30 (5569): app exit status: 0x45
13:45:30 (5569): called boinc_finish(195)

</stderr_txt>
]]>

Machine is an AMD FX8350 at 4.1 GHz with Fedora 28 kernel 4.16.13 with both gcc and glibc-devel installed.

I had been running both my 8 core machines in 4 core mode and 2 concurrent QC WU's for several months with only one simultaneous start error the whole time but with no other project to compete for the CPU. I found that when running both WCG and QC CPU WU's, simultaneous starts occurred more frequently and unfortunately when it did happen, then failures would leap frog through the queue quickly and blacklist the computer from more work for a day.
ID: 49634 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 49636 - Posted: 6 Jun 2018, 21:35:46 UTC - in response to Message 49634.  

Yes, now simultaneous starts crash with "text file busy". Will look for yet another workaround.
ID: 49636 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 49640 - Posted: 7 Jun 2018, 8:32:32 UTC - in response to Message 49636.  

Attempting fix at version 321
ID: 49640 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
PappaLitto

Send message
Joined: 21 Mar 16
Posts: 513
Credit: 4,673,458,277
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 49641 - Posted: 7 Jun 2018, 10:48:54 UTC

Just got a couple of errors from last night's WUs version 3.20.

Here are the links:
https://www.gpugrid.net/result.php?resultid=17740487
https://www.gpugrid.net/result.php?resultid=17740592

All the rest ran fine.
ID: 49641 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 49642 - Posted: 7 Jun 2018, 12:03:25 UTC - in response to Message 49641.  

^^ These seem connection errors (network down or so)
ID: 49642 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
PappaLitto

Send message
Joined: 21 Mar 16
Posts: 513
Credit: 4,673,458,277
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 49643 - Posted: 7 Jun 2018, 12:37:07 UTC - in response to Message 49642.  

^^ These seem connection errors (network down or so)

Network can affect computation after the WU is already downloaded?
ID: 49643 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kain

Send message
Joined: 3 Sep 14
Posts: 152
Credit: 927,557,369
RAC: 132
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwat
Message 49644 - Posted: 7 Jun 2018, 13:25:26 UTC

I have exactly the same problems so I dont think this is connection related.
ID: 49644 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
captainjack

Send message
Joined: 9 May 13
Posts: 171
Credit: 4,739,796,466
RAC: 1,441
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49645 - Posted: 7 Jun 2018, 13:26:02 UTC

Version 321 looks promising. Just finished two that started at the same time and they finished normally. Just started three at the same time and they are all processing as they should.
ID: 49645 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
STARBASEn
Avatar

Send message
Joined: 17 Feb 09
Posts: 91
Credit: 1,603,303,394
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwat
Message 49646 - Posted: 7 Jun 2018, 14:31:14 UTC

Success with two QC WU's simultaneous start using app 3.21. Both WU's happily crunching @ 1.098% completed so far.

Event Log Excerpt:

Thu 07 Jun 2018 07:22:04 AM MST | GPUGRID | task m0000000638_2278bdac_n00020-SDOERR_QMML50-0-1-RND5957_0 resumed by user
Thu 07 Jun 2018 07:22:04 AM MST | GPUGRID | task m0000000643_5ed133e9_n00020-SDOERR_QMML50-0-1-RND3172_0 resumed by user
Thu 07 Jun 2018 07:22:05 AM MST | GPUGRID | Starting task m0000000638_2278bdac_n00020-SDOERR_QMML50-0-1-RND5957_0
Thu 07 Jun 2018 07:22:05 AM MST | GPUGRID | Starting task m0000000643_5ed133e9_n00020-SDOERR_QMML50-0-1-RND3172_0
ID: 49646 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 49647 - Posted: 7 Jun 2018, 14:55:56 UTC - in response to Message 49643.  

^^ These seem connection errors (network down or so)

Network can affect computation after the WU is already downloaded?


Yes. WUs check the latest version of conda packages/libraries right after start (from conda cloud).
ID: 49647 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mmonnin

Send message
Joined: 2 Jul 16
Posts: 339
Credit: 8,281,341,558
RAC: 3,417
Level
Tyr
Scientific publications
watwatwatwatwat
Message 49652 - Posted: 7 Jun 2018, 20:30:07 UTC - in response to Message 49647.  

^^ These seem connection errors (network down or so)

Network can affect computation after the WU is already downloaded?


Yes. WUs check the latest version of conda packages/libraries right after start (from conda cloud).


Dang is that only after start? If tasks were started, paused, another started, etc could networking be disabled after they start? Or at each start/resume? Not me, but I've heard of some setup schedules to allow downloads at certain times of the day due to varying bandwidth costs.
ID: 49652 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
STARBASEn
Avatar

Send message
Joined: 17 Feb 09
Posts: 91
Credit: 1,603,303,394
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwat
Message 49654 - Posted: 7 Jun 2018, 21:14:00 UTC

Today I have had three successful simultaneous starts and returns without error on two different FX8350 machines. As far as I am concerned, this bug has been resolved at least for the Fedora distro.
ID: 49654 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
PappaLitto

Send message
Joined: 21 Mar 16
Posts: 513
Credit: 4,673,458,277
RAC: 0
Level
Arg
Scientific publications
watwatwatwatwatwatwatwat
Message 49655 - Posted: 8 Jun 2018, 1:40:53 UTC

Haven't gotten a single error with version 3.21 and it's been running all day. What did you change?
ID: 49655 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Toni
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 9 Dec 08
Posts: 1006
Credit: 5,068,599
RAC: 0
Level
Ser
Scientific publications
watwatwatwat
Message 49656 - Posted: 8 Jun 2018, 12:01:04 UTC - in response to Message 49655.  
Last modified: 8 Jun 2018, 12:06:29 UTC

The main change was locking the miniconda directory upon initial installation/update. This in turn required some workarounds. May not be perfect but should be much better.

Regarding network access, it is attempted at each WU start or re-start. The amount of downloaded data should be usually negligible (except the first time).

Edit to add: the network accesses are only to "conda cloud", a python distribution and package manager.
ID: 49656 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
[VENETO] boboviz

Send message
Joined: 10 Sep 10
Posts: 164
Credit: 388,132
RAC: 0
Level

Scientific publications
wat
Message 49657 - Posted: 8 Jun 2018, 15:18:33 UTC

No more error, but a strange behaviour.
Remanining time for every wus are over 20h, but wus are crunched in 40/50 minutes...
ID: 49657 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1424
Credit: 9,189,946,190
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwat
Message 49661 - Posted: 9 Jun 2018, 1:27:58 UTC

Didn't get any response from my post in the cpu tasks thread.

How do you get the cpu tasks? I tried and failed. Is the QC app still considered a Test app? That was the only Preference toggle I didn't select.
ID: 49661 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
klepel

Send message
Joined: 23 Dec 09
Posts: 189
Credit: 4,813,881,008
RAC: 181
Level
Arg
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49662 - Posted: 9 Jun 2018, 1:59:22 UTC - in response to Message 49661.  
Last modified: 9 Jun 2018, 2:02:42 UTC

Did you check: use CPU as well? You might not have allowed it.

No it is not necessary to check BETA tasks. I do not have checked it either, but I do get QC tasks.
ID: 49662 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1424
Credit: 9,189,946,190
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwat
Message 49663 - Posted: 9 Jun 2018, 18:47:33 UTC - in response to Message 49662.  

Yes CPU was checked for use both places and the QC app. Didn't get any QC tasks both time I tried. Just gpu tasks. The scheduler request was for both cpu and gpu work. I know there is plenty of cpu tasks to farm out. Couldn't explain why I didn't get any work. I'll have to try again without gpu work checked I guess.
ID: 49663 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 49664 - Posted: 9 Jun 2018, 21:42:15 UTC - in response to Message 49663.  
Last modified: 9 Jun 2018, 22:28:02 UTC

Yes CPU was checked for use both places and the QC app. Didn't get any QC tasks both time I tried. Just gpu tasks.

That is your problem. The BOINC scheduler can get all mixed up when you select both CPU and GPU work on the same project, and you are eventually left high and dry on one or the other.

There are several discussions on it at Einstein, which has the same problem since they do both CPU and GPU work. Here is one recent discussion, where the moderator explains why the requester is not getting GPU work.
https://einsteinathome.org/content/not-getting-gpu-wus-anymore#comment-165295

I use separate machines for the CPU work and the GPU work on GPUGrid.
ID: 49664 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1424
Credit: 9,189,946,190
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwat
Message 49665 - Posted: 9 Jun 2018, 23:05:59 UTC - in response to Message 49664.  

Thanks for the reply. I don't know. It worked a couple of months ago when the QC app and tasks first showed up. I was crunching both gpu and cpu at the same time. I know that shutting off a gpu request will probably work just to get some of the new QC tasks along with the latest 3.21 app.

I was just wondering how well the app works now with concurrent starts.
ID: 49665 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Multicore CPUs : Simultaneously starting MCs

©2026 Universitat Pompeu Fabra