Experimental Python tasks (beta)

Message boards : News : Experimental Python tasks (beta)
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6

AuthorMessage
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1416
Credit: 9,119,446,190
RAC: 614,515
Level
Tyr
Scientific publications
watwatwatwatwat
Message 56947 - Posted: 11 Jun 2021, 19:15:29 UTC - in response to Message 56946.  

Ah, yes . . . . forgot about that small matter . . . .
ID: 56947 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile trigggl

Send message
Joined: 6 Mar 09
Posts: 25
Credit: 102,324,681
RAC: 0
Level
Cys
Scientific publications
watwatwatwatwatwatwatwat
Message 56949 - Posted: 12 Jun 2021, 17:09:44 UTC - in response to Message 56944.  

I would like to know whether we are supposed to do the things requested in the output file. Things like updating the various packages that are called out.

Or are we supposed to do nothing and let the app/task packagers sort it out before generation?


I'm relatively sure these tasks are sandboxed. the packages being referenced are part of the whole WU (tensorflow). they are installed by the extraction phase in the beginning of the WU. if you check your system you will find that you do not have tensorflow installed most likely.

the package updates need to happen on the project side before distribution to us.

Furthermore, I checked on my Gentoo system what it would take to install Tensorflow. The only vesion available to me required python 3.8. I didn't even bother to check it out because my system is using python 3.9 stable.

Things may become easier with the app if they are able to upgrade to python 3.8. I don't know how this will work with python 3.7. Is it just Gentoo taking the 3.7 option away?

emerge -v1p tensorflow

These are the packages that would be merged, in order:

Calculating dependencies .....

!!! Problem resolving dependencies for sci-libs/tensorflow
... done!

!!! The ebuild selected to satisfy "tensorflow" has unmet requirements.
- sci-libs/tensorflow-2.4.0::gentoo USE="cuda python -mpi -xla" ABI_X86="(64)" CPU_FLAGS_X86="avx avx2 fma3 sse sse2 sse3 sse4_1 sse4_2 -fma4" PYTHON_TARGETS="-python3_8"

The following REQUIRED_USE flag constraints are unsatisfied:
python? ( python_targets_python3_8 )
ID: 56949 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1416
Credit: 9,119,446,190
RAC: 614,515
Level
Tyr
Scientific publications
watwatwatwatwat
Message 56950 - Posted: 12 Jun 2021, 21:38:59 UTC

My Ubuntu 20.04.2 LTS distro has Python 3.8.5 installed so should satisfy the tensorflow requirements.

I'm curious enough to experiment and install tensorflow to see if the tasks will actually do something other than unpack the packages.
ID: 56950 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 11 Jul 09
Posts: 1639
Credit: 10,159,968,649
RAC: 295,172
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56958 - Posted: 15 Jun 2021, 11:11:25 UTC

I was trying to catch some ACEMD tasks to test the oversized upload file report, but got a block of Pythons instead.

http://www.gpugrid.net/result.php?resultid=32623625

Can anyone advise on the multitude of gcc failures, starting with

    gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
    compilation terminated.

Machine is Linux Mint 20.1, freshly updated today (including BOINC v7.16.17, which is an auto-build test for the Mac release last week - not relevant here, but useful to keep an eye on to make sure they haven't broken anything else).

I have a couple of spare tasks suspended - I'll look through the actual runtime packaging to see what they're trying to achieve.
ID: 56958 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1416
Credit: 9,119,446,190
RAC: 614,515
Level
Tyr
Scientific publications
watwatwatwatwat
Message 56960 - Posted: 15 Jun 2021, 13:58:15 UTC - in response to Message 56958.  

Richard, I haven't had any GCC errors with any of the Python tasks on my hosts.

Never see it invoked.

Just see lots of unpacking and attempts to run other ML programs.
ID: 56960 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 11 Jul 09
Posts: 1639
Credit: 10,159,968,649
RAC: 295,172
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56962 - Posted: 15 Jun 2021, 14:54:08 UTC - in response to Message 56960.  

Interestingly, the task that failed to run gcc was re-issued to a computer on ServicEnginIC's account - and ran successfully. That gives me a completely different stderr_txt file to compare with mine. I'll make a permanent copy of both for reference, and try to work out what went wrong.
ID: 56962 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ServicEnginIC
Avatar

Send message
Joined: 24 Sep 10
Posts: 592
Credit: 11,972,186,510
RAC: 998,578
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56963 - Posted: 15 Jun 2021, 15:27:38 UTC - in response to Message 56962.  

Interestingly, the task that failed to run gcc was re-issued to a computer on ServicEnginIC's account - and ran successfully. That gives me a completely different stderr_txt file to compare with mine. I'll make a permanent copy of both for reference, and try to work out what went wrong.

I remember that I applied to all my hosts a kind remedy suggested by you at your message #55967.
I related it at message #55986
Thank you again.
ID: 56963 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 11 Jul 09
Posts: 1639
Credit: 10,159,968,649
RAC: 295,172
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56964 - Posted: 15 Jun 2021, 15:48:43 UTC - in response to Message 56963.  

Thanks for the kind words. Yes, that's necessary, but not sufficient. My host 132158 got a block of four tasks when I re-enabled work fetch this morning.

The first task I ran - ID _621 - got

[1008937] INTERNAL ERROR: cannot create temporary directory!
11:23:17 (1008908): /usr/bin/flock exited; CPU time 0.132604

- that same old problem. I stopped the machine, did a full update and restart, and verified that the new .service file had the fix for that bug.

Then I fired off task ID _625 - that's the one with the cpp errors.

Unfortunately, we only get the last 64 KB of the file, and it's not enough in this case - we can't see what stage it's reached. But since the first task only ran 3 seconds, and the second lasted for 190 seconds, I assume we fell at the second hurdle.

My second Linux machine has just picked up two of the ADRIA tasks I was hunting for - I'll sort those out next.
ID: 56964 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
valterc

Send message
Joined: 21 Jun 10
Posts: 21
Credit: 10,863,141,443
RAC: 2,947,209
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56975 - Posted: 16 Jun 2021, 12:11:07 UTC - in response to Message 56958.  

I was trying to catch some ACEMD tasks to test the oversized upload file report, but got a block of Pythons instead.

http://www.gpugrid.net/result.php?resultid=32623625

Can anyone advise on the multitude of gcc failures, starting with

    gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
    compilation terminated.

Machine is Linux Mint 20.1, freshly updated today (including BOINC v7.16.17, which is an auto-build test for the Mac release last week - not relevant here, but useful to keep an eye on to make sure they haven't broken anything else).

I have a couple of spare tasks suspended - I'll look through the actual runtime packaging to see what they're trying to achieve.

I got a similar error some time ago. A memory module was faulty, started to get segmentation fault errors. Eventually my compiling environment (gcc etc.) became messed up. Solved the situation by removing the bad module and completely reinstalling the compiling environment. What I might suggest to do is try to verify if gcc/g++ are actually working, by compiling something of your choice.


ID: 56975 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 11 Jul 09
Posts: 1639
Credit: 10,159,968,649
RAC: 295,172
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56980 - Posted: 17 Jun 2021, 16:29:54 UTC

Finally got time to google my gcc error. Simples: turns out the app requires g++, and it's not installed by default on Ubuntu - and, one assumes, derivatives like mine.

All it needed was

sudo apt-get install g++

No restart needed, of either BOINC or Linux, and task 32623619 completed successfully.

No much sign of any checkpointing: one update at 10%, then nothing until the end.
ID: 56980 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56981 - Posted: 17 Jun 2021, 16:40:25 UTC - in response to Message 56980.  

Finally got time to google my gcc error. Simples: turns out the app requires g++, and it's not installed by default on Ubuntu - and, one assumes, derivatives like mine.

Hummm...
I have run a few on Ubuntu 20.04.2 and did not do anything special, unless maybe something else I was working on required it.
http://www.gpugrid.net/results.php?hostid=452287
ID: 56981 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.

Send message
Joined: 21 Feb 20
Posts: 1114
Credit: 40,838,535,595
RAC: 4,302,611
Level
Trp
Scientific publications
wat
Message 56982 - Posted: 17 Jun 2021, 17:47:38 UTC - in response to Message 56980.  
Last modified: 17 Jun 2021, 17:50:31 UTC

Finally got time to google my gcc error. Simples: turns out the app requires g++, and it's not installed by default on Ubuntu - and, one assumes, derivatives like mine.

All it needed was

sudo apt-get install g++

No restart needed, of either BOINC or Linux, and task 32623619 completed successfully.

No much sign of any checkpointing: one update at 10%, then nothing until the end.


I think this just might be your distribution. I never installed this (specifically) on my Ubuntu 20.04 systems. if it's there, it was there by default or through some other package as a dependency.
ID: 56982 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Richard Haselgrove

Send message
Joined: 11 Jul 09
Posts: 1639
Credit: 10,159,968,649
RAC: 295,172
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56983 - Posted: 17 Jun 2021, 18:20:34 UTC - in response to Message 56982.  

This was a fairly recent (February 2021) clean installation of Linux Mint 20.1 'Ulyssa' - I decided to throw away my initial fumblings with Mint 19.1, and start afresh. So, let this be a warning: not every distro is as complete as you might expect.

Anyway, the solution is in public now, in case anyone else needs it.
ID: 56983 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.

Send message
Joined: 21 Feb 20
Posts: 1114
Credit: 40,838,535,595
RAC: 4,302,611
Level
Trp
Scientific publications
wat
Message 56999 - Posted: 22 Jun 2021, 12:11:05 UTC

errors on the Python tasks again.

https://www.gpugrid.net/result.php?resultid=32626713
ID: 56999 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 57001 - Posted: 22 Jun 2021, 15:39:11 UTC - in response to Message 56999.  

errors on the Python tasks again.

I see them too.

http://www.gpugrid.net/results.php?hostid=452287
UnsatisfiableError: The following specifications were found to be incompatible with each other:

That will give them something to work on.
ID: 57001 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile robertmiles

Send message
Joined: 16 Apr 09
Posts: 503
Credit: 769,991,668
RAC: 0
Level
Glu
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 57006 - Posted: 23 Jun 2021, 0:51:34 UTC

I'm now using my GPU preferably for tasks related to medical research. Could you mention whether the Python tasks are related to medical research and whether they use the GPU?
ID: 57006 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.

Send message
Joined: 21 Feb 20
Posts: 1114
Credit: 40,838,535,595
RAC: 4,302,611
Level
Trp
Scientific publications
wat
Message 57007 - Posted: 23 Jun 2021, 3:07:49 UTC - in response to Message 57006.  

I'm now using my GPU preferably for tasks related to medical research. Could you mention whether the Python tasks are related to medical research and whether they use the GPU?


right now these Python tasks are using machine learning to do what we assume is some medical kind of research, but the admins haven't given many specifics on exactly how or what type of medical research specifically is being done. GPUGRID as a whole does various types of medical research. see more info about it in the other thread here: https://www.gpugrid.net/forum_thread.php?id=5233

the tasks are labelled as being GPU tasks, and they do otherwise reserve the GPU in BOINC (ie, other tasks wont run on it), however in reality the GPU is not actually used. it sits idle and all the computation happens on the CPU thread that's assigned to the job. the admins have stated that these early tasks are still in testing and they will use the GPU in the future. but right now they don't.

the other thing to keep in mind, the Python application is Linux-only (at least right now). you wont be able to get these tasks on your Windows system.
ID: 57007 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Keith Myers
Avatar

Send message
Joined: 13 Dec 17
Posts: 1416
Credit: 9,119,446,190
RAC: 614,515
Level
Tyr
Scientific publications
watwatwatwatwat
Message 57036 - Posted: 29 Jun 2021, 22:56:36 UTC

Just finished a new Python task that didn't error out. Hope this is the start of a trend.
ID: 57036 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 . . . 3 · 4 · 5 · 6

Message boards : News : Experimental Python tasks (beta)

©2025 Universitat Pompeu Fabra