On Linux Mint hosts 537311 and 508381

Sample error messages:
[26912] INTERNAL ERROR: cannot create temporary directory!
[26916] INTERNAL ERROR: cannot create temporary directory!

wrapper uses absolute path (from root)
in my case, the path contains spaces - so the path is transmitted truncated


other projects use relative path - from slot/N/

it's interesting that even though all the wingmen experience some error, they are not receiving the same error, and some are running for quite a while before erroring out.
____________

My Linux machines run under the systemd daemon install. We found a problem recently where systemd was set up to have no access at all to the Linux tmp area - couldn't even read it. My machines are now able to read there, but may still be prevented from writing - wherever flock is trying to unpack python for installation.

Anybody successful yet in the new Python tasks?

I have just watched two tasks run and are currently at 100% progress and time remaining, dashed out. Runtime currently at 18 minutes for one and 12 minutes for the other, and counting.

Maybe making progress in the task development. At least ran for more than the couple of seconds of the previous tries.

Looks like Kevvy's system completed a bunch of them.

And also was awarded over 1,500,000 credits for each one! I think the credit award calculation is off a bit LOL. its about 10x the credit given from MDAD
____________

Well, if mine really are going to run for 3 hours, then that is about 1.5X longer than ADRIA tasks on my hosts.

So at least 1.5X in credits.

But agree the credit awarded is way out there.

But maybe an exorbitant award for being a beta tester.

looks like I picked up a couple resends (i only just enabled Beta tasks). I'll see if they complete on my system. both picked up on my RTX 2070 system.
____________

Everyone has failed again on my two hosts.

I wonder what Mr. Kevvy has done right for the crunching environment to get them to complete successfully?

Finally had success on several new Python tasks. Wonder if the failures are logged to forward correct the task configuration so that future attempts are finished correctly.

I thought I'd turned off Python tasks after the failures, but I forgot to turn off test tasks as well, so I've been sent another one.

The initial runtime estimate is 43 seconds...

I'll go and grab the full task specification before it runs, and see if I can spot anything.

The latest round of tasks look fairly good.
The current batch of errors I have seen are related to disk space and proxy pass through errors (download anaconda environment doesn't use proxy)

Can we post the STDerr output for errored tasks to see what other issues remain?

Well, it failed again, with the program 'flock' failing to create a temporary directory. That's the very first task in the job spec, and it seems to be trying to install miniconda.

The strange thing is that I seem to have a full miniconda installation already, created on 11 November, but that job has long since vanished from my task list.

Forgot to mention - a machine belonging to ServicEnginIC has also tried the same task, and failed in the same way.

Well, it failed again, with the program 'flock' failing to create a temporary directory. That's the very first task in the job spec, and it seems to be trying to install miniconda.

The strange thing is that I seem to have a full miniconda installation already, created on 11 November, but that job has long since vanished from my task list.

Forgot to mention - a machine belonging to ServicEnginIC has also tried the same task, and failed in the same way.

Looks like I have 11 successful tasks, and 2 failures.

the two failures both failed with "196 (0xc4) EXIT_DISK_LIMIT_EXCEEDED" after a few mins and on different hosts.
https://www.gpugrid.net/result.php?resultid=31680145
https://www.gpugrid.net/result.php?resultid=31678136

curious, since both systems have plenty of free space, and I've allowed BOINC to use 90% of it.

these tasks also have much different behavior compared to the default new version acemd tasks. and they don't seem well optimized yet.
-less reliance on PCIe bandwidth, seeing 2-8% PCIe 3.0 bus utilization
-more reliance on GPU VRAM, seeing 2-3GB memory used
-less GPU utilization, seeing 65-85% GPU utilization. (maybe more dependent on a fast CPU/mem subsystem. my 3900X system gets better GPU% than my slower EPYC systems)

contrast that with the default acemd3 tasks:
-25-50% PCIe 3.0 bus utilization
-about 500MB GPU VRAM used
-95+% GPU utilization

thinking about the GPU utilization being dependent on CPU speed. It could also have to do with the relative speed between the GPU:CPU. just something I observed on my systems. slower GPUs seem to tolerate slower CPUs better, which makes sense if the CPU speed is a limiting factor.

Ryzen 3900X @4.20GHz w/ 2080ti = 85% GPU Utilization
EPYC 7402P @3.30GHz w/ 2080ti = 65% GPU Utilization
EPYC 7402P @3.30GHz w/ 2070 = 76% GPU Utilization
EPYC 7642 @2.80GHz w/ 1660Super = 71% GPU Utilization

needs more optimization IMO. the default app sees much better performance keeping the GPU fully loaded.
____________

Several failures for me as well.
Some because of time-limit 1800 which abort themselves after 1786 sec.
Unhid my machines for the case someone is interested in the stderr output.
There might be different output on different machines, so just help yourselves.
____________
- - - - - - - - - -
Greetings, Jens

Just spitballing, but it seems the people failing all their tasks are using a repository/service install of BOINC.

I see successful completions from myself, Keith, and Kevvy. all of us run an all-in-one install from an executable in our home directories.

all of the failures on repo/service installs might be down to a permissions issue with how the new app is trying to work.
____________

First thing I would ask of the BOINC service install users, is their user added to the boinc group.

Also is video added to the boinc group.

Permissions get tricky with the service BOINC install.

It appears the tmp folder is a sub folder of the gpugrid.net project folder - miniconda and in the slot N folder gpugridpy.

YAY! I got one, and it's running. Installed conda, got through all the setup stages, and is now running ACEMD, and reporting proper progress.

Task 31712246, in case I go to bed before it finishes.

YAY! I got one, and it's running. Installed conda, got through all the setup stages, and is now running ACEMD, and reporting proper progress.

Task 31712246, in case I go to bed before it finishes.

I applied the (very few) system updates that were waiting. I don't think anything would have been relevant to running tasks. [I remember a new version of curl, and of SSL, which might have affected external comms, but nothing caught my eye apart from those]

I spotted one problem, though. Initial estimate was 1 minute 46 seconds. I checked the <rsc_fpops_bound>, and it was a mere x50 over <rsc_fpops_est>. At the speed it was running, it would probably have timed out - so I've given it a couple of extra noughts on the bound.

The ACEMD phase started at 10% done, and is counting up in about 1% increments. Didn't seem to have checkpointed before I stopped it for the bound change, so I lost about 15 minutes of work.

you probably didn't need to change the bounds to avoid a timeout. my early tasks all estimated sub 1min run times, but all ran to completion anyway, even my 1660Super which took 5+hrs to run.
____________

Somebody mentioned a time out, so I thought it was better to be safe than sorry. After all the troubles, it would have been sad to lose a successful test to that (besides being a waste of time. At least the temp directory errors only wasted a couple of seconds!).

The last change to the BOINC service installation came as a result of https://github.com/BOINC/boinc/issues/3715#issuecomment-727609578. Previously, there was no access at all to /tmp/, which blocked idle detection.

YAY! I got one, and it's running. Installed conda, got through all the setup stages, and is now running ACEMD, and reporting proper progress.

I'm not entirely convinced by that. It's certainly the final destination for the installation, but is it the temp directory as well?

Call me naive, but wouldn't a generic temp folder be created in /tmp/, unless directed otherwise?

Ian&Steve C. wrote:

Looks like I have 11 successful tasks, and 2 failures.

the two failures both failed with "196 (0xc4) EXIT_DISK_LIMIT_EXCEEDED" after a few mins and on different hosts.
https://www.gpugrid.net/result.php?resultid=31680145
https://www.gpugrid.net/result.php?resultid=31678136

curious, since both systems have plenty of free space, and I've allowed BOINC to use 90% of it.

Several failures for me as well.
Some because of time-limit 1800 which abort themselves after 1786 sec.
Unhid my machines for the case someone is interested in the stderr output.
There might be different output on different machines, so just help yourselves.

EDIT:
[quote]I'm not entirely convinced by that. It's certainly the final destination for the installation, but is it the temp directory as well?

Call me naive, but wouldn't a generic temp folder be created in /tmp/, unless directed otherwise?

Debian based distros seem to use /proc/locks directory for lock file location.
If you use lslocks command, you will see the current locks, including any boinc/Gpugrid locks. (cat /proc/locks will also work, but you will need to know the PID of the processes listed to interpret the output)
I cant think of anything that would prevent access to this directory structure, as it is critical in OS operations.
I am leaning towards a bug in the Experimental Tasks scripts that causes this issue (and has since been cleaned up).

gemini8
3 of your hosts with older CUDA driver report this: ACEMD failed: Error loading CUDA module: CUDA_ERROR_INVALID_PTX (218).
Perhaps time for a driver update. It wont fix the errors being reported (reported errors are task bugs), but may prevent other issues developing in the future

Not a technical guru like you all here, but if it helps, my system has had one Anaconda failure back on 4 Dec, and now completed 4 in the last several days with one more in progress. Let me know if I can provide any information that helps.

Richard Haselgrove:
Your suggested action at Message 55967 seems to have worked for me also.
Thank you very much!

I had many failed Anaconda Python tasks, reporting errors regarding:

INTERNAL ERROR: cannot create temporary directory!

sudo systemctl edit boinc-client.service

[Service]
PrivateTmp=true

The systemd file as distributed now has PrivateTmp=false, and my machine had that setting when the error messages appeared. I changed it back, so I now have PrivateTmp=true again. And with that one change (plus an increased time limit for safety), the task has completed successfully.

I have a second machine, where I haven't yet made that change. Compare tasks 31712246 (WU created 10 Dec 2020 | 20:54:14 UTC, PrivateTmp=true) and 31712588 (WU created 10 Dec 2020 | 21:05:06 UTC, PrivateTmp=false). The first succeeded, the second failed with 'cannot create temporary directory'. I don't think it was a design change by the project.

21:48:18 (21729): wrapper: running /usr/bin/flock (/var/lib/boinc-client/projects/www.gpugrid.net/miniconda.lock -c "/bin/bash ./miniconda-installer.sh -b -u -p /var/lib/boinc-client/projects/www.gpugrid.net/miniconda &&
/var/lib/boinc-client/projects/www.gpugrid.net/miniconda/bin/conda install -m -y -p gpugridpy --file requirements.txt ")
[21755] INTERNAL ERROR: cannot create temporary directory!
[21759] INTERNAL ERROR: cannot create temporary directory!
21:48:19 (21729): /usr/bin/flock exited; CPU time 0.118700

Richard Haselgrove:
Your suggested action at Message 55967 seems to have worked for me also.
Thank you very much!

I had many failed Anaconda Python tasks, reporting errors regarding:

INTERNAL ERROR: cannot create temporary directory!

I executed the stated command:

sudo systemctl edit boinc-client.service

And I added to the file the suggested lines:

[Service]
PrivateTmp=true

After saving changes and rebooting, the mentioned error vanished.
Task 2p95010002r00-RAIMIS_NNPMM-0-1-RND3897_0 has succeeded on my Host #482132.
This host had previously failed 28 Anaconda Python tasks one after another...
I've applied the same remedy to several other hosts, and three more tasks seem to be progressing normally.

You have hit the mark

Yes, I too would like to hear the feedback from the developers, in my case what can be done with the project DCF of 93 on all my hosts.

That makes GPUGrid completely monopolize the gpus and prevent my other gpu projects from running.

The standard new acemd application tasks are the ones that are causing the week long estimated completion times.

The new beta anaconda/python tasks have reasonable estimated completion times.

The problem is the disparate individual DCF values for each application with a 100:1 ratio for python to acemd3 and BOINC only being able to handle one DCF value from a project with multiple sub-apps.

My current solution is manual intervention by stopping allowing new work and finishing off what is in the cache so that the other gpu projects can work on their caches. Then I let the other gpu projects run for a day or so so that the REC values of the projects are close to balancing out.

But this is an undesirable solution. I can only think that stopping one or the other sub-apps and exclusively running only one app is the only permanent solution now once the DCF normalizes against a single application.

On my hosts that are not reporting this issue, PrivateTmp is not set (so defaults to false)

On my hosts that are not reporting this issue, PrivateTmp is not set (so defaults to false)

All your three successful hosts have 7.9.3 BOINC version in use.
That version was default to PrivateTmp=true
Newer versions 7.16.xx changed this criterium to PrivateTmp=false by default.

Thanks guys. I've written up the problem for upstream at https://github.com/BOINC/boinc/issues/4125, and I'm hoping to get further ideas from there. At the moment, the situation seems to be that you can either use idle detection to keep BOINC out of the way when working, or run Python tasks, but not both together. That seems unsatisfactory.

There are a couple of suggestions so far:

either adding /var/tmp and /tmp to ReadWritePaths=
or commenting out:
ProtectSystem=strict

the tmp directory creation error is reported on a different PID to the wrapper.

does ProtectHome=true imply PrivateTmp=true (or vice versa)?

what can be done with the project DCF of 93 on all my hosts.

Newer versions 7.16.xx changed this criterium to PrivateTmp=false by default.

Thanks guys. I've written up the problem for upstream at https://github.com/BOINC/boinc/issues/4125, and I'm hoping to get further ideas from there. At the moment, the situation seems to be that you can either use idle detection to keep BOINC out of the way when working, or run Python tasks, but not both together. That seems unsatisfactory.

Either way, the project has the BOINC writable directory - why not use that? You can make your own tmp directory there if I'm not mistaken (haven't tried it)

His suggestion will mean any temp files will be protected by the Boinc security system already in place, bypasses the need for PrivateTmp=true and be accessible for Boinc process calls.

His suggestion will mean any temp files will be protected by the Boinc security system already in place, bypasses the need for PrivateTmp=true and be accessible for Boinc process calls.

Which circles back to how flock uses /tmp directory. Perhaps BryanQuigley can clear that point up.

21:48:18 (21729): wrapper: running /usr/bin/flock (/var/lib/boinc-client/projects/www.gpugrid.net/miniconda.lock -c "/bin/bash ./miniconda-installer.sh -b -u -p /var/lib/boinc-client/projects/www.gpugrid.net/miniconda &&
/var/lib/boinc-client/projects/www.gpugrid.net/miniconda/bin/conda install -m -y -p gpugridpy --file requirements.txt ")
[21755] INTERNAL ERROR: cannot create temporary directory!
[21759] INTERNAL ERROR: cannot create temporary directory!
21:48:19 (21729): /usr/bin/flock exited; CPU time 0.118700

I think it's more likely to be the /bin/bash which actually needs to write temporary files. I've now managed (with some difficulty) to separate the 569 lines of actual script from the 90 MB of payload. There's

For those users waiting to receive these new Anaconda Python tasks:
- They are currently available only for Linux OS based hosts, so they won't be sent to Windows OS based hosts. This can be seen at https://www.gpugrid.net/apps.php, and was announced by Toni at Message #55588.
- And (to be confirmed): Not all GPUs currently processing ACEMD3 WUs under Linux environment are eligible for processing Python WUs.

This second assertion is based in my own experience, and needs further confirmation.
None of my 2 GB RAM Graphics cards have received Python WUs so far.
I suspect that graphics card with more than 2 GB internal RAM is a requirement (?)
The most clear example sustaining this theory:
My Host #325908 received two Python tasks at the same time.
It is a double GPU host, GPU #0 being a GTX 950 with 2 GB internal RAM, and GPU #1 being a GTX 1650 with 4 GB internal RAM.
As soon as GPU #1 got free, it started processing the first Python task.
As can be seen at previous image, more than 2 GB of internal memory (2377 MB) were in use.
Compared to this, GPU #0 was processing an ACEMD3 task, requiring only 210 MB RAM for this.
After finishing this ACEMD3 WU, GTX 950 did not start the waiting Python task, but it asked to scheduler for work and it started a new downloaded ACEMD3 WU.
The waiting second Python WU was started at GTX 1650 GPU as soon as it finished the first one.

Two more curiosities that can be gleaned from the images:
- NVIDIA X Server Settings and BOINC Manager are reversely classifying both GPUs: GPU #0 for NVIDIA X Server Settings (GTX 950) is Device #1 in BOINC Manager, and vice versa.
GTX 950 is installed at PCIE slot #0, the nearest one to the CPU, and it is the device delivering video to the connected monitor.
- Mentioned DCF issue can be appreciated also: ACEMD3 WU 10% remaining was calculated to last two (2) days, while completed 90% fraction of it had been processed in about one only hour...

I suspect that graphics card with more than 2 GB internal RAM is a requirement (?)

I think it's more likely to be the /bin/bash which actually needs to write temporary files. I've now managed (with some difficulty) to separate the 569 lines of actual script from the 90 MB of payload. There's

export TMP_BACKUP="$TMP"
export TMP=$PREFIX/install_tmp

but no sign of the TMPDIR mentioned in https://linux.die.net/man/1/bash. More than that is above my pay-grade, I'm afraid.

You can run the script in your home directory if you want to know more about it. (Like the End User License Agreement, the Notice of Third Party Software Licenses, and the Export; Cryptography Notice!)

wrapper starts the script successfully,
miniconda folder is installed silently in /www.gpugrid.net/ directory,
conda install starts the setup of the environment, unpacks files then initiates processes to compile the task.

The environment setup is the step I am thinking causes the error.



Directories and Files of Interest:

/miniconda/etc/profile.d/ directory,
/miniconda/lib/python3.7/<_sysconfigdata_> files

These files and directories seem to contain Gpugrid environment information, which might be of interest. (probably more which I haven't found yet)


I am sure the team at Gpugrid are already 10 steps ahead of us and all issues well in hand!

Curious observation. I've got a Python task running, with all the usual values - 3,000 GFLOPS size, progress rate 14.760% per hour - but BOINC is estimating over 13 days until completion. Normally it's the TONI tasks which are messed up by DCF = 87.8106.

The event log was odd, too:

Returning to graphics card RAM size and Python tasks:
My Host #567828 and Host #557889, both running acemd3 tasks fine on their 2GB RAM graphics cards, haven't been able to process none of the Python tasks that they have received.
I gave up, and set their GPUGrid preferences for them not to ask for Python tasks any more.
This way, they won't be delaying these tasks uselessly.

Returning to graphics card RAM size and Python tasks:
My Host #567828 and Host #557889, both running acemd3 tasks fine on their 2GB RAM graphics cards, haven't been able to process none of the Python tasks that they have received.
I gave up, and set their GPUGrid preferences for them not to ask for Python tasks any more.
This way, they won't be delaying these tasks uselessly.

I happened to receive my first beta task today on a 750Ti (2GB VRAM). It started okay and the task progress bar advanced normally just to fail after ~800 sec. What I have seen so far on other hosts was that either the task seemed to fail immediately within seconds or it finished successfully, so it's strange that it did compute for quite some time. Saw that some of you reported >2GB VRAM used for the task so that could be an issue here as well, but I can't find in the stderr file what might have caused this error. Task 26520658

What is strange for my host is that normally for the ACEMD tasks that CPU time = runtime, but here CPU time was well below it.

I happened to receive my first beta task today on a 750Ti (2GB VRAM). It started okay and the task progress bar advanced normally just to fail after ~800 sec. What I have seen so far on other hosts was that either the task seemed to fail immediately within seconds or it finished successfully, so it's strange that it did compute for quite some time. Saw that some of you reported >2GB VRAM used for the task so that could be an issue here as well, but I can't find in the stderr file what might have caused this error. Task 26520658

What is strange for my host is that normally for the ACEMD tasks that CPU time = runtime, but here CPU time was well below it.

Traceback (most recent call last):
File "run.py", line 50, in <module>
assert os.path.exists('output.coor')
AssertionError
15:13:02 (13979): ./gpugridpy/bin/python exited; CPU time 31.339888

Thanks for the pointer Ian&Steve. Seems like this issue prevails. Just got another beta task that failed and gave me the same error message.

I just read through this whole thread, and I am still not clear what changes I need to make to fix the permissions error. FWIW, I have the regular BOINC installation, not a service installation.

Which file do I need to edit? What changed do I make to that file?

TIA!
____________
Reno, NV
Team: SETI.USA

I just read through this whole thread, and I am still not clear what changes I need to make to fix the permissions error. FWIW, I have the regular BOINC installation, not a service installation.

Which file do I need to edit? What changed do I make to that file?

TIA!

I just read through this whole thread, and I am still not clear what changes I need to make to fix the permissions error. FWIW, I have the regular BOINC installation, not a service installation.

Which file do I need to edit? What changed do I make to that file?

TIA!

Apply this modification:

Extracted from post by ServicEnginIC
(https://www.gpugrid.net/forum_thread.php?id=5204&nowrap=true#55988

I executed the stated command: sudo systemctl edit boinc-client.service And I added to the file the suggested lines: [Service] PrivateTmp=true

A reboot is recommended after applying this change. (Or at the least, restart the boinc-client service)

It would help if you identified exactly which version you have now installed (by version number, not some relative generality like "the latest"), and who released it. Berkeley has not released a new version for Linux, and so far as I know every Linux distribution or repository version available is designed to install as a service. I find the newest versions publicly available are in Gianfranco Costamagna's (LocutusOfBorg) PPA. That's where my problem originated.

I was running 7.9.3, which is what was in the default repository for mint. It had the other problem, with only 20 credits per task.

Now I am running the version from ppa:costamagnagianfranco/boinc , which is version 7.16.14. This version had the permissions problem. The PrivateTmp=true change seems to have fixed the issue on the machines that have received work so far.

Interesting to learn that all linux installations do it as a service install. I thought that caused a problem with GPU crunching? Maybe that is only for windows? Or maybe I am thinking of something else? In any case, good to know.
____________
Reno, NV
Team: SETI.USA

I've never seen a Windows installation that WASN'T a service install.

but for Linux, I prefer the standard non-service application, "install" is too strong a word for this, nothing is installed to the OS. that way I can keep all aspects of BOINC confined to a single folder in my Home folder, and moving or backing up the entire instance of BOINC is as easy as zipping the whole folder and copying it off to external media. very easy to preserve BOINC in cases where you have to maybe reinstall the OS, or simply want to move the BOINC stats between physical computers without a lot of headache. and upgrading the BOINC client or manager is as easy as replacing the core executables. It just works.
____________

Interesting to learn that all linux installations do it as a service install. I thought that caused a problem with GPU crunching?

Interesting to learn that all linux installations do it as a service install. I thought that caused a problem with GPU crunching?

Yes, your statement is right. The "GPUs are not available to services" is a specific, Windows only, driver security matter. Mac and Linux have different ways of handling it.

Under Windows, the installer gives you the choice of how and where to install BOINC. There are defaults, which are designed to 'just work', or you can enter the 'advanced' page and choose your own. Any personal choice will be remembered and offered as the default the next time round.

GPUs and service installs have been incompatible since the driver model was changed in (I think) Windows Vista. Certainly Windows XP could use GPUs in service mode, Windows 7 and later couldn't.

You have a service install if BOINC is listed in the 'services' applet linked from "Control Panel\All Control Panel Items\Administrative Tools" - that location generated from Windows 7: 8 or 10 might be different.

A service install will start running at machine startup: a non-service install will start running at user login. If you machine waits for a password, that might mean 'never'. A service install will never show a Manager icon in the system tray, unles you start the manager manually. A non-service install will try to show a manager icon in the system tray, but Windows will try to hide it. Windows 10 may refuse to show it at all.

ah, that might be the difference then. the past installs were all on windows 7 and I never did anything special or advanced for the install of BOINC. But since they were just crunchers I never bothered having a login prompt, and just let them login automatically at startup. guess that's why I never noticed any issues with GPU crunching this way. To me, if it's running automatically in any way without me actually manually executing the file, it's being run as some sort of service.
____________

When a user logs on,

I increased the FLOPS estimate to 5e15 (was 3e12) and disk usage limit to 10 GB.

Also: spaces in directory names may well create problems. But why spaces? that's inviting problems.

Also: spaces in directory names may well create problems. But why spaces? that's inviting problems.

Sadly, you've missed that boat. Long file names and directory names, including spaces in either, have been de rigeur in Windows since 1995!

The accepted solution is to enclose affected names "in quotation marks".

Been getting "1 (0x1) Unknown error number" after a couple of minutes on a few of my machines (one of which also had valid ones a few days ago). Is this something on my end?

@azmodes, all of my python tasks have failed with the same error. Resetting the project did not make a difference.

@azmodes, all of my python tasks have failed with the same error. Resetting the project did not make a difference.

application ./gpugridpy/bin/python missing

: wrapper: running ./gpugridpy/bin/python (run.py)

I did have one successful task today, but it might have been from the old set.

https://www.gpugrid.net/result.php?resultid=32364018

but all the others have been the same failure, about missing python. must be an error in the task creation, likely to be fixed by the admins when they are aware of it.
____________

I had a long series of failures on 537311 earlier today, while I was out. Then saw that I had two waiting to run on 508381 - looked through the task specs, but couldn't see anything odd. And behold, they're now running normally (one approaching 50%). First-run tasks, created around 16:20 UTC today, so maybe they've found the fault and sent out a new batch.

I've also got a recent new task waiting to start on 537311, so we'll see how that goes.

Edit - the new task on 537311 has started properly, as well. We might just be out of this particular thicket, even if not yet fully out of the woods.

The first one finished successfully, but its replacement has a different early failure:

interesting error message on this one too. looks like it ran to term, but file size too big to upload

upload failure: <file_xfer_error>
<file_name>1k4g219000-RAIMIS_NNPMM-0-1-RND3269_1_0</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

Had one finish and validate.
https://www.gpugrid.net/result.php?resultid=32368128

Some things I noticed about it while it was running.

Ran at 99% GPU use
Started with a Time Remaining of 189 days
Used under 400mb of GPU ram

I'll be removing this for the time being. Been getting a lot of tasks that error out after hours, lots of wasted computing time.

interesting error message on this one too. looks like it ran to term, but file size too big to upload

upload failure: <file_xfer_error>
<file_name>1k4g219000-RAIMIS_NNPMM-0-1-RND3269_1_0</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

http://gpugrid.net/result.php?resultid=32369678

interesting error message on this one too. looks like it ran to term, but file size too big to upload

upload failure: <file_xfer_error>
<file_name>1k4g219000-RAIMIS_NNPMM-0-1-RND3269_1_0</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

http://gpugrid.net/result.php?resultid=32369678

I've had quite a few of these errors. From boinc messages:

52988: 29-Dec-2020 11:24:44 (low) [GPUGRID] Output file 2za0216000-RAIMIS_NNPMM-0-1-RND5553_3_0 for task 2za0216000-RAIMIS_NNPMM-0-1-RND5553_3 exceeds size limit.
52989: 29-Dec-2020 11:24:44 (low) [GPUGRID] File size: 158101319.000000 bytes. Limit: 100000000.000000 bytes

interesting error message on this one too. looks like it ran to term, but file size too big to upload

upload failure: <file_xfer_error>
<file_name>1k4g219000-RAIMIS_NNPMM-0-1-RND3269_1_0</file_name>
<error_code>-131 (file size too big)</error_code>
</file_xfer_error>

http://gpugrid.net/result.php?resultid=32369678

I've had quite a few of these errors. From boinc messages:

52988: 29-Dec-2020 11:24:44 (low) [GPUGRID] Output file 2za0216000-RAIMIS_NNPMM-0-1-RND5553_3_0 for task 2za0216000-RAIMIS_NNPMM-0-1-RND5553_3 exceeds size limit.
52989: 29-Dec-2020 11:24:44 (low) [GPUGRID] File size: 158101319.000000 bytes. Limit: 100000000.000000 bytes

I went through the event log to look for more of the "output file exceeds size limit" errors. It looks like increasing the file upload size limit to 160000000 bytes or more would eliminate this problem.

Is it possible to have the app clean up after itself in the /tmp folder? Each task leaves multiple files. Keep in mind that some people use ram for their /tmp folder, so zombie files are wasting ram

gpugrid files in my /tmp folder

Returning to graphics card RAM size and Python tasks:
My Host #567828 and Host #557889, both running acemd3 tasks fine on their 2GB RAM graphics cards, haven't been able to process none of the Python tasks that they have received.

yeah i noticed the baseline behavior changed from the last round of Python to this latest round.

VRAM use dropped from about 2GB to about 200MB.
PCIe use returned to the same as the MDAD tasks (using about as much as PCIe3.0 x4)
____________

This week (february 22nd) I happened to catch one RAIMIS Python mt CPU task and an ADRIA ACEMD3 GPU task that ran concurrently.
They ran at this 4 core CPU system.
I have it configured to run CPU tasks at two CPU cores, leaving one more core for feeding GPU tasks, and another free core for smoothly attending OS needs.
Ok, RAIMIS Python mt CPU task "kidnapped" for nearly one full day (85640 seconds) both CPU cores, preventing for executing apps from other projects.
Anyway, It fits what could be expected for a multi thread CPU task.
But to my surprise, it also slowed down notoriously the execution of the concurrent GPU task, making it to miss its usually awarded full bonus.
GTX 1660 Ti GPU at this system takes usually 80 to 83 thousand seconds to process ADRIA ACEMD3 GPU tasks, while task that ran concurrently with the RAIMIS Python mt CPU task took more than 96 thousand seconds to process.

I've made that observation back in 2017 on my 4th generation CPUs. I'm avare that this haven't changed much throughout the years, so I basically don't run CPU tasks and GPU tasks simultaneously on the same PC, except when I need that much heat output in the winter. Lately I rather reanimate a few older GPUs, instead of crunching on CPUs.