When can we expect to start getting these new tasks?

When can we expect to start getting these new tasks?

I only get "No tasks sent".

Test applications are allowed and i have compute capability 8.6 with 12GB of GPU Mem running Ubuntu.

Steve,

there is an issue with this application, that will only be apparent for multi-GPU systems.

the application seems to be hard coded in some way to always use GPU0, or the BOINC device assignment is somehow not being correctly communicated to the app.

this results in all tasks running on the same GPU when they should be split up to different GPUs. due to the high VRAM use, this fills the VRAM on most GPUs and causes errors.

see here:

GLaDOS:~$ nvidia-smi
Fri Jan 12 10:05:59 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA TITAN V On | 00000000:21:00.0 On | N/A |
| 80% 55C P2 88W / 150W | 9453MiB / 12288MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA TITAN V On | 00000000:22:00.0 Off | N/A |
| 80% 34C P2 36W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA TITAN V On | 00000000:42:00.0 Off | N/A |
| 80% 42C P2 39W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA TITAN V On | 00000000:61:00.0 Off | N/A |
| 80% 35C P2 36W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1612 G /usr/lib/xorg/Xorg 94MiB |
| 0 N/A N/A 1961 C+G ...libexec/gnome-remote-desktop-daemon 311MiB |
| 0 N/A N/A 2000 G /usr/bin/gnome-shell 67MiB |
| 0 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 0 N/A N/A 223543 M+C python 4490MiB |
| 0 N/A N/A 223769 M+C python 4462MiB |
| 1 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 1 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 2 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 2 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 3 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 3 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
+---------------------------------------------------------------------------------------+

also, could you please add explicit QChem for GPU selections in the project preferences page? currently it is only possible to get this app if you have ALL apps selected + test apps. I want to exclude some apps but still get this one.
____________

just a hunch but I think the problem is with your export command in the run.sh

you have:

just a hunch but I think the problem is with your export command in the run.sh

you have:

export CUDA_VISIBLE_DEVICES=$CUDA_DEVICE

which if I'm reading it right, will set all visible devices to just one GPU. this will have a bad impact for any other tasks running in the BOINC environment i think.

normally on my 4x GPU system, I have CUDA_VISIBLE_DEVICES=0,1,2,3, and if you override that to just the single CUDA device it seems to shuffle all tasks there instead.

yup. I just ran 4 tasks on the same 4-GPU system and each one went to a different GPU as it should.

I see in the stderr that the device was selected properly.
____________

also, does this app make much use of FP64? I'm noticing very fast runtimes on a Titan V, even faster than something like a RTX 3090. the titan V is slower in FP32, but like 14x faster in FP64.

it's hard to follow the code, but I did see that you use cupy a lot, and maybe something in cupy is able to accelerate the Titan V in some way.

or maybe Tensor core difference? does this QChem app use the tensor cores?
____________

Awesome, thanks for that info.

Looking forward to you re-releasing all the tasks you had to pull back earlier :)
____________

+1

+1
____________

Everything working as expected at my hosts.
Well done!
👍️

Steve, so far the first few tasks are completing and being validated for me on single and multi-GPU systems.

My host is an R9-3900X, RTX 3070-Ti running ubuntu 20.04.06 LTS but it doesn't receive Quantum chemistry work units. I selected it in the preferences, test work and "ok to send work of other subprojects". Did I miss anything?

My host is an R9-3900X, RTX 3070-Ti running ubuntu 20.04.06 LTS but it doesn't receive Quantum chemistry work units. I selected it in the preferences, test work and "ok to send work of other subprojects". Did I miss anything?

Drago,
You also need to check the "Run test applications?" box.

Sending out work for this app today. The work units take an hour (very approximately). They should be using different GPUs on multigpu systems. Please let me know if you see anything not working as you would normally expect

Yeah! I got all boxes checked but I still don't get work. Maybe it is a problem with the driver? I have version 470 installed which worked fine for me so far...

I made a couple tests with these new PYSCFbeta tasks.
I tested to stop two of them, and they restarted without erroring. This is good... but both of them got reset their execution times and restarted from the beginning. This is not so good...
And the tests were made at a blend double GPU system (GTX 1660 Ti + GTX 1650). Conversely to ACEMD tasks, both tasks were restarted on the different GPU model than they started, and they did not crash. This is good!

Also, I've noticed a considerable reduction in power draw (about halved) comparing to ACEMD tasks.
GPU power draw at GTX 1660 Ti GPU with PYSCFbeta tasks is half than I'm familiar to see with ACEMD tasks.
And the same happens to GTX 1650 GPU.
Consequently, although 100% GPU usage is shown, working temperatures are much lower...

it's inexplicable since it is the exact same configuration as a system that is receiving tasks just fine.

Ok thanks for this information. There must be something unexpected going on with the scheduler.

I have the same problem too: no tasks sent!

Here is a tale of 2 computers, one that was getting units, and the other was not.

https://www.gpugrid.net/hosts_user.php?userid=19626

They both have the same GPUGRID preferences.

another observation is keep an eye on your CPU use.

these look to be another mt+cuda setup for which BOINC is not prepared to handle, much like the PythonGPU work. i saw upwards of 30 threads utilized per task. but it wasn't sustained, it would come in bursts.

on average reported cpu_time and runtime was about 4x actual (15min actual would be reported as about an hour runtime)
____________

i think if you get a discrete check box selection in the project preferences for QChem on GPU, that will solve the issues of requesting work for this project.
____________

Thank you to whoever got the discrete checkbox implemeted in the settings :). this should make getting work less trivial.
____________

Here is a tale of 2 computers, one that was getting units, and the other was not.

https://www.gpugrid.net/hosts_user.php?userid=19626

They both have the same GPUGRID preferences.

Yes, thanks! this works much better now.

more observed behavior. this batch seems to use less VRAM, and also more limited in CPU use. it's pretty much stuck to just 1 thread per process now. not sure if it's a consequence of the CPU limiting or some other change that limited the VRAM use, but these tasks run a bit slower than the last batch.

if it's bottlenecked by the CPU limiting, maybe there's a middleground? like letting it use up to 4 cores?

the lasst batch 2 days ago, on a Titan V I was running 2x in about 15mins total (7.5 mins per task). this new batch was doing about 25mins for two tasks (12.5mins per task). since VRAM use has gone down, I'm doing 3x in about 25-30mins (8.5-10mins per task). and I'm experimenting with 4x now as well. but the old tasks were undoubtedly faster for some reason.
____________

oh ok, that explains it!
____________

The most recent WUs are just twice the size of the previous test set. There Are 100 molecules in each WU now, previously there were 50.

Steve,

Can you see if you can lift the task download limits? currently it looks like each schedule request will only send one task. instead of a few at a time.

and with multiple computers at the same location, coupled with the DoS protection on your network reventing multiple requests from the same IP, i get sched request failures pretty often, which is limiting how many tasks I can download and not allowing all the GPUs to work.

i know you probably can't do anything about the network DoS protections, but can you allow multiple tasks to download in a single request?
____________

I made fresh Linux Mint installation and it is OK for me now. Now I can dowload new WU.

The app will not work on GPUs with compute capability less than 6.0. It should not be sending them to these cards but I think at the moment this functionality is not working properly.

The app will not work on GPUs with compute capability less than 6.0. It should not be sending them to these cards but I think at the moment this functionality is not working properly.

WUs are being sent to GPUs like GTX 960 (cc=5.2, 2 GB VRAM) and they fail. E.g.,
https://www.gpugrid.net/show_host_detail.php?hostid=550055
https://developer.nvidia.com/cuda-gpus

When you send out WUs with 0.991C + 1NV BOINC does not assign a CPU core to that task. You should designate them 1C.
I've been changing my Use At Most N-2 CPUs to accommodate these tasks. If not they slow down significantly.

That GTX 960 I pointed out also has BOINC 7.7 installed and may be a Science United member so many failures can be expected. But with 7 errors allowed they'll probably find a qualified cruncher before they die.

The 1,92GB file downloads with only ~19,75KBps.

No chance to get the file within the deadline.

I mentioned this problem multiple times in multiple threads. Seems like nobody cares even if the problem affects multiple users.

Steve,

Can you see if you can lift the task download limits? currently it looks like each schedule request will only send one task. instead of a few at a time.

and with multiple computers at the same location, coupled with the DoS protection on your network reventing multiple requests from the same IP, i get sched request failures pretty often, which is limiting how many tasks I can download and not allowing all the GPUs to work.

i know you probably can't do anything about the network DoS protections, but can you allow multiple tasks to download in a single request?

When you send out WUs with 0.991C + 1NV BOINC does not assign a CPU core to that task. You should designate them 1C.
I've been changing my Use At Most N-2 CPUs to accommodate these tasks. If not they slow down significantly.

That GTX 960 I pointed out also has BOINC 7.7 installed and may be a Science United member so many failures can be expected. But with 7 errors allowed they'll probably find a qualified cruncher before they die.

Hello.

The 1,92GB file downloads with only ~19,75KBps.

Hello.

The 1,92GB file downloads with only ~19,75KBps.

I'm also encountering the issue of slow downloads on several hosts.
It would be nice if the project infrastructure worked a little bit faster on our downloads.
Thank you.

Here is a tale of 2 computers, one that was getting units, and the other was not.

https://www.gpugrid.net/hosts_user.php?userid=19626

They both have the same GPUGRID preferences.

I am getting tasks on both computers, now. So far, all tasks are completing successfully.

Good morning.
Wu Pyscfbeta: Quantum Chemistry Calculations On GPU work well on my 1080TI and 1650TI.
Unfortunately on my GTX 970 with 4 GB VRAM, I receive many WUs but they quikly go in error.
I run with Debian 11 and the Nvidia 470 driver.
Is this material too old?
For the moment I have removed Quantum Chemistry on GPU (BETA) on this machine. Because I quickly arrived a daily maximum and would still want to do other wu gpugrid if there are any.
____________
PC are like air conditioning, they becomes useless when you open Windows (L.T)

In a world without walls and fences, who needs windows and gates?

Good morning.
Wu Pyscfbeta: Quantum Chemistry Calculations On GPU work well on my 1080TI and 1650TI.
Unfortunately on my GTX 970 with 4 GB VRAM, I receive many WUs but they quikly go in error.
I run with Debian 11 and the Nvidia 470 driver.
Is this material too old?
For the moment I have removed Quantum Chemistry on GPU (BETA) on this machine. Because I quickly arrived a daily maximum and would still want to do other wu gpugrid if there are any.

Okay ... the answer was in the first message:

The app will not work on GPUs with compute capability less than 6.0. It should not be sending them to these cards but I think at the moment this functionality is not working properly.

OMG, LOL, I love this and must go abuse it...

PC are like air conditioning, they becomes useless when you open Windows (L.T)

When you send out WUs with 0.991C + 1NV BOINC does not assign a CPU core to that task. You should designate them 1C.
I've been changing my Use At Most N-2 CPUs to accommodate these tasks. If not they slow down significantly.

That GTX 960 I pointed out also has BOINC 7.7 installed and may be a Science United member so many failures can be expected. But with 7 errors allowed they'll probably find a qualified cruncher before they die.

You can always override that with an app_config.xml file in the project folder and assign 1.0 cpu threads to the task.

Here is a tale of 2 computers, one that was getting units, and the other was not.

https://www.gpugrid.net/hosts_user.php?userid=19626

They both have the same GPUGRID preferences.

I am getting tasks on both computers, now. So far, all tasks are completing successfully.

After running these tasks successfully for almost a day on both of my computers, now my BOINC manager, task tab, Remaining (estimated) "time" is telling approximately 24 days to complete on one computer and 62 days on the other, at the task's beginning, and incrementally counts down from there. The task actually completes successfully in a little over an hour. A few hours ago, they were showing the correct times to complete.

Everything else is working fine, but this is definitely unusual. Did anyone else observed this?

When you send out WUs with 0.991C + 1NV BOINC does not assign a CPU core to that task. You should designate them 1C.
I've been changing my Use At Most N-2 CPUs to accommodate these tasks. If not they slow down significantly.

That GTX 960 I pointed out also has BOINC 7.7 installed and may be a Science United member so many failures can be expected. But with 7 errors allowed they'll probably find a qualified cruncher before they die.

You can always override that with an app_config.xml file in the project folder and assign 1.0 cpu threads to the task.

I know I can. What about the many people that leave BOINC on autopilot?
I've seen multiple instances of 5 errors before a WU got to me. It's in Steve's best interest.

Here is a tale of 2 computers, one that was getting units, and the other was not.

https://www.gpugrid.net/hosts_user.php?userid=19626

They both have the same GPUGRID preferences.

I am getting tasks on both computers, now. So far, all tasks are completing successfully.

After running these tasks successfully for almost a day on both of my computers, now my BOINC manager, task tab, Remaining (estimated) "time" is telling approximately 24 days to complete on one computer and 62 days on the other, at the task's beginning, and incrementally counts down from there. The task actually completes successfully in a little over an hour. A few hours ago, they were showing the correct times to complete.

Everything else is working fine, but this is definitely unusual. Did anyone else observed this?

At first I did. But including <fraction_done_exact/> seems to heal that fairly quickly.

<app>
<name>PYSCFbeta</name>
<!-- Quantum chemistry calculations on GPU -->
<plan_class>cuda1121</plan_class>
<gpu_versions>
<cpu_usage>1</cpu_usage>
<gpu_usage>1</gpu_usage>
</gpu_versions>
<fraction_done_exact/>
</app>

nah the multi thread issue has already been fixed. the app only uses a single thread now.
____________

The work-units require a lot of GPU memory.

The work-units require a lot of GPU memory.

How much is "a lot" exactly? I have a pacal card, so it meets the compute capability requirement. But it has only 2gb of VRAM. But without knowing the amount of VRAM required, I am not sure if it will work.

It requires more than 2GB

Hello Steeve.

[quote]

The throughput was very high! Especially considering that it is only on Linux hosts and not Windows.

... Especially considering that it is only on Linux hosts and not Windows. We will be sending some similar batches over the next few weeks.

Still no work for windows 11 operating systems?
I see the occasional task that failed but nothing processed.
It worked well for months and then just stopped before xmas.
All my software is up to date.
I have a dedicated GPU for this project.
Where is the best place to find an update on GPUgrid's software migration?

Tasks completed
134
Tasks failed
55
Credit
User
491,814,968 total, 13,657.85 average
Host
150,562,500 total, 13,650.92 average
Scheduling
Scheduling priority
-0.93
Don't request tasks for CPU
Project has no apps for CPU
NVIDIA GPU task request deferred for
00:03:35
NVIDIA GPU task request deferral interval
00:10:00
Last scheduler reply
2024-01-21 1:55:15 PM

Most of the work released lately has been the Quantum Chemistry tasks. The researcher said that since most educational and research labs run Linux OS', that Windows applications are a second thought.

The only tasks with a Windows app that has appeared somewhat regularly are the acemd tasks.

You will have to try and snag one of those when they show up.

The researcher said that since most educational and research labs run Linux OS', that Windows applications are a second thought.

The lack of current Windows applications has more to do with the type of applications and API's being used currently.

The latest and current sub-projects are all Python based. Python runs much better on Linux compared to Windows since most development is done in Linux to begin with.

Even Microsoft advises that Python application development should be done in Linux rather than Windows.

So - in short - bad times for Windows crunchers. Now and in the future :-(

So - in short - bad times for Windows crunchers. Now and in the future :-(

...But including <fraction_done_exact/> seems to heal that fairly quickly.

Choosing not to release Windows apps is a choice they can take, obviously. And maybe their use cases warrant taking the tradeoff inherent in that.

If there's often large volumes of work to process in a small time (i.e. you'd need something like a supercomputer ideally if it didn't cost that much), then you'd want to design your apps for what BOINC intended to be all along. Meaning you try to get them ported to as many platforms as you possibly can in order to reach maximum compute power. Or you leverage the power of VBox for non-native platforms.

If however the volumes are never going to be that large, where basically any single platform user group can easily provide the necessary compute power, then indeed why bother.

Although it would be nice of them to make that choice public and explicit so all non-Linux users can gracefully detach instead of posting frustrated "why no work" messages along the forums.
Or indeed spend hours trying to help fix Windows apps ;-)

I believe that the issue of "0.991" CPUs or whatever is a byproduct of the BOINC serverside software. from what I've read elsewhere, this value is not intentionally set by the researchers, it is automatically selected by the BOINC server somewhere along the way, and the researchers here have previously commented that they are not aware of any way to override this serverside.

The work-units require a lot of GPU memory.

How much is "a lot" exactly? I have a pacal card, so it meets the compute capability requirement. But it has only 2gb of VRAM. But without knowing the amount of VRAM required, I am not sure if it will work.

The work-units require a lot of GPU memory.

How much is "a lot" exactly? I have a pacal card, so it meets the compute capability requirement. But it has only 2gb of VRAM. But without knowing the amount of VRAM required, I am not sure if it will work.

The highest being used today on my Pascal cards is 795 MB.

Retraction: I'm monitoring with the BoincTasks Js 2.4.2.2 and it has bugs.
I loaded NVITOP and it does use 2 GB VRAM with 100% GPU utilization.

BTW, if anyone wants to try NVITOP here's my notes to install for Ubuntu 22.04:
sudo apt update
sudo apt upgrade -y
sudo apt install python3-pip -y
python3 -m pip install --user pipx
python3 -m pip install --user --upgrade pipx
python3 -m pipx ensurepath
# if requested: sudo apt install python3.8-venv -y
For LM 21.3: sudo apt install python3.10-venv -y
Open a new terminal:
pip3 install --upgrade nvitop
pipx run nvitop --colorful -m full

I'm not seeing any different behavior on my titan Vs. the VRAM use still exceeds 3GB at times. but it's spikey. you have to watch it for a few mins. instantaneous measurements might not catch it.
____________

I am seeing spikes to ~7.6 GB with these. Not long lasting (in the context of the whole work unit) but consistently elevated during that part of the work unit. I want to say that I saw that spike at about 5% complete and then at 95% complete, but that also could be somewhat coincidental versus factual.

I am seeing spikes to ~7.6 GB with these. Not long lasting (in the context of the whole work unit) but consistently elevated during that part of the work unit. I want to say that I saw that spike at about 5% complete and then at 95% complete, but that also could be somewhat coincidental versus factual.

I am seeing spikes to ~7.6 GB with these. Not long lasting (in the context of the whole work unit) but consistently elevated during that part of the work unit. I want to say that I saw that spike at about 5% complete and then at 95% complete, but that also could be somewhat coincidental versus factual.

to add on to this, for everyone's info.

these tasks (and a lot of CUDA applications in general) do not require any set absolute value of VRAM. VRAM will scale to the GPU individually. generally, the more SMs you have, to more VRAM will be used. it's not linear, but there is some portion of the allocated VRAM that scales directly with how many SMs are being used.

to put it simply, different GPUs with different core counts, will have different amounts of VRAM utilization.

so even if one powerful GPU like an RTX 4090 with 100+ SMs on the die might need 7+GB, doesn't mean that something much smaller like a GTX 1070 needs that much. it needs to be evaluated on a case by case basis.

Why do I allways get segmentation fault
on Windows/wsl2/Ubuntu 22.04.3 LTS
12 processors, 28 GB memory, 16GB swap, GPU RTX 4070 Ti Super with 16 GB, driver version 551.23

https://www.gpugrid.net/result.php?resultid=33759912
https://www.gpugrid.net/result.php?resultid=33758940
https://www.gpugrid.net/result.php?resultid=33759139
https://www.gpugrid.net/result.php?resultid=33759328

Why do I allways get segmentation fault
on Windows/wsl2/Ubuntu 22.04.3 LTS
12 processors, 28 GB memory, 16GB swap, GPU RTX 4070 Ti Super with 16 GB, driver version 551.23

https://www.gpugrid.net/result.php?resultid=33759912
https://www.gpugrid.net/result.php?resultid=33758940
https://www.gpugrid.net/result.php?resultid=33759139
https://www.gpugrid.net/result.php?resultid=33759328

Steve,

these TEST units you have out right now. they seem to be using a ton of reserved memory. one process right now is using 30+GB. that seems much higher than usual. and i even have another one reserving 64GB of memory. that's way too high.


____________

Here's one that died on my Ubuntu system which has 32 GB RAM:
https://www.gpugrid.net/result.php?resultid=33764282

i see v3 being deployed now

the memory limiting you're trying isn't working. I'm seeing it spike to near 100%

i see you put export CUPY_GPU_MEMORY_LIMIT=50%

a quick google seems to indicate that you need to put the percentage in quotes. like this - export CUPY_GPU_MEMORY_LIMIT="50%". or additionally you can set a discrete memory amount as the limit. for example, export CUPY_GPU_MEMORY_LIMIT="1073741824" to limit to 1GB.

and the system memory use is still a little high, around 10GB each. EDIT - system memory use still climbed to ~30GB by the end
____________

v4 report.

i see you attempted to add some additional VRAM limiting. but the task is still trying to allocate more VRAM, and instead of using more VRAM, the process gets killed for trying to allocate more than the limit.

https://gpugrid.net/result.php?resultid=33764464
https://gpugrid.net/result.php?resultid=33764469
____________

no problem! glad to see you were monitoring my feedback and making changes.

looking forward to another stable batch tomorrow :) should be similar to previous runs like yesterday right?
____________

looking forward to it :)

____________

I have Task 33765246 running on a RTX 3060 Ti under Linux Mint 21.3

It's running incredibly slowly, and with zero GPU usage. I've found this in stderr.txt:

+ python compute_dft.py
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/dft/libxc.py:771: UserWarning: Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, corresponding to the original definition by Stephens et al. (issue 1480) and the same as the B3LYP functional in Gaussian. To restore the VWN5 definition, you can put the setting "B3LYP_WITH_VWN5 = True" in pyscf_conf.py
warnings.warn('Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, '
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/gpu4pyscf/lib/cutensor.py:174: UserWarning: using cupy as the tensor contraction engine.
warnings.warn(f'using {contract_engine} as the tensor contraction engine.')
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/gto/mole.py:1280: UserWarning: Function mol.dumps drops attribute charge because it is not JSON-serializable
warnings.warn(msg)
Exception:
Fallback to CPU
Exception:
Fallback to CPU

Steve,

this new batch, right off the bat, is loading up the GPU VRAM nearly full again.

edit, that's for a v1 tasks, will check out the v2s
____________

OK. looks like the v2 tasks are back to normal. it was only that v1 task that was using lots of vram

____________

Thanks for the info Steve.

In general, I don't have much problem with using a large amount of VRAM, if that's what you require for your science goals. Personally I just wish to have expectations set so that I can setup my hosts accordingly. If VRAM use is low, I can set my host to run multiple tasks at a time for better efficiency. if VRAM use is high, I'll need to cut it back to only 2 or 1 tasks per GPU, which hurts overall efficiency on my end and requires me to reconfigure some things, but it's fine if that's how they will be. I just prefer to know which way it will be so that I don't leave it in a bad configuration and cause errors.

the bigger problem for me (and maybe many others) was the batch yesterday with VERY high system memory use per task. when system ram filled up it would crash the system, which requires some more manual intervention to get it running again. anyone with multi-GPU would be at risk there. just something to consider.

for overall VRAM use. again you can require whatever you need for your science goals. but you might consider making sure you can at least keep them under 8GB. I'd say many people on GPUGRID these days have a GPU with at least 8GB. all of mine have 12GB. and less with 16+. if you can keep them below 8GB I think you'll be able to maintain a large pool of users rather than dealing with the tasks running out of memory and having to be resent multiple times to land on a host with enough VRAM.
____________

I have Task 33765246 running on a RTX 3060 Ti under Linux Mint 21.3

It's running incredibly slowly, and with zero GPU usage. I've found this in stderr.txt:

+ python compute_dft.py
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/dft/libxc.py:771: UserWarning: Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, corresponding to the original definition by Stephens et al. (issue 1480) and the same as the B3LYP functional in Gaussian. To restore the VWN5 definition, you can put the setting "B3LYP_WITH_VWN5 = True" in pyscf_conf.py
warnings.warn('Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, '
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/gpu4pyscf/lib/cutensor.py:174: UserWarning: using cupy as the tensor contraction engine.
warnings.warn(f'using {contract_engine} as the tensor contraction engine.')
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/gto/mole.py:1280: UserWarning: Function mol.dumps drops attribute charge because it is not JSON-serializable
warnings.warn(msg)
Exception:
Fallback to CPU
Exception:
Fallback to CPU

My CPU fallback task has now completed and validated, in not much longer than is usual for tasks on that host. I assume it was a shortened test task, running on a slower device? I now have just completed what looks like a similar task, with similarly large jumps in progress %age, but much more quickly. Task 33765553

No problem Steve. I definitely understand the beta aspect of this and the need to test things. I’m just giving honest feedback from my POV. Sometimes it’s hard to tell if a radical change in behavior is intended or a sign of some problem or misconfiguration.

Maybe it’s not possible for all the various molecules you want to test, but the size of the previous large batch last week I feel was very appropriate. Moderate VRAM use and consistent size/runtimes. Those worked well with the consumer hardware.

Oh if everyone had A100s with 40-80GB of VRAM life would be nice LOL.
____________

I had an odd work unit come through (and just abandoned). I have not had any issues with these work units so thought I would mention this one specifically.

https://www.gpugrid.net/result.php?resultid=33764946

I think there was a memory error with it but I am not very skilled at reading the results. It hung at ~75% but I let it work for 5 hours (honestly, I just didn't notice that it was hung up...).

When looking at properties of the work unit:
Virtual memory: 56GB
Working Size Set: 3.59GB

I thought this was an odd one so thought I would post.

I had an odd work unit come through (and just abandoned). I have not had any issues with these work units so thought I would mention this one specifically.

https://www.gpugrid.net/result.php?resultid=33764946

I think there was a memory error with it but I am not very skilled at reading the results. It hung at ~75% but I let it work for 5 hours (honestly, I just didn't notice that it was hung up...).

When looking at properties of the work unit:
Virtual memory: 56GB
Working Size Set: 3.59GB

I thought this was an odd one so thought I would post.

Yeah you can see several out of memory errors. Are you running more than one at a time?

I’ve had many like this. And many that seem to just fall back to CPU without any reason and get stuck for a long time. I’ve been aborting them when I notice. But it is troublesome :(

I’m now getting a 3rd type of error across all of my hosts.

“AssertionError”

https://www.gpugrid.net/result.php?resultid=33766654
____________

I've had a few of those too, mainly of the form

File "/hdd/boinc-client/slots/6/lib/python3.11/site-packages/gpu4pyscf/df/grad/rhf.py", line 163, in get_jk
assert k1-k0 <= block_size

150.000 credits for a few 100 seconds? I'm in! ;)
https://www.gpugrid.net/result.php?resultid=33771102
https://www.gpugrid.net/result.php?resultid=33771333
https://www.gpugrid.net/result.php?resultid=33771431
https://www.gpugrid.net/result.php?resultid=33771446
https://www.gpugrid.net/result.php?resultid=33771539

I have Task 33765246 running on a RTX 3060 Ti under Linux Mint 21.3

It's running incredibly slowly, and with zero GPU usage. I've found this in stderr.txt:

+ python compute_dft.py
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/dft/libxc.py:771: UserWarning: Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, corresponding to the original definition by Stephens et al. (issue 1480) and the same as the B3LYP functional in Gaussian. To restore the VWN5 definition, you can put the setting "B3LYP_WITH_VWN5 = True" in pyscf_conf.py
warnings.warn('Since PySCF-2.3, B3LYP (and B3P86) are changed to the VWN-RPA variant, '
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/gpu4pyscf/lib/cutensor.py:174: UserWarning: using cupy as the tensor contraction engine.
warnings.warn(f'using {contract_engine} as the tensor contraction engine.')
/hdd/boinc-client/slots/5/lib/python3.11/site-packages/pyscf/gto/mole.py:1280: UserWarning: Function mol.dumps drops attribute charge because it is not JSON-serializable
warnings.warn(msg)
Exception:
Fallback to CPU
Exception:
Fallback to CPU

I'm getting several of these also. this is a problem too. you can always tell when the task basically stalls with almost no progress.

Did we encounter this one already?

that looks like a driver issue.

but something else I noticed is that these tasks for the most part are having a very high failure rate. 30-50% on most hosts.

there are a few hosts that have few or no errors however, and all of them are hosts with 24-48GB of VRAM. so it seems something like 30-50% of the tasks require more than 12-16GB.

I'm sure the project has a very large error percentage to sort through, as there arent enough 24-48GB GPUs to catch all the resends
____________

The present batch has a far worse failure ratio than the previous one.

that looks like a driver issue.

but something else I noticed is that these tasks for the most part are having a very high failure rate. 30-50% on most hosts.

there are a few hosts that have few or no errors however, and all of them are hosts with 24-48GB of VRAM. so it seems something like 30-50% of the tasks require more than 12-16GB.

I'm sure the project has a very large error percentage to sort through, as there arent enough 24-48GB GPUs to catch all the resends

Last week, I had a 100% success rate. This week, it's a different story. Maybe, it's time to step back and dial it down a bit. You have to work with the resources that you have, not the ones that you wish you had.

Boca,

How much VRAM do you see actually being used on some of these tasks? Mind watching a few? You’ll have to run a watch command to see continuous output of VRAM utilization since the usage isn’t constant. It spikes up and down. I’m just curious how much is actually needed. Most of the tasks I was running I would see spike up to about 8GB. But i assume the tasks that needed more just failed instead so I can’t know how much they are trying to use. Even though these Titan Vs are great DP performers they only have 12GB VRAM. Even most of the 16GB cards like V100 and P100 are seeing very high error rates.

MPS helps. But not enough with this current batch. I was getting good throughput with running 3x tasks at once on the batches last week.
____________

that looks like a driver issue.

tout le monde ne dipose de carte graphique a 5000 euros avec 24 gigas de vram voir plus.vous devriez penser au plus modeste d'entre nous.
mo j'ai une rtx 4060 et une gtx 1650 mais je n'ai que des erreurs par exemple.
je pense que lq plupart des gens qui calcule pour gpugrid et qui attende avec impatience du travail pour leur gpu sont comme moi.

everyone only dipose graphics card has 5000 euros with 24 gigas of vram see more.you should think of the most modest of us.
mo I have a 4060 rtx and a 1650 gtx but I have only errors for example.
I think most people who compute for gpugrid and look forward to work for their gpu are like me.

je pense toujours que le probleme est mon installation systeme alors je reformate et refais une installation propre en espérant que cela fonctionnera correctement. en vain car le probleme vient de vos unités de calcul défaillantes

I still think the problem is my system installation so I reformat and redo a clean installation hoping it will work properly. in vain because the problem comes from your faulty computing units
____________

I've disabled getting new GPUGrid tasks GPUGrid on my host with "small" amount (below 24GB) of GPU memory.
This gigantic memory requirement is ridiculous in my opinion.
This is not a user error, if the workunits can't be changed, then the project should not send these tasks to hosts that have less than ~20GB of GPU memory.
There could be another solution, if the workunit would allocate memory in a less careless way.
I've started a task on my RTX 4090 (it has 24GiB RAM), and I've monitored the memory usage:

Nothing to do at this time for my currently working GPUs with PYSCFbeta tasks.
5 GTX 1650 4GB, 1 GTX 1650 SUPER 4GB, 1 GTX 1660 Ti 6GB.
100% errors with current PYSCFbeta tasks, now I can realize why...
I've disabled Quantum chemistry on GPU (beta) at my project preferences in the wait for a correction, if any.
Conversely, they are performing right with ATMbeta tasks.

I agree it does seem these tasks have a spike in memory usage. I "rented" an RTX A5000 GPU which also has 24 GB memory, and running 1 task at a time, at least the first task completed:
https://www.gpugrid.net/workunit.php?wuid=27678500
I will try a few more

I've disabled Quantum chemistry on GPU (beta) at my project preferences in the wait for a correction, if any.
Conversely, they are performing right with ATMbeta tasks.

I have one machine still taking on GPUGrid tasks.
The others are using their GPUs for the Tour de Primes over at PrimeGrid only.
If there really is a driver issue (see earlier post and answers) with this machine I'd like to know which, as its GPU is running fine on other BOINC projects apart from SRBase. Not being able to run SRBase is related to libc, not the GPU driver.
____________
- - - - - - - - - -
Greetings, Jens

bonjour
existe t'il un moyen de simuler de la vram pour gpu en utilisant la ram ou un ssd sous linux.
cela éviterait les erreurs de calcul.
J'ai augmenter le swap file a 50 gigas comme sous windows mais cela ne fonctionne pas.
Merci

hello
Is there a way to simulate vram for GPU using RAM or SSD under linux.
this would avoid miscalculation.
I increased the swap file to 50 gigas as under windows but it does not work.
Thanks
____________

Boca,

How much VRAM do you see actually being used on some of these tasks? Mind watching a few? You’ll have to run a watch command to see continuous output of VRAM utilization since the usage isn’t constant. It spikes up and down. I’m just curious how much is actually needed. Most of the tasks I was running I would see spike up to about 8GB. But i assume the tasks that needed more just failed instead so I can’t know how much they are trying to use. Even though these Titan Vs are great DP performers they only have 12GB VRAM. Even most of the 16GB cards like V100 and P100 are seeing very high error rates.

MPS helps. But not enough with this current batch. I was getting good throughput with running 3x tasks at once on the batches last week.

thanks. that's kind of what I expected was happening.

and yeah, they must have seen the problems and just abandoned the remainder of this run to reassess how to tweak them.

it seemed like they sweaked the input files to give the assertion error instead of just hanging like the earlier (index numbers below ~1000). the early tasks would hang with the fallback to CPU issue, and after that it changed to the assertion error if it ran out of vram. that was better behavior for the user since a quick failure is better than hanging for hours on end doing nothing. but they were probably getting back a majority of errors as the VRAM requirements grew beyond what most people have for available hardware.
____________

New batch just come through- seeing the same VRAM spikes and patterns.

I'm seeing the same spikes, but so far so good. biggest spike i saw was ~9GB

no errors ...yet.

spoke too soon. did get one failure

https://gpugrid.net/result.php?resultid=33801391
____________

seeing some errors on Titan V (12GB). not a huge amount. but certainly a noteworthy amount. maybe you can correlate these specific WUs and see why these kind (number of atoms or molecules?) might be requesting more VRAM than the ones you tried on your 1080.

most of the ones i've observed running will hover around ~3-4GB constant VRAM use, with spikes to the 8-11GB range.

https://gpugrid.net/result.php?resultid=33802055
https://gpugrid.net/result.php?resultid=33801492
https://gpugrid.net/result.php?resultid=33801447
https://gpugrid.net/result.php?resultid=33801391
https://gpugrid.net/result.php?resultid=33801238
____________

Still seeing a vram spike above 8GB

2024/02/02 08:07:08.774, 71, 100 %, 40 %, 8997 MiB
2024/02/02 08:07:09.774, 71, 100 %, 34 %, 8999 MiB
2024/02/02 08:07:10.775, 71, 22 %, 1 %, 8989 MiB
2024/02/02 08:07:11.775, 70, 96 %, 2 %, 10209 MiB
2024/02/02 08:07:12.775, 71, 98 %, 7 %, 10721 MiB
2024/02/02 08:07:13.775, 71, 93 %, 8 %, 5023 MiB
2024/02/02 08:07:14.775, 72, 96 %, 24 %, 5019 MiB
2024/02/02 08:07:15.776, 72, 100 %, 0 %, 5019 MiB
2024/02/02 08:07:16.776, 72, 100 %, 0 %, 5019 MiB

Seems like credit has gone down from 150K to 15K.

Agreed- it seems that there are fewer spikes and most of them are in the 8-9GB range. A few higher but it seems less frequent? Difficult to quantify an actual difference since the work units can be so different. Is there a difference in VRAM usage or does the actual work unit just happen to need less VRAM?