Hello GPUGRID!

We are deploying a new app "PYSCFbeta: Quantum chemistry calculations on GPU". It is currently in testing/beta stage. It is only on Linux at the moment.

The app performs quantum chemistry calculations. At the moment we are using it specifically for Density Functional Theory calculations: http://en.wikipedia.org/wiki/Density_functional_theory

These types of calculations allow us to accurately compute specific properties of small molecules.


The current test work units have a runtime of the order 1hr (very much dependent on the GPU speed and size of molecule). Each work unit currently contains 1 molecule with ~10 configurations.

The app will not work on GPUs with compute capability less than 6.0. It should not be sending them to these cards but I think at the moment this functionality is not working properly.

The work-units require a lot of GPU memory. It works best if the work-unit is the only thing running on the GPU. If other programs are using significant GPU memory the work-unit might fail.

Looking forward to hearing feedback from you.

Steve

When can we expect to start getting these new tasks?

Now, if you are using Linux and have "run test applications?" selected

When can we expect to start getting these new tasks?

They are being distributed RIGHT NOW.
The first 6 WU have arrived here.

I only get "No tasks sent".

Test applications are allowed and i have compute capability 8.6 with 12GB of GPU Mem running Ubuntu.

Steve,

there is an issue with this application, that will only be apparent for multi-GPU systems.

the application seems to be hard coded in some way to always use GPU0, or the BOINC device assignment is somehow not being correctly communicated to the app.

this results in all tasks running on the same GPU when they should be split up to different GPUs. due to the high VRAM use, this fills the VRAM on most GPUs and causes errors.

see here:

GLaDOS:~$ nvidia-smi
Fri Jan 12 10:05:59 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA TITAN V On | 00000000:21:00.0 On | N/A |
| 80% 55C P2 88W / 150W | 9453MiB / 12288MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA TITAN V On | 00000000:22:00.0 Off | N/A |
| 80% 34C P2 36W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA TITAN V On | 00000000:42:00.0 Off | N/A |
| 80% 42C P2 39W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA TITAN V On | 00000000:61:00.0 Off | N/A |
| 80% 35C P2 36W / 150W | 42MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1612 G /usr/lib/xorg/Xorg 94MiB |
| 0 N/A N/A 1961 C+G ...libexec/gnome-remote-desktop-daemon 311MiB |
| 0 N/A N/A 2000 G /usr/bin/gnome-shell 67MiB |
| 0 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 0 N/A N/A 223543 M+C python 4490MiB |
| 0 N/A N/A 223769 M+C python 4462MiB |
| 1 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 1 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 2 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 2 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
| 3 N/A N/A 1612 G /usr/lib/xorg/Xorg 6MiB |
| 3 N/A N/A 5931 C nvidia-cuda-mps-server 30MiB |
+---------------------------------------------------------------------------------------+

in bold, both processes running on the same GPU.

---

also, could you please add explicit QChem for GPU selections in the project preferences page? currently it is only possible to get this app if you have ALL apps selected + test apps. I want to exclude some apps but still get this one.

---

Ah yes thank you for confirming this! This is an omission in the scripts from my end. My test machine has one GPU so I missed it. This can be fixed thank you.

I will try and get the web interface updated but this will take longer due to my unfamiliarity with it. Thanks

just a hunch but I think the problem is with your export command in the run.sh

you have:

export CUDA_VISIBLE_DEVICES=$CUDA_DEVICE

which if I'm reading it right, will set all visible devices to just one GPU. this will have a bad impact for any other tasks running in the BOINC environment i think.

normally on my 4x GPU system, I have CUDA_VISIBLE_DEVICES=0,1,2,3, and if you override that to just the single CUDA device it seems to shuffle all tasks there instead.

---

just a hunch but I think the problem is with your export command in the run.sh

you have:

export CUDA_VISIBLE_DEVICES=$CUDA_DEVICE

which if I'm reading it right, will set all visible devices to just one GPU. this will have a bad impact for any other tasks running in the BOINC environment i think.

normally on my 4x GPU system, I have CUDA_VISIBLE_DEVICES=0,1,2,3, and if you override that to just the single CUDA device it seems to shuffle all tasks there instead.

I guess this wasnt the problem after all :) I see a new small batch went out and i downloaded some and they are working fine now.

---

just a hunch but I think the problem is with your export command in the run.sh

you have:

export CUDA_VISIBLE_DEVICES=$CUDA_DEVICE

which if I'm reading it right, will set all visible devices to just one GPU. this will have a bad impact for any other tasks running in the BOINC environment i think.

normally on my 4x GPU system, I have CUDA_VISIBLE_DEVICES=0,1,2,3, and if you override that to just the single CUDA device it seems to shuffle all tasks there instead.

I guess this wasnt the problem after all :) I see a new small batch went out and i downloaded some and they are working fine now.

Hello, Can you confirm the latest WUs are getting assigned to different GPUs in the way you would expect?


The line in the script you have mentioned is actually the fix I just did. In the first round I had forgotten to put this line.

When the boinc client runs the app via the wrapper mechanism it specifies the gpu device which we capture in the variable CUDA_DEVICE. The Python CUDA code in our app uses the CUDA_VISIBLE_DEVICES variable to choose the GPU. When it is not set (as in the first round of jobs) it defaults to zero. So all jobs end up on GPU zero. With this fix the WUs will be run on the device specified by the boinc client.

yup. I just ran 4 tasks on the same 4-GPU system and each one went to a different GPU as it should.

I see in the stderr that the device was selected properly.

---

Thanks very much for the help!

also, does this app make much use of FP64? I'm noticing very fast runtimes on a Titan V, even faster than something like a RTX 3090. the titan V is slower in FP32, but like 14x faster in FP64.

it's hard to follow the code, but I did see that you use cupy a lot, and maybe something in cupy is able to accelerate the Titan V in some way.

or maybe Tensor core difference? does this QChem app use the tensor cores?

---

Yes this app does make use of some double precision arithmetic. High precision is needed in QM calculations. The bulk of the crunching is done by Nvidia's cusolver library which I believe uses tensor cores when available.

Awesome, thanks for that info.

Looking forward to you re-releasing all the tasks you had to pull back earlier :)

---

Yes we will restart the large scale test next week!

+1

+1

---