Task 33800896

Name	inputs_ace_pch_ms_gc_filt_af05_index_25228_to_25231-SFARR_PYSCF_ace_pch_ms_gc_filt_af05_v2-0-1-RND8952_1
Workunit	27678473
Created	2 Feb 2024, 12:07:47 UTC
Sent	2 Feb 2024, 12:18:08 UTC
Report deadline	7 Feb 2024, 12:18:08 UTC
Received	2 Feb 2024, 12:22:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	195 (0x000000C3) EXIT_CHILD_FAILED
Computer ID	612468
Run time	1 min 46 sec
CPU time	7 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	1,426.39 GFLOPS
Application version	Quantum chemistry calculations on GPU v1.04 (cuda1121) x86_64-pc-linux-gnu
Peak working set size	0.00 MB
Peak swap size	0.00 MB
Peak disk usage	0.00 MB

Stderr output

<core_client_version>7.24.2</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
06:18:24 (1345526): wrapper (7.7.26016): starting
06:19:56 (1345526): wrapper (7.7.26016): starting
06:19:56 (1345526): wrapper: running bin/python (bin/conda-unpack)
06:19:58 (1345526): bin/python exited; CPU time 0.744084
06:19:58 (1345526): wrapper: running bin/tar (xjvf input.tar.bz2)
06:19:59 (1345526): bin/tar exited; CPU time 0.004539
06:19:59 (1345526): wrapper: running bin/bash (run.sh)
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/var/lib/boinc/slots/4/bin
+++ dirname /var/lib/boinc/slots/4/bin
++ local full_path_env=/var/lib/boinc/slots/4
+++ basename /var/lib/boinc/slots/4
++ local env_name=4
++ '[' -n '' ']'
++ export CONDA_PREFIX=/var/lib/boinc/slots/4
++ CONDA_PREFIX=/var/lib/boinc/slots/4
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/var/lib/boinc/slots/4/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(4) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/var/lib/boinc/slots/4/etc/conda/activate.d
++ '[' -d /var/lib/boinc/slots/4/etc/conda/activate.d ']'
+ export PATH=/var/lib/boinc/slots/4:/var/lib/boinc/slots/4/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/var/lib/boinc/slots/4:/var/lib/boinc/slots/4/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/var/lib/boinc/slots/4/tmp
+ TMP=/var/lib/boinc/slots/4/tmp
+ mkdir -p /var/lib/boinc/slots/4/tmp
+ export OMP_NUM_THREADS=1
+ OMP_NUM_THREADS=1
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ export CUPY_CUDA_LIB_PATH=/var/lib/boinc/slots/4/cupy
+ CUPY_CUDA_LIB_PATH=/var/lib/boinc/slots/4/cupy
+ echo 'Running PySCF'
+ python compute_dft.py
/var/lib/boinc/slots/4/lib/python3.11/site-packages/gpu4pyscf/lib/cutensor.py:174: UserWarning: using cupy as the tensor contraction engine.
  warnings.warn(f'using {contract_engine} as the tensor contraction engine.')
Traceback (most recent call last):
  File "/var/lib/boinc/slots/4/compute_dft.py", line 12, in <module>
    from gpu4pyscf.dft import rks as gpu_rks
  File "/var/lib/boinc/slots/4/lib/python3.11/site-packages/gpu4pyscf/__init__.py", line 1, in <module>
    from . import lib, grad, hessian, solvent, scf, dft
  File "/var/lib/boinc/slots/4/lib/python3.11/site-packages/gpu4pyscf/lib/__init__.py", line 19, in <module>
    from gpu4pyscf.lib import cupy_helper
  File "/var/lib/boinc/slots/4/lib/python3.11/site-packages/gpu4pyscf/lib/cupy_helper.py", line 26, in <module>
    from gpu4pyscf.lib.cusolver import eigh, cholesky  #NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/boinc/slots/4/lib/python3.11/site-packages/gpu4pyscf/lib/cusolver.py", line 25, in <module>
    _handle = device.get_cusolver_handle()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/cuda/device.pyx", line 72, in cupy.cuda.device.get_cusolver_handle
  File "cupy/cuda/device.pyx", line 73, in cupy.cuda.device.get_cusolver_handle
  File "cupy/cuda/device.pyx", line 271, in cupy.cuda.device.Device.cusolver_handle.__get__
  File "cupy/cuda/device.pyx", line 246, in cupy.cuda.device.Device._get_handle
  File "cupy_backends/cuda/libs/cusolver.pyx", line 1104, in cupy_backends.cuda.libs.cusolver.create
  File "cupy_backends/cuda/libs/cusolver.pyx", line 1108, in cupy_backends.cuda.libs.cusolver.create
  File "cupy_backends/cuda/libs/cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status
cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_INTERNAL_ERROR
06:20:10 (1345526): bin/bash exited; CPU time 6.632365
06:20:10 (1345526): app exit status: 0x1
06:20:10 (1345526): called boinc_finish(195)

</stderr_txt>
]]>