Task 38577281

Name	wu_5fb7b296-GIANNI_GPROTO7-0-1-RND9409_0
Workunit	31542834
Created	24 Sep 2025, 20:50:51 UTC
Sent	24 Sep 2025, 20:51:03 UTC
Report deadline	29 Sep 2025, 20:51:03 UTC
Received	24 Sep 2025, 21:26:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	195 (0x000000C3) EXIT_CHILD_FAILED
Computer ID	643932
Run time	1 min 9 sec
CPU time	31 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	109,735.62 GFLOPS
Application version	LLM: LLMs for chemistry v1.00 (cuda124L) x86_64-pc-linux-gnu
Peak working set size	777.66 MB
Peak swap size	10.09 GB
Peak disk usage	8.13 GB

Stderr output

<core_client_version>8.3.0</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
2025-09-24 21:18:08 (1569244): wrapper (8.1.26018): starting
2025-09-24 21:18:44 (1569244): wrapper: running bin/python (bin/conda-unpack)
2025-09-24 21:18:44 (1569244): wrapper: created child process 1569449
2025-09-24 21:18:45 (1569244): bin/python exited; CPU time 0.620657
2025-09-24 21:18:45 (1569244): wrapper: running bin/tar (xjvf input.tar.bz2)
2025-09-24 21:18:45 (1569244): wrapper: created child process 1569469
2025-09-24 21:18:46 (1569244): bin/tar exited; CPU time 0.018526
2025-09-24 21:18:46 (1569244): wrapper: running bin/bash (run.sh)
2025-09-24 21:18:46 (1569244): wrapper: created child process 1569488
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/var/lib/boinc/slots/43/bin
+++ dirname /var/lib/boinc/slots/43/bin
++ local full_path_env=/var/lib/boinc/slots/43
+++ basename /var/lib/boinc/slots/43
++ local env_name=43
++ '[' -n '' ']'
++ export CONDA_PREFIX=/var/lib/boinc/slots/43
++ CONDA_PREFIX=/var/lib/boinc/slots/43
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/var/lib/boinc/slots/43/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(43) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/var/lib/boinc/slots/43/etc/conda/activate.d
++ '[' -d /var/lib/boinc/slots/43/etc/conda/activate.d ']'
+ export PATH=/var/lib/boinc/slots/43:/var/lib/boinc/slots/43/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/var/lib/boinc/slots/43:/var/lib/boinc/slots/43/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/var/lib/boinc/slots/43/tmp
+ TMP=/var/lib/boinc/slots/43/tmp
+ mkdir -p /var/lib/boinc/slots/43/tmp
+ which python
+ pip install main_generation-0.1.0-py3-none-any.whl -v --no-deps
+ export CUDA_VISIBLE_DEVICES=3
+ CUDA_VISIBLE_DEVICES=3
+ export HF_HOME=../.cache
+ HF_HOME=../.cache
+ export VLLM_ASSETS_CACHE=../.cache
+ VLLM_ASSETS_CACHE=../.cache
+ export VLLM_CACHE_ROOT=../.cache
+ VLLM_CACHE_ROOT=../.cache
+ echo RUNNING
+ pythonbinary=/var/lib/boinc/slots/43/lib/python3.12/site-packages/aiengine/main_generation.pyc
+ python /var/lib/boinc/slots/43/lib/python3.12/site-packages/aiengine/main_generation.pyc --conf conf.yaml

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2500 examples [00:00, 107778.39 examples/s]
/var/lib/boinc/slots/43/lib/python3.12/site-packages/torch/cuda/__init__.py:235: UserWarning: 
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
run.sh: line 26: 1569586 Killed                  python ${pythonbinary} --conf conf.yaml
2025-09-24 21:19:13 (1569244): bin/bash exited; CPU time 34.962305
2025-09-24 21:19:13 (1569244): app exit status: 0x89
2025-09-24 21:19:13 (1569244): called boinc_finish(195)

</stderr_txt>
]]>