Task 38577356
| Name | wu_c970f1cd-GIANNI_GPROTO7-0-1-RND9442_0 |
| Workunit | 31542893 |
| Created | 24 Sep 2025, 22:27:20 UTC |
| Sent | 24 Sep 2025, 22:27:27 UTC |
| Report deadline | 29 Sep 2025, 22:27:27 UTC |
| Received | 24 Sep 2025, 22:55:17 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Compute error |
| Exit status | 195 (0x000000C3) EXIT_CHILD_FAILED |
| Computer ID | 644258 |
| Run time | 2 min 15 sec |
| CPU time | 1 min 35 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 62,846.44 GFLOPS |
| Application version | LLM: LLMs for chemistry v1.00 (cuda124L) x86_64-pc-linux-gnu |
| Peak working set size | 1.86 GB |
| Peak swap size | 31.20 GB |
| Peak disk usage | 14.82 GB |
Stderr output
<core_client_version>8.3.0</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
2025-09-24 22:32:03 (3365): wrapper (8.1.26018): starting
2025-09-24 22:32:26 (3365): wrapper: running bin/python (bin/conda-unpack)
2025-09-24 22:32:26 (3365): wrapper: created child process 3386
2025-09-24 22:32:27 (3365): bin/python exited; CPU time 0.332603
2025-09-24 22:32:27 (3365): wrapper: running bin/tar (xjvf input.tar.bz2)
2025-09-24 22:32:27 (3365): wrapper: created child process 3387
2025-09-24 22:32:28 (3365): bin/tar exited; CPU time 0.025237
2025-09-24 22:32:28 (3365): wrapper: running bin/bash (run.sh)
2025-09-24 22:32:28 (3365): wrapper: created child process 3389
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/workspace/BOINC/slots/2/bin
+++ dirname /workspace/BOINC/slots/2/bin
++ local full_path_env=/workspace/BOINC/slots/2
+++ basename /workspace/BOINC/slots/2
++ local env_name=2
++ '[' -n '' ']'
++ export CONDA_PREFIX=/workspace/BOINC/slots/2
++ CONDA_PREFIX=/workspace/BOINC/slots/2
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(2) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/workspace/BOINC/slots/2/etc/conda/activate.d
++ '[' -d /workspace/BOINC/slots/2/etc/conda/activate.d ']'
+ export PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/workspace/BOINC/slots/2/tmp
+ TMP=/workspace/BOINC/slots/2/tmp
+ mkdir -p /workspace/BOINC/slots/2/tmp
+ which python
+ pip install main_generation-0.1.0-py3-none-any.whl -v --no-deps
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ export HF_HOME=../.cache
+ HF_HOME=../.cache
+ export VLLM_ASSETS_CACHE=../.cache
+ VLLM_ASSETS_CACHE=../.cache
+ export VLLM_CACHE_ROOT=../.cache
+ VLLM_CACHE_ROOT=../.cache
+ echo RUNNING
+ pythonbinary=/workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc
+ python /workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc --conf conf.yaml
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2500 examples [00:00, 916427.20 examples/s]
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.25it/s]
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.14it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.13it/s]
2025-09-24 22:36:01 (3673): wrapper (8.1.26018): starting
2025-09-24 22:36:01 (3673): wrapper: running bin/bash (run.sh)
2025-09-24 22:36:01 (3673): wrapper: created child process 3675
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/workspace/BOINC/slots/2/bin
+++ dirname /workspace/BOINC/slots/2/bin
++ local full_path_env=/workspace/BOINC/slots/2
+++ basename /workspace/BOINC/slots/2
++ local env_name=2
++ '[' -n '' ']'
++ export CONDA_PREFIX=/workspace/BOINC/slots/2
++ CONDA_PREFIX=/workspace/BOINC/slots/2
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(2) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/workspace/BOINC/slots/2/etc/conda/activate.d
++ '[' -d /workspace/BOINC/slots/2/etc/conda/activate.d ']'
+ export PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/workspace/BOINC/slots/2/tmp
+ TMP=/workspace/BOINC/slots/2/tmp
+ mkdir -p /workspace/BOINC/slots/2/tmp
+ which python
+ pip install main_generation-0.1.0-py3-none-any.whl -v --no-deps
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ export HF_HOME=../.cache
+ HF_HOME=../.cache
+ export VLLM_ASSETS_CACHE=../.cache
+ VLLM_ASSETS_CACHE=../.cache
+ export VLLM_CACHE_ROOT=../.cache
+ VLLM_CACHE_ROOT=../.cache
+ echo RUNNING
+ pythonbinary=/workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc
+ python /workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc --conf conf.yaml
Traceback (most recent call last):
File "wheel_contents/aiengine/main_generation.py", line 2, in <module>
File "wheel_contents/aiengine/model.py", line 1, in <module>
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/vllm/__init__.py", line 10, in <module>
import vllm.env_override # isort:skip # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/vllm/env_override.py", line 4, in <module>
import torch
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/torch/__init__.py", line 2222, in <module>
from torch import quantization as quantization # usort: skip
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/torch/quantization/__init__.py", line 2, in <module>
from .fake_quantize import * # noqa: F403
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/torch/quantization/fake_quantize.py", line 10, in <module>
from torch.ao.quantization.fake_quantize import (
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/torch/ao/quantization/__init__.py", line 12, in <module>
from .pt2e._numeric_debugger import ( # noqa: F401
File "/workspace/BOINC/slots/2/lib/python3.12/site-packages/torch/ao/quantization/pt2e/_numeric_debugger.py", line 7, in <module>
from torch.ao.ns.fx.utils import compute_sqnr
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap_external>", line 1128, in get_code
File "<frozen importlib._bootstrap_external>", line 757, in _compile_bytecode
KeyboardInterrupt
2025-09-24 22:40:12 (3814): wrapper (8.1.26018): starting
2025-09-24 22:40:12 (3814): wrapper: running bin/bash (run.sh)
2025-09-24 22:40:12 (3814): wrapper: created child process 3816
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/workspace/BOINC/slots/2/bin
+++ dirname /workspace/BOINC/slots/2/bin
++ local full_path_env=/workspace/BOINC/slots/2
+++ basename /workspace/BOINC/slots/2
++ local env_name=2
++ '[' -n '' ']'
++ export CONDA_PREFIX=/workspace/BOINC/slots/2
++ CONDA_PREFIX=/workspace/BOINC/slots/2
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(2) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/workspace/BOINC/slots/2/etc/conda/activate.d
++ '[' -d /workspace/BOINC/slots/2/etc/conda/activate.d ']'
+ export PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/workspace/BOINC/slots/2/tmp
+ TMP=/workspace/BOINC/slots/2/tmp
+ mkdir -p /workspace/BOINC/slots/2/tmp
+ which python
+ pip install main_generation-0.1.0-py3-none-any.whl -v --no-deps
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ export HF_HOME=../.cache
+ HF_HOME=../.cache
+ export VLLM_ASSETS_CACHE=../.cache
+ VLLM_ASSETS_CACHE=../.cache
+ export VLLM_CACHE_ROOT=../.cache
+ VLLM_CACHE_ROOT=../.cache
+ echo RUNNING
+ pythonbinary=/workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc
+ python /workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc --conf conf.yaml
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2500 examples [00:00, 905193.37 examples/s]
2025-09-24 22:50:58 (5670): wrapper (8.1.26018): starting
2025-09-24 22:50:58 (5670): wrapper: running bin/bash (run.sh)
2025-09-24 22:50:58 (5670): wrapper: created child process 5672
+ echo 'Setup environment'
+ source bin/activate
++ _conda_pack_activate
++ local _CONDA_SHELL_FLAVOR
++ '[' -n x ']'
++ _CONDA_SHELL_FLAVOR=bash
++ local script_dir
++ case "$_CONDA_SHELL_FLAVOR" in
+++ dirname bin/activate
++ script_dir=bin
+++ cd bin
+++ pwd
++ local full_path_script_dir=/workspace/BOINC/slots/2/bin
+++ dirname /workspace/BOINC/slots/2/bin
++ local full_path_env=/workspace/BOINC/slots/2
+++ basename /workspace/BOINC/slots/2
++ local env_name=2
++ '[' -n '' ']'
++ export CONDA_PREFIX=/workspace/BOINC/slots/2
++ CONDA_PREFIX=/workspace/BOINC/slots/2
++ export _CONDA_PACK_OLD_PS1=
++ _CONDA_PACK_OLD_PS1=
++ PATH=/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
++ PS1='(2) '
++ case "$_CONDA_SHELL_FLAVOR" in
++ hash -r
++ local _script_dir=/workspace/BOINC/slots/2/etc/conda/activate.d
++ '[' -d /workspace/BOINC/slots/2/etc/conda/activate.d ']'
+ export PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ PATH=/workspace/BOINC/slots/2:/workspace/BOINC/slots/2/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.
+ echo 'Create a temporary directory'
+ export TMP=/workspace/BOINC/slots/2/tmp
+ TMP=/workspace/BOINC/slots/2/tmp
+ mkdir -p /workspace/BOINC/slots/2/tmp
+ which python
+ pip install main_generation-0.1.0-py3-none-any.whl -v --no-deps
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ export HF_HOME=../.cache
+ HF_HOME=../.cache
+ export VLLM_ASSETS_CACHE=../.cache
+ VLLM_ASSETS_CACHE=../.cache
+ export VLLM_CACHE_ROOT=../.cache
+ VLLM_CACHE_ROOT=../.cache
+ echo RUNNING
+ pythonbinary=/workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc
+ python /workspace/BOINC/slots/2/lib/python3.12/site-packages/aiengine/main_generation.pyc --conf conf.yaml
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.12it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.17it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.16it/s]
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 2.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.11it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.10it/s]
run.sh: line 26: 5683 Killed python ${pythonbinary} --conf conf.yaml
2025-09-24 22:52:44 (5670): bin/bash exited; CPU time 12.990214
2025-09-24 22:52:44 (5670): app exit status: 0x89
2025-09-24 22:52:44 (5670): called boinc_finish(195)
</stderr_txt>
]]>
©2025 Universitat Pompeu Fabra