Name | e00007a07208-ABOU_rnd_ppod_expand_demos30_2_test2-0-1-RND9876_1 |
Workunit | 27386453 |
Created | 22 Jan 2023, 15:41:39 UTC |
Sent | 22 Jan 2023, 17:07:28 UTC |
Report deadline | 27 Jan 2023, 17:07:28 UTC |
Received | 24 Jan 2023, 6:51:56 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 195 (0x000000C3) EXIT_CHILD_FAILED |
Computer ID | 551125 |
Run time | 1 days 22 hours 16 min 18 sec |
CPU time | 1 days 22 hours 16 min 18 sec |
Validate state | Invalid |
Credit | 0.00 |
Device peak FLOPS | 9,464.95 GFLOPS |
Application version | Python apps for GPU hosts v4.04 (cuda1131) windows_x86_64 |
Peak working set size | 0.00 MB |
Peak swap size | 0.00 MB |
Peak disk usage | 0.00 MB |
<core_client_version>7.20.2</core_client_version> <![CDATA[ <message> (unknown error) - exit code 195 (0xc3)</message> <stderr_txt> 11:10:35 (21548): wrapper (7.9.26016): starting 11:10:35 (21548): wrapper: running .\7za.exe (x pythongpu_windows_x86_64__cuda1131.txz -y) 7-Zip (a) 22.01 (x86) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15 Scanning the drive for archives: 1 file, 1976180228 bytes (1885 MiB) Extracting archive: pythongpu_windows_x86_64__cuda1131.txz -- Path = pythongpu_windows_x86_64__cuda1131.txz Type = xz Physical Size = 1976180228 Method = LZMA2:22 CRC64 Streams = 1523 Blocks = 1523 Cluster Size = 4210688 Everything is Ok Size: 6410311680 Compressed: 1976180228 11:12:04 (21548): .\7za.exe exited; CPU time 79.375000 11:12:04 (21548): wrapper: running C:\Windows\system32\cmd.exe (/C "del pythongpu_windows_x86_64__cuda1131.txz") 11:12:05 (21548): C:\Windows\system32\cmd.exe exited; CPU time 0.000000 11:12:05 (21548): wrapper: running .\7za.exe (x pythongpu_windows_x86_64__cuda1131.tar -y) 7-Zip (a) 22.01 (x86) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15 Scanning the drive for archives: 1 file, 6410311680 bytes (6114 MiB) Extracting archive: pythongpu_windows_x86_64__cuda1131.tar -- Path = pythongpu_windows_x86_64__cuda1131.tar Type = tar Physical Size = 6410311680 Headers Size = 19965952 Code Page = UTF-8 Characteristics = GNU LongName ASCII Everything is Ok Files: 38141 Size: 6380353601 Compressed: 6410311680 11:14:42 (21548): .\7za.exe exited; CPU time 22.453125 11:14:42 (21548): wrapper: running C:\Windows\system32\cmd.exe (/C "del pythongpu_windows_x86_64__cuda1131.tar") 11:14:44 (21548): C:\Windows\system32\cmd.exe exited; CPU time 0.000000 11:14:44 (21548): wrapper: running python.exe (run.py) Starting!! Windows fix!! Define rollouts storage Define scheme Created CWorker with worker_index 0 Created GWorker with worker_index 0 Created UWorker with worker_index 0 Created training scheme. Define learner Created Learner. Detected memory leaks! Dumping objects -> ..\api\boinc_api.cpp(309) : {7032} normal block at 0x000001D03059FF80, 8 bytes long. Data: < V0 > 00 00 56 30 D0 01 00 00 ..\lib\diagnostics_win.cpp(417) : {5805} normal block at 0x000001D0320A1B50, 1080 bytes long. Data: < ' 4 > 94 27 00 00 CD CD CD CD 34 01 00 00 00 00 00 00 ..\zip\boinc_zip.cpp(122) : {551} normal block at 0x000001D0305A0960, 260 bytes long. Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 {537} normal block at 0x000001D0305994C0, 52 bytes long. Data: < r > 01 00 00 00 72 00 CD CD 00 00 00 00 00 00 00 00 {532} normal block at 0x000001D0305993E0, 43 bytes long. Data: < p > 01 00 00 00 70 00 CD CD 00 00 00 00 00 00 00 00 {527} normal block at 0x000001D0305996F0, 44 bytes long. Data: < Y0 > 01 00 00 00 00 00 CD CD 11 97 59 30 D0 01 00 00 {522} normal block at 0x000001D030599370, 44 bytes long. Data: < Y0 > 01 00 00 00 00 00 CD CD 91 93 59 30 D0 01 00 00 Object dump complete. 21:10:08 (26612): wrapper (7.9.26016): starting 21:10:08 (26612): wrapper: running python.exe (run.py) Starting!! Windows fix!! Define rollouts storage Define scheme Created CWorker with worker_index 0 Created GWorker with worker_index 0 Created UWorker with worker_index 0 Created training scheme. Define learner Created Learner. Look for a progress_last_chk file - if exists, adjust target_env_steps Define train loop Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 196, in get_data self.next_batch = self.batches.__next__() StopIteration During handling of the above exception, another exception occurred: Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 312, in _recv_bytes nread, err = ov.GetOverlappedResult(True) BrokenPipeError: [WinError 109] The pipe has been ended During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 139, in main learner.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\learner.py", line 46, in step info = self.update_worker.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 118, in step self.updater.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 259, in step grads = self.local_worker.step(self.decentralized_update_execution) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 178, in step self.get_data() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 211, in get_data self.collector.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 490, in step rollouts = self.local_worker.collect_data(listen_to=["sync"], data_to_cpu=False) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 168, in collect_data train_info = self.collect_train_data(listen_to=listen_to) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 242, in collect_train_data obs2, reward, done2, episode_infos = self.envs_train.step(clip_act) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 67, in step_wait obs, reward, done, info = self.venv.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in step_wait results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in <listcomp> results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 250, in recv buf = self._recv_bytes() File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError EOFError Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 196, in get_data self.next_batch = self.batches.__next__() StopIteration During handling of the above exception, another exception occurred: Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 312, in _recv_bytes nread, err = ov.GetOverlappedResult(True) BrokenPipeError: [WinError 109] The pipe has been ended During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 139, in main learner.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\learner.py", line 46, in step info = self.update_worker.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 118, in step self.updater.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 259, in step grads = self.local_worker.step(self.decentralized_update_execution) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 178, in step self.get_data() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 211, in get_data self.collector.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 490, in step rollouts = self.local_worker.collect_data(listen_to=["sync"], data_to_cpu=False) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 168, in collect_data train_info = self.collect_train_data(listen_to=listen_to) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 242, in collect_train_data obs2, reward, done2, episode_infos = self.envs_train.step(clip_act) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 67, in step_wait obs, reward, done, info = self.venv.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in step_wait results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in <listcomp> results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 250, in recv buf = self._recv_bytes() File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError EOFError 00:52:24 (9048): wrapper (7.9.26016): starting 00:52:24 (9048): wrapper: running python.exe (run.py) Starting!! Windows fix!! Define rollouts storage Define scheme Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 312, in _recv_bytes nread, err = ov.GetOverlappedResult(True) BrokenPipeError: [WinError 109] The pipe has been ended During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 114, in main scheme = Scheme(algo_factory, actor_factory, storage_factory, train_envs_factory) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\scheme.py", line 117, in __init__ self._update_worker = UWorker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 78, in __init__ self.grad_workers = grad_workers_factory(dev, index_worker) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker_set.py", line 132, in grad_worker_set_factory return cls( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker_set.py", line 72, in __init__ super(GWorkerSet, self).__init__( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 69, in __init__ self._local_worker = self._make_worker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 97, in _make_worker w = cls(index_worker=index_worker, **worker_params) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 100, in __init__ self.col_workers = col_workers_factory(dev, initial_weights, index_worker) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker_set.py", line 157, in collection_worker_set_factory return cls( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker_set.py", line 83, in __init__ super(CWorkerSet, self).__init__( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 69, in __init__ self._local_worker = self._make_worker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 97, in _make_worker w = cls(index_worker=index_worker, **worker_params) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 114, in __init__ self.algo = algo_factory(self.device, self.actor, self.envs_train) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\algorithms\on_policy\rnd_ppo.py", line 309, in create_algo_instance return cls(lr=lr, File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\algorithms\on_policy\rnd_ppo.py", line 196, in __init__ obs, _, _, _ = envs.step(clipped_action) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 67, in step_wait obs, reward, done, info = self.venv.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in step_wait results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in <listcomp> results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 250, in recv buf = self._recv_bytes() File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError EOFError Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 312, in _recv_bytes nread, err = ov.GetOverlappedResult(True) BrokenPipeError: [WinError 109] The pipe has been ended During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 114, in main scheme = Scheme(algo_factory, actor_factory, storage_factory, train_envs_factory) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\scheme.py", line 117, in __init__ self._update_worker = UWorker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 78, in __init__ self.grad_workers = grad_workers_factory(dev, index_worker) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker_set.py", line 132, in grad_worker_set_factory return cls( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker_set.py", line 72, in __init__ super(GWorkerSet, self).__init__( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 69, in __init__ self._local_worker = self._make_worker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 97, in _make_worker w = cls(index_worker=index_worker, **worker_params) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 100, in __init__ self.col_workers = col_workers_factory(dev, initial_weights, index_worker) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker_set.py", line 157, in collection_worker_set_factory return cls( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker_set.py", line 83, in __init__ super(CWorkerSet, self).__init__( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 69, in __init__ self._local_worker = self._make_worker( File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\base\worker_set.py", line 97, in _make_worker w = cls(index_worker=index_worker, **worker_params) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 114, in __init__ self.algo = algo_factory(self.device, self.actor, self.envs_train) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\algorithms\on_policy\rnd_ppo.py", line 309, in create_algo_instance return cls(lr=lr, File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\algorithms\on_policy\rnd_ppo.py", line 196, in __init__ obs, _, _, _ = envs.step(clipped_action) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 67, in step_wait obs, reward, done, info = self.venv.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in step_wait results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\subproc_vec_env.py", line 84, in <listcomp> results = [remote.recv() for remote in self.remotes] File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 250, in recv buf = self._recv_bytes() File "Q:\ProgramData\BOINC\slots\2\lib\multiprocessing\connection.py", line 321, in _recv_bytes raise EOFError EOFError Detected memory leaks! Dumping objects -> ..\api\boinc_api.cpp(309) : {6942} normal block at 0x00000222C913FD00, 8 bytes long. Data: < " > 00 00 C0 CA 22 02 00 00 ..\lib\diagnostics_win.cpp(417) : {5715} normal block at 0x00000222CABF1B50, 1080 bytes long. Data: < 7 < > 8C 37 00 00 CD CD CD CD 3C 01 00 00 00 00 00 00 ..\zip\boinc_zip.cpp(122) : {551} normal block at 0x00000222C9140960, 260 bytes long. Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 {537} normal block at 0x00000222C9139770, 52 bytes long. Data: < r > 01 00 00 00 72 00 CD CD 00 00 00 00 00 00 00 00 {532} normal block at 0x00000222C9139690, 43 bytes long. Data: < p > 01 00 00 00 70 00 CD CD 00 00 00 00 00 00 00 00 {527} normal block at 0x00000222C9139150, 44 bytes long. Data: < q " > 01 00 00 00 00 00 CD CD 71 91 13 C9 22 02 00 00 {522} normal block at 0x00000222C9139700, 44 bytes long. Data: < ! " > 01 00 00 00 00 00 CD CD 21 97 13 C9 22 02 00 00 Object dump complete. 17:51:58 (24828): wrapper (7.9.26016): starting 17:51:58 (24828): wrapper: running python.exe (run.py) Starting!! Windows fix!! Define rollouts storage Define scheme Created CWorker with worker_index 0 Created GWorker with worker_index 0 Created UWorker with worker_index 0 Created training scheme. Define learner Created Learner. Detected memory leaks! Dumping objects -> ..\api\boinc_api.cpp(309) : {6942} normal block at 0x000001EA435FFBC0, 8 bytes long. Data: < \C > 00 00 5C 43 EA 01 00 00 ..\lib\diagnostics_win.cpp(417) : {5715} normal block at 0x000001EA43604F40, 1080 bytes long. Data: < f 8 > 14 66 00 00 CD CD CD CD 38 01 00 00 00 00 00 00 ..\zip\boinc_zip.cpp(122) : {551} normal block at 0x000001EA43600960, 260 bytes long. Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 {537} normal block at 0x000001EA435F9170, 52 bytes long. Data: < r > 01 00 00 00 72 00 CD CD 00 00 00 00 00 00 00 00 {532} normal block at 0x000001EA435F9790, 43 bytes long. Data: < p > 01 00 00 00 70 00 CD CD 00 00 00 00 00 00 00 00 {527} normal block at 0x000001EA435F9560, 44 bytes long. Data: < _C > 01 00 00 00 00 00 CD CD 81 95 5F 43 EA 01 00 00 {522} normal block at 0x000001EA435F9410, 44 bytes long. Data: < 1 _C > 01 00 00 00 00 00 CD CD 31 94 5F 43 EA 01 00 00 Object dump complete. 21:57:44 (3236): wrapper (7.9.26016): starting 21:57:44 (3236): wrapper: running python.exe (run.py) Starting!! Windows fix!! Define rollouts storage Define scheme Created CWorker with worker_index 0 Created GWorker with worker_index 0 Created UWorker with worker_index 0 Created training scheme. Define learner Created Learner. Look for a progress_last_chk file - if exists, adjust target_env_steps Define train loop Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 196, in get_data self.next_batch = self.batches.__next__() StopIteration During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 139, in main learner.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\learner.py", line 46, in step info = self.update_worker.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 118, in step self.updater.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 259, in step grads = self.local_worker.step(self.decentralized_update_execution) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 178, in step self.get_data() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 211, in get_data self.collector.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 490, in step rollouts = self.local_worker.collect_data(listen_to=["sync"], data_to_cpu=False) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 168, in collect_data train_info = self.collect_train_data(listen_to=listen_to) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 242, in collect_train_data obs2, reward, done2, episode_infos = self.envs_train.step(clip_act) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 73, in step_wait obs = torch.from_numpy(obs).float().to(self.device) RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\c10\core\impl\alloc_cpu.cpp:81] data. DefaultCPUAllocator: not enough memory: you tried to allocate 3612672 bytes. Traceback (most recent call last): File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 196, in get_data self.next_batch = self.batches.__next__() StopIteration During handling of the above exception, another exception occurred: Traceback (most recent call last): File "run.py", line 489, in <module> main() File "run.py", line 139, in main learner.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\learner.py", line 46, in step info = self.update_worker.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 118, in step self.updater.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\updates\u_worker.py", line 259, in step grads = self.local_worker.step(self.decentralized_update_execution) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 178, in step self.get_data() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 211, in get_data self.collector.step() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\gradients\g_worker.py", line 490, in step rollouts = self.local_worker.collect_data(listen_to=["sync"], data_to_cpu=False) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 168, in collect_data train_info = self.collect_train_data(listen_to=listen_to) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\scheme\collection\c_worker.py", line 242, in collect_train_data obs2, reward, done2, episode_infos = self.envs_train.step(clip_act) File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\openai_baselines_dependencies\vec_envs\vec_env_base.py", line 83, in step return self.step_wait() File "Q:\ProgramData\BOINC\slots\2\lib\site-packages\pytorchrl\agent\env\vector_wrappers.py", line 73, in step_wait obs = torch.from_numpy(obs).float().to(self.device) RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\c10\core\impl\alloc_cpu.cpp:81] data. DefaultCPUAllocator: not enough memory: you tried to allocate 3612672 bytes. 00:50:24 (3236): python.exe exited; CPU time 57368.500000 00:50:24 (3236): app exit status: 0x1 00:50:24 (3236): called boinc_finish(195) 0 bytes in 0 Free Blocks. 272 bytes in 7 Normal Blocks. 2288 bytes in 2 CRT Blocks. 0 bytes in 0 Ignore Blocks. 0 bytes in 0 Client Blocks. Largest number used: 0 bytes. Total allocations: 199935601 bytes. Dumping objects -> {7110} normal block at 0x000001ED6F101170, 48 bytes long. Data: <PSI_SCRATCH=Q:\P> 50 53 49 5F 53 43 52 41 54 43 48 3D 51 3A 5C 50 {7069} normal block at 0x000001ED6F100FB0, 48 bytes long. Data: <HOMEPATH=Q:\Prog> 48 4F 4D 45 50 41 54 48 3D 51 3A 5C 50 72 6F 67 {7058} normal block at 0x000001ED6F100D10, 48 bytes long. Data: <HOME=Q:\ProgramD> 48 4F 4D 45 3D 51 3A 5C 50 72 6F 67 72 61 6D 44 {7047} normal block at 0x000001ED6F100D80, 48 bytes long. Data: <TMP=Q:\ProgramDa> 54 4D 50 3D 51 3A 5C 50 72 6F 67 72 61 6D 44 61 {7036} normal block at 0x000001ED6F101090, 48 bytes long. Data: <TEMP=Q:\ProgramD> 54 45 4D 50 3D 51 3A 5C 50 72 6F 67 72 61 6D 44 {7025} normal block at 0x000001ED6F100BC0, 48 bytes long. Data: <TMPDIR=Q:\Progra> 54 4D 50 44 49 52 3D 51 3A 5C 50 72 6F 67 72 61 ..\api\boinc_api.cpp(309) : {6942} normal block at 0x000001ED6D630620, 8 bytes long. Data: < ^mí > 00 00 5E 6D ED 01 00 00 {5716} normal block at 0x000001ED6D630710, 8 bytes long. Data: < Zcmí > 80 5A 63 6D ED 01 00 00 ..\zip\boinc_zip.cpp(122) : {551} normal block at 0x000001ED6D6318C0, 260 bytes long. Data: < > 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 {538} normal block at 0x000001ED6D62CD50, 32 bytes long. Data: <  bmí 0£bmí > 90 A0 62 6D ED 01 00 00 30 A3 62 6D ED 01 00 00 {537} normal block at 0x000001ED6D62A170, 52 bytes long. Data: < r ÍÍ > 01 00 00 00 72 00 CD CD 00 00 00 00 00 00 00 00 {532} normal block at 0x000001ED6D629DF0, 43 bytes long. Data: < p ÍÍ > 01 00 00 00 70 00 CD CD 00 00 00 00 00 00 00 00 {527} normal block at 0x000001ED6D62A330, 44 bytes long. Data: < ÍÍQ£bmí > 01 00 00 00 00 00 CD CD 51 A3 62 6D ED 01 00 00 {522} normal block at 0x000001ED6D62A090, 44 bytes long. Data: < Íͱ bmí > 01 00 00 00 00 00 CD CD B1 A0 62 6D ED 01 00 00 {512} normal block at 0x000001ED6D634630, 16 bytes long. Data: < bmí > 80 9D 62 6D ED 01 00 00 00 00 00 00 00 00 00 00 {511} normal block at 0x000001ED6D629D80, 40 bytes long. Data: <0Fcmí input.zi> 30 46 63 6D ED 01 00 00 69 6E 70 75 74 2E 7A 69 {504} normal block at 0x000001ED6D6349A0, 16 bytes long. Data: <(;cmí > 28 3B 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {503} normal block at 0x000001ED6D6349F0, 16 bytes long. Data: < ;cmí > 00 3B 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {502} normal block at 0x000001ED6D6341D0, 16 bytes long. Data: <Ø:cmí > D8 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {501} normal block at 0x000001ED6D634770, 16 bytes long. Data: <°:cmí > B0 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {500} normal block at 0x000001ED6D634DB0, 16 bytes long. Data: < :cmí > 88 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {499} normal block at 0x000001ED6D6345E0, 16 bytes long. Data: <`:cmí > 60 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {497} normal block at 0x000001ED6D634590, 16 bytes long. Data: <¨Öamí > A8 D6 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {496} normal block at 0x000001ED6D62C3F0, 32 bytes long. Data: <username=Compsci> 75 73 65 72 6E 61 6D 65 3D 43 6F 6D 70 73 63 69 {495} normal block at 0x000001ED6D634130, 16 bytes long. Data: < Öamí > 80 D6 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {494} normal block at 0x000001ED6D6156C0, 64 bytes long. Data: <PYTHONPATH=.\lib> 50 59 54 48 4F 4E 50 41 54 48 3D 2E 5C 6C 69 62 {493} normal block at 0x000001ED6D634950, 16 bytes long. Data: <XÖamí > 58 D6 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {492} normal block at 0x000001ED6D62CF30, 32 bytes long. Data: <PATH=.\Library\b> 50 41 54 48 3D 2E 5C 4C 69 62 72 61 72 79 5C 62 {491} normal block at 0x000001ED6D634D60, 16 bytes long. Data: <0Öamí > 30 D6 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {490} normal block at 0x000001ED6D634900, 16 bytes long. Data: < Öamí > 08 D6 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {489} normal block at 0x000001ED6D634720, 16 bytes long. Data: <àÕamí > E0 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {488} normal block at 0x000001ED6D6348B0, 16 bytes long. Data: <¸Õamí > B8 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {487} normal block at 0x000001ED6D634540, 16 bytes long. Data: < Õamí > 90 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {486} normal block at 0x000001ED6D634BD0, 16 bytes long. Data: <hÕamí > 68 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {485} normal block at 0x000001ED6D62C5D0, 32 bytes long. Data: <SystemRoot=C:\Wi> 53 79 73 74 65 6D 52 6F 6F 74 3D 43 3A 5C 57 69 {484} normal block at 0x000001ED6D634860, 16 bytes long. Data: <@Õamí > 40 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {483} normal block at 0x000001ED6D62C450, 32 bytes long. Data: <GPU_DEVICE_NUM=0> 47 50 55 5F 44 45 56 49 43 45 5F 4E 55 4D 3D 30 {482} normal block at 0x000001ED6D6344A0, 16 bytes long. Data: < Õamí > 18 D5 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {481} normal block at 0x000001ED6D62CED0, 32 bytes long. Data: <NTHREADS=1 THREA> 4E 54 48 52 45 41 44 53 3D 31 00 54 48 52 45 41 {480} normal block at 0x000001ED6D634D10, 16 bytes long. Data: <ðÔamí > F0 D4 61 6D ED 01 00 00 00 00 00 00 00 00 00 00 {479} normal block at 0x000001ED6D61D4F0, 480 bytes long. Data: < Mcmí ÐÎbmí > 10 4D 63 6D ED 01 00 00 D0 CE 62 6D ED 01 00 00 {478} normal block at 0x000001ED6D634400, 16 bytes long. Data: <@:cmí > 40 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {477} normal block at 0x000001ED6D634AE0, 16 bytes long. Data: < :cmí > 18 3A 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {476} normal block at 0x000001ED6D633F50, 16 bytes long. Data: <ð9cmí > F0 39 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {475} normal block at 0x000001ED6D62A020, 48 bytes long. Data: </C "del pythongp> 2F 43 20 22 64 65 6C 20 70 79 74 68 6F 6E 67 70 {474} normal block at 0x000001ED6D633F00, 16 bytes long. Data: <89cmí > 38 39 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {473} normal block at 0x000001ED6D6340E0, 16 bytes long. Data: < 9cmí > 10 39 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {472} normal block at 0x000001ED6D634B80, 16 bytes long. Data: <è8cmí > E8 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {471} normal block at 0x000001ED6D634090, 16 bytes long. Data: <À8cmí > C0 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {470} normal block at 0x000001ED6D634040, 16 bytes long. Data: < 8cmí > 98 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {469} normal block at 0x000001ED6D6344F0, 16 bytes long. Data: <p8cmí > 70 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {468} normal block at 0x000001ED6D6343B0, 16 bytes long. Data: <P8cmí > 50 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {467} normal block at 0x000001ED6D634B30, 16 bytes long. Data: <(8cmí > 28 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {466} normal block at 0x000001ED6D62CE10, 32 bytes long. Data: <C:\Windows\syste> 43 3A 5C 57 69 6E 64 6F 77 73 5C 73 79 73 74 65 {465} normal block at 0x000001ED6D634360, 16 bytes long. Data: < 8cmí > 00 38 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {464} normal block at 0x000001ED6D629E60, 48 bytes long. Data: <x pythongpu_wind> 78 20 70 79 74 68 6F 6E 67 70 75 5F 77 69 6E 64 {463} normal block at 0x000001ED6D633FA0, 16 bytes long. Data: <H7cmí > 48 37 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {462} normal block at 0x000001ED6D6346D0, 16 bytes long. Data: < 7cmí > 20 37 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {461} normal block at 0x000001ED6D634450, 16 bytes long. Data: <ø6cmí > F8 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {460} normal block at 0x000001ED6D634CC0, 16 bytes long. Data: <Ð6cmí > D0 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {459} normal block at 0x000001ED6D633FF0, 16 bytes long. Data: <¨6cmí > A8 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {458} normal block at 0x000001ED6D634310, 16 bytes long. Data: < 6cmí > 80 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {457} normal block at 0x000001ED6D634C20, 16 bytes long. Data: <`6cmí > 60 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {456} normal block at 0x000001ED6D634C70, 16 bytes long. Data: <86cmí > 38 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {455} normal block at 0x000001ED6D634680, 16 bytes long. Data: < 6cmí > 10 36 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {454} normal block at 0x000001ED6D62A100, 48 bytes long. Data: </C "del pythongp> 2F 43 20 22 64 65 6C 20 70 79 74 68 6F 6E 67 70 {453} normal block at 0x000001ED6D634220, 16 bytes long. Data: <X5cmí > 58 35 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {452} normal block at 0x000001ED6D634180, 16 bytes long. Data: <05cmí > 30 35 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {451} normal block at 0x000001ED6D6342C0, 16 bytes long. Data: < 5cmí > 08 35 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {450} normal block at 0x000001ED6D634270, 16 bytes long. Data: <à4cmí > E0 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {449} normal block at 0x000001ED6D634A90, 16 bytes long. Data: <¸4cmí > B8 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {448} normal block at 0x000001ED6D6347C0, 16 bytes long. Data: < 4cmí > 90 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {447} normal block at 0x000001ED6D633EB0, 16 bytes long. Data: <p4cmí > 70 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {446} normal block at 0x000001ED6D630300, 16 bytes long. Data: <H4cmí > 48 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {445} normal block at 0x000001ED6D62C030, 32 bytes long. Data: <C:\Windows\syste> 43 3A 5C 57 69 6E 64 6F 77 73 5C 73 79 73 74 65 {444} normal block at 0x000001ED6D6300D0, 16 bytes long. Data: < 4cmí > 20 34 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {443} normal block at 0x000001ED6D629FB0, 48 bytes long. Data: <x pythongpu_wind> 78 20 70 79 74 68 6F 6E 67 70 75 5F 77 69 6E 64 {442} normal block at 0x000001ED6D630800, 16 bytes long. Data: <h3cmí > 68 33 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {441} normal block at 0x000001ED6D630670, 16 bytes long. Data: <@3cmí > 40 33 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {440} normal block at 0x000001ED6D6304E0, 16 bytes long. Data: < 3cmí > 18 33 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {439} normal block at 0x000001ED6D630030, 16 bytes long. Data: <ð2cmí > F0 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {438} normal block at 0x000001ED6D630EE0, 16 bytes long. Data: <È2cmí > C8 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {437} normal block at 0x000001ED6D6302B0, 16 bytes long. Data: < 2cmí > A0 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {436} normal block at 0x000001ED6D630E90, 16 bytes long. Data: < 2cmí > 80 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {435} normal block at 0x000001ED6D630E40, 16 bytes long. Data: <X2cmí > 58 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {434} normal block at 0x000001ED6D6309E0, 16 bytes long. Data: <02cmí > 30 32 63 6D ED 01 00 00 00 00 00 00 00 00 00 00 {433} normal block at 0x000001ED6D633230, 2976 bytes long. Data: <à cmí .\7za.ex> E0 09 63 6D ED 01 00 00 2E 5C 37 7A 61 2E 65 78 {72} normal block at 0x000001ED6D61B260, 16 bytes long. Data: < êIúö > 80 EA 49 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {71} normal block at 0x000001ED6D61C600, 16 bytes long. Data: <@éIúö > 40 E9 49 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {70} normal block at 0x000001ED6D61C060, 16 bytes long. Data: <øWFúö > F8 57 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {69} normal block at 0x000001ED6D61C010, 16 bytes long. Data: <ØWFúö > D8 57 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {68} normal block at 0x000001ED6D61BD90, 16 bytes long. Data: <P Fúö > 50 04 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {67} normal block at 0x000001ED6D61C560, 16 bytes long. Data: <0 Fúö > 30 04 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {66} normal block at 0x000001ED6D61BC00, 16 bytes long. Data: <à Fúö > E0 02 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {65} normal block at 0x000001ED6D61BBB0, 16 bytes long. Data: < Fúö > 10 04 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {64} normal block at 0x000001ED6D61C920, 16 bytes long. Data: <p Fúö > 70 04 46 FA F6 7F 00 00 00 00 00 00 00 00 00 00 {63} normal block at 0x000001ED6D61C740, 16 bytes long. Data: < ÀDúö > 18 C0 44 FA F6 7F 00 00 00 00 00 00 00 00 00 00 Object dump complete. </stderr_txt> ]]>
©2025 Universitat Pompeu Fabra