Server Out of Disk Space

Author	Message
Rion Family Send message Joined: 13 Jan 14 Posts: 21 Credit: 15,415,926,517 RAC: 0 Level Scientific publications	Message 42893 - Posted: 6 Mar 2016, 2:02:26 UTC Noticed Upload failures - starting little over an hour ago Kind Regards, BER 3/5/2016 6:54:29 PM \| GPUGRID \| [error] Error reported by file upload server: can't write file /home/ps3grid/projects/PS3GRID/upload/279/002_640_902x1x48-GERARD_CXCL12_CostaMonomerx_umbrella-0-1-RND9106_0_9: No space left on server 3/5/2016 6:54:29 PM \| GPUGRID \| Temporarily failed upload of 002_640_902x1x48-GERARD_CXCL12_CostaMonomerx_umbrella-0-1-RND9106_0_9: transient upload error 3/5/2016 6:54:29 PM \| GPUGRID \| Backing off 00:03:29 on upload of 002_640_902x1x48-GERARD_CXCL12_CostaMonomerx_umbrella-0-1-RND9106_0_9 3/5/2016 6:57:59 PM \| GPUGRID \| Started upload of 002_640_902x1x48-GERARD_CXCL12_CostaMonomerx_umbrella-0-1-RND9106_0_9 3/5/2016 6:58:01 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/5/2016 6:58:01 PM \| GPUGRID \| Temporarily failed upload of 002_640_902x1x48-GERARD_CXCL12_CostaMonomerx_umbrella-0-1-RND9106_0_9: transient upload error ID: 42893 · Rating: 0 · rate: / Reply Quote

NottiusMaximus Send message Joined: 12 May 12 Posts: 1 Credit: 430,401 RAC: 0 Level Scientific publications	Message 42923 - Posted: 7 Mar 2016, 2:17:55 UTC - in response to Message 42893. Still receiving this message ID: 42923 · Rating: 0 · rate: / Reply Quote

johnnymc Send message Joined: 7 Apr 11 Posts: 6 Credit: 92,079,090 RAC: 0 Level Scientific publications	Message 42926 - Posted: 7 Mar 2016, 4:23:22 UTC I encountered this error first this morning - and throughout the day. 3/6/2016 6:45:01 AM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 8:20:54 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space ID: 42926 · Rating: 0 · rate: / Reply Quote

Aegis Maelstrom Send message Joined: 6 Jan 15 Posts: 5 Credit: 4,633,108 RAC: 0 Level Scientific publications	Message 42927 - Posted: 7 Mar 2016, 7:02:14 UTC The problem lasts for 10 hours already and one cannot return the results. \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space ID: 42927 · Rating: 0 · rate: / Reply Quote

olsen_gg Send message Joined: 19 Jun 13 Posts: 5 Credit: 116,472,742 RAC: 0 Level Scientific publications	Message 42929 - Posted: 7 Mar 2016, 7:57:05 UTC Last modified: 7 Mar 2016, 8:19:06 UTC only three days left to fix this problem: 07.03.2016 08:37:23 \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 07.03.2016 08:37:23 \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space I was just preparing for this team challenge: CRUNCHATHLON GPUGRID [MONTHLY RUN - RACE XIX] Start time: 2016-03-10 18:00 UTC End time: 2016-03-14 18:00 UTC[/url] ID: 42929 · Rating: 0 · rate: / Reply Quote

Gerard Send message Joined: 26 Mar 14 Posts: 101 Credit: 0 RAC: 0 Level Scientific publications	Message 42932 - Posted: 7 Mar 2016, 9:15:27 UTC - in response to Message 42929. We are now trying to fix this issue. Sorry for the inconvinience! ID: 42932 · Rating: 0 · rate: / Reply Quote

Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1639 Credit: 10,159,968,649 RAC: 0 Level Scientific publications	Message 42933 - Posted: 7 Mar 2016, 12:38:27 UTC - in response to Message 42932. We are now trying to fix this issue. Sorry for the inconvinience! That worked brilliantly to clear the backlog - thanks! - but I'm afraid we've filled it up again... ID: 42933 · Rating: 0 · rate: / Reply Quote

JMrkvicka Send message Joined: 25 Dec 09 Posts: 8 Credit: 246,817,898 RAC: 0 Level Scientific publications	Message 42935 - Posted: 7 Mar 2016, 12:53:18 UTC I am seeing this in my event log in addition to the out of space messages: 3/7/2016 7:44:07 AM \| GPUGRID \| Sending scheduler request: To fetch work. 3/7/2016 7:44:07 AM \| GPUGRID \| Requesting new tasks for NVIDIA GPU 3/7/2016 7:44:09 AM \| GPUGRID \| Scheduler request completed: got 0 new tasks 3/7/2016 7:44:09 AM \| GPUGRID \| No tasks sent 3/7/2016 7:44:09 AM \| GPUGRID \| This computer has reached a limit on tasks in progress ID: 42935 · Rating: 0 · rate: / Reply Quote

WPrion Send message Joined: 30 Apr 13 Posts: 109 Credit: 3,977,737,860 RAC: 575,436 Level Scientific publications	Message 42936 - Posted: 7 Mar 2016, 12:55:25 UTC Last modified: 7 Mar 2016, 13:42:40 UTC Same problems since yesterday. On the bright side it's great news for Asteroids and SETI! ;-) ETA: 3/6/2016 11:51:40 AM \| GPUGRID \| Sending scheduler request: Requested by project. 3/6/2016 11:51:40 AM \| GPUGRID \| Requesting new tasks for NVIDIA GPU 3/6/2016 11:51:42 AM \| GPUGRID \| Scheduler request completed: got 0 new tasks 3/6/2016 11:51:42 AM \| GPUGRID \| No tasks sent 3/6/2016 11:51:42 AM \| GPUGRID \| This computer has reached a limit on tasks in progress 3/6/2016 12:09:06 PM \| GPUGRID \| Computation for task e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0 finished 3/6/2016 12:09:19 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_0 3/6/2016 12:09:19 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_1 3/6/2016 12:09:20 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 12:09:20 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 12:09:20 PM \| GPUGRID \| Temporarily failed upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_0: transient upload error 3/6/2016 12:09:20 PM \| GPUGRID \| Backing off 00:02:33 on upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_0 3/6/2016 12:09:20 PM \| GPUGRID \| Temporarily failed upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_1: transient upload error 3/6/2016 12:09:20 PM \| GPUGRID \| Backing off 00:02:09 on upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_1 3/6/2016 12:09:20 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_2 3/6/2016 12:09:20 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_3 3/6/2016 12:09:21 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 12:09:21 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 12:09:21 PM \| GPUGRID \| Temporarily failed upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_2: transient upload error 3/6/2016 12:09:21 PM \| GPUGRID \| Backing off 00:03:27 on upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_2 3/6/2016 12:09:21 PM \| GPUGRID \| Temporarily failed upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_3: transient upload error 3/6/2016 12:09:21 PM \| GPUGRID \| Backing off 00:03:28 on upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_3 3/6/2016 12:09:21 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_7 3/6/2016 12:09:21 PM \| GPUGRID \| Started upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_9 3/6/2016 12:09:23 PM \| GPUGRID \| [error] Error reported by file upload server: Server is out of disk space 3/6/2016 12:09:23 PM \| GPUGRID \| Finished upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_7 3/6/2016 12:09:23 PM \| GPUGRID \| Temporarily failed upload of e1s9_1-GERARD_A2AR_luf6632_a_1Dx1-1-2-RND8793_0_9: transient upload error 3/6/2016 12:09:23 PM \| GPUGRID \| Backing off 00:02:24 on up ID: 42936 · Rating: 0 · rate: / Reply Quote

olsen_gg Send message Joined: 19 Jun 13 Posts: 5 Credit: 116,472,742 RAC: 0 Level Scientific publications	Message 42939 - Posted: 7 Mar 2016, 14:05:24 UTC No changes yet. Continuously same stupid message "out of disk space" :-( O.K., we try to be patient. ID: 42939 · Rating: 0 · rate: / Reply Quote

WPrion Send message Joined: 30 Apr 13 Posts: 109 Credit: 3,977,737,860 RAC: 575,436 Level Scientific publications	Message 42941 - Posted: 7 Mar 2016, 14:29:10 UTC Last modified: 7 Mar 2016, 14:45:14 UTC I now have an active upload! Yeah!!! I really didn't want to dump a day's work to get new tasks. Thanks!! W EDIT - it's back in retry mode. The largest file reports 100% uploaded, but still didn't clear from the queue. The retry delays for the remaining files range between 1 and 5 hours. :-( ID: 42941 · Rating: 0 · rate: / Reply Quote

olsen_gg Send message Joined: 19 Jun 13 Posts: 5 Credit: 116,472,742 RAC: 0 Level Scientific publications	Message 42942 - Posted: 7 Mar 2016, 15:17:17 UTC Same here! Great! ID: 42942 · Rating: 0 · rate: / Reply Quote

Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1639 Credit: 10,159,968,649 RAC: 0 Level Scientific publications	Message 42943 - Posted: 7 Mar 2016, 15:27:53 UTC And I've just been able to clear half-a-dozen of the tasks I received following this morning's 'clearance'. But it still feels as if the storage server is teetering on a knife-edge - I take it the staff are scouring the local corner-shops for fresh hard disks and memory sticks? ;-) ID: 42943 · Rating: 0 · rate: / Reply Quote

TJ Send message Joined: 26 Jun 09 Posts: 815 Credit: 1,470,385,294 RAC: 0 Level Scientific publications	Message 42944 - Posted: 7 Mar 2016, 15:49:01 UTC Uploading just ready but was not able since yesterday, also no new work. And sites is slow. But all was a result of the 7000+ short runs off course that where ran with the power horses that doe the long runs normally so a lot of server traffic. This will happen again if the same situation occurs. P.S. downloading new work goes very slow at the moment. Happy crunching Greetings from TJ ID: 42944 · Rating: 0 · rate: / Reply Quote

Gerard Send message Joined: 26 Mar 14 Posts: 101 Credit: 0 RAC: 0 Level Scientific publications	Message 42945 - Posted: 7 Mar 2016, 16:00:09 UTC - in response to Message 42944. I am freeing more space now. Seems that we overloaded the server! :D ID: 42945 · Rating: 0 · rate: / Reply Quote