upload problems

Message boards : Server and website : upload problems
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · Next

AuthorMessage
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 54162 - Posted: 31 Mar 2020, 11:18:39 UTC - in response to Message 54158.  

This also coincides with an outage I've noted today at Rosetta@home.

Rosetta is based in the U.S. (Seattle, Washington).
They just have a shortage of work at the moment. Their users have exploded four times since the virus started.
ID: 54162 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Erich56

Send message
Joined: 1 Jan 15
Posts: 1171
Credit: 12,662,148,501
RAC: 1,014,572
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 54163 - Posted: 31 Mar 2020, 11:52:49 UTC - in response to Message 54159.  

Suspending to try and fix the full disk.

The server's disk is just a buffer. It is emptied continuously. Disk full conditions happen when there is even a temporary imbalance between in (uploads) and out (moving to the main servers) rates. A few hours of imbalance are sufficient to fill it. At such high volumes there is no "easy fix".


thanks for explaining.

After uploads worked again for a few hours this late morning, they have stopped once more since some time ago :-(
ID: 54163 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Zhouanguo

Send message
Joined: 7 Mar 20
Posts: 1
Credit: 635,496
RAC: 0
Level
Gly
Scientific publications
wat
Message 54164 - Posted: 31 Mar 2020, 12:54:35 UTC

Oh. I had suffer from it. It just started and turn to "Try again later" immediately in few seconds. Ohh... Please! fix it, quick is better.
ID: 54164 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jul 17
Posts: 404
Credit: 17,412,649,587
RAC: 8,996
Level
Trp
Scientific publications
watwatwat
Message 54177 - Posted: 31 Mar 2020, 18:08:08 UTC - in response to Message 54159.  

The server's disk is just a buffer. It is emptied continuously. Disk full conditions happen when there is even a temporary imbalance between in (uploads) and out (moving to the main servers) rates. A few hours of imbalance are sufficient to fill it. At such high volumes there is no "easy fix".

Now I understand the 2 WU per GPU limitation, you're trying to balance the goes inners and the goes outters.
ID: 54177 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.

Send message
Joined: 21 Feb 20
Posts: 1116
Credit: 40,876,970,595
RAC: 2
Level
Trp
Scientific publications
wat
Message 54182 - Posted: 1 Apr 2020, 0:06:58 UTC - in response to Message 54159.  

At such high volumes there is no "easy fix".


a larger disk buffer?
ID: 54182 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ProDigit

Send message
Joined: 13 Nov 19
Posts: 6
Credit: 87,400,696
RAC: 0
Level
Thr
Scientific publications
wat
Message 54205 - Posted: 2 Apr 2020, 16:24:23 UTC

Having the same issue. 6 GPU tasks refusing to upload.

*Edit: Ah, they just went through...
ID: 54205 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jul 17
Posts: 404
Credit: 17,412,649,587
RAC: 8,996
Level
Trp
Scientific publications
watwatwat
Message 55288 - Posted: 14 Sep 2020, 4:37:47 UTC

This is still a big problem. I have to retry to submit completed WUs several times a day to get idle GPUs working again. Please fix server issue.
ID: 55288 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jul 17
Posts: 404
Credit: 17,412,649,587
RAC: 8,996
Level
Trp
Scientific publications
watwatwat
Message 55289 - Posted: 14 Sep 2020, 13:20:56 UTC

If you can't fix your server problems you could at least let us download a day's worth of WUs. That would be a minimum 12 WUs per GPU.
ID: 55289 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55291 - Posted: 14 Sep 2020, 19:48:57 UTC - in response to Message 55288.  

This is still a big problem.

I haven't seen it, though I am running only two GTX 1060s at the moment. Since your computers are hidden, there is not much more to say.

ID: 55291 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jul 17
Posts: 404
Credit: 17,412,649,587
RAC: 8,996
Level
Trp
Scientific publications
watwatwat
Message 55295 - Posted: 15 Sep 2020, 12:40:38 UTC

Every morning I wake up to most of my GPUs sitting idle waiting for GG to UL & subsequently DL WUs. I know of no other project that is any where near as inefficient at keeping work supplied.
Why don't you get someone from another project to look at your server configuration??? Maybe valterc at TN-GRID, the most efficient BOINC project going.
ID: 55295 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Ian&Steve C.

Send message
Joined: 21 Feb 20
Posts: 1116
Credit: 40,876,970,595
RAC: 2
Level
Trp
Scientific publications
wat
Message 55297 - Posted: 15 Sep 2020, 14:17:29 UTC - in response to Message 55295.  

I'm not having any issue with uploads or downloads.
ID: 55297 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Trotador

Send message
Joined: 25 Mar 12
Posts: 103
Credit: 14,957,179,771
RAC: 105,599
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55298 - Posted: 16 Sep 2020, 7:01:17 UTC

I have also frequent upload problems. Not always, not across all my systems, I haven't found a common way of ocurring but it is really happening since many months ago. And the access to the project website is very slow most of the time.

I'm using ubuntu in all my hosts.
ID: 55298 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jul 17
Posts: 404
Credit: 17,412,649,587
RAC: 8,996
Level
Trp
Scientific publications
watwatwat
Message 55299 - Posted: 16 Sep 2020, 16:04:58 UTC

After about 3 hours all GPUs have gone idle because WUs do not UL and so the paltry 2 WUs do not DL.
ID: 55299 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Jim1348

Send message
Joined: 28 Jul 12
Posts: 819
Credit: 1,591,285,971
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55301 - Posted: 16 Sep 2020, 19:49:06 UTC - in response to Message 55299.  

After about 3 hours all GPUs have gone idle because WUs do not UL and so the paltry 2 WUs do not DL.

There is a known issue on GPUGrid that after you access the server, there is some dead time before you can access it again. I am not seeing it at the moment, but it bites everyone eventually. That may be it, depending on how many machines you have.

I would guess that it is some sort of anti-DDOS protection feature on the campus network, but no one knows (or admits to) what the problem is.
ID: 55301 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile ServicEnginIC
Avatar

Send message
Joined: 24 Sep 10
Posts: 595
Credit: 13,083,686,510
RAC: 2,983,710
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 55303 - Posted: 16 Sep 2020, 21:40:45 UTC
Last modified: 16 Sep 2020, 21:44:29 UTC

There is a known issue on GPUGrid that after you access the server, there is some dead time before you can access it again...

This problem was discussed at the end of 2019 on thread Unable to load units
Currently I have 7 hosts in production, all of them attached to the same local network.
I've tested that (for me), when I want to replenish WU buffers, the most effective way is manually asking for WUs one by one host, from lowest to highest local IP. I've configured fixed IP address for each one.
These 7 hosts manage 11 GPUs in total (some of them are multiGPU systems), so the maximum WUs I can aspire to simultaneously download from GPUGrid is 22 (two at a time per each GPU)...
But most of the time, the whole group is quite well automanaging by means of their individual BOINC Managers, without human intervention.
ID: 55303 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Erich56

Send message
Joined: 1 Jan 15
Posts: 1171
Credit: 12,662,148,501
RAC: 1,014,572
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 56057 - Posted: 19 Dec 2020, 7:44:21 UTC

no finished task can be upoloaded.

19.12.2020 08:42:24 | GPUGRID | [error] Error reported by file upload server: Server is out of disk space
ID: 56057 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Gelirhil

Send message
Joined: 19 Dec 08
Posts: 3
Credit: 22,289,033
RAC: 0
Level
Pro
Scientific publications
wat
Message 56058 - Posted: 19 Dec 2020, 8:49:54 UTC

Same here:
19.12.2020 11:45:10 | GPUGRID | Started upload of 3m0eA01_379_2-TONI_MDADex2sm-43-50-RND8091_0_10
19.12.2020 11:45:11 | GPUGRID | [error] Error reported by file upload server: can't open file /home/ps3grid/projects/PS3GRID/upload/96/3m0eA01_379_2-TONI_MDADex2sm-43-50-RND8091_0_10: No space left on device
19.12.2020 11:45:11 | GPUGRID | Temporarily failed upload of 3m0eA01_379_2-TONI_MDADex2sm-43-50-RND8091_0_10: transient upload error
ID: 56058 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile [PUGLIA] kidkidkid3
Avatar

Send message
Joined: 23 Feb 11
Posts: 103
Credit: 1,774,251,957
RAC: 795,724
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 56062 - Posted: 19 Dec 2020, 11:43:08 UTC

Today is Saturday ... I hope they solve the problem next week ...
K.
Dreams do not always come true. But not because they are too big or impossible. Why did we stop believing.
(Martin Luther King)
ID: 56062 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Erich56

Send message
Joined: 1 Jan 15
Posts: 1171
Credit: 12,662,148,501
RAC: 1,014,572
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwat
Message 56063 - Posted: 19 Dec 2020, 12:19:05 UTC - in response to Message 56062.  

Today is Saturday ... I hope they solve the problem next week ...

it's not the first time this happens; so I am surprised that they havn't installed any warning system to the effect that someone gets a notification as soon as the disk is about 80% full or so.
ID: 56063 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Greg _BE

Send message
Joined: 30 Jun 14
Posts: 153
Credit: 131,154,684
RAC: 2,594
Level
Cys
Scientific publications
watwatwatwatwatwat
Message 56064 - Posted: 19 Dec 2020, 12:20:10 UTC

Still down: 12/19/2020 1:13:30 PM (CET) | GPUGRID | [error] Error reported by file upload server: Server is out of disk space
ID: 56064 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · Next

Message boards : Server and website : upload problems

©2026 Universitat Pompeu Fabra