I think I've encountered the "[error] Error reported by file upload server: can't open file" message before on other BOINC projects, and found it a difficult one to decipher.

IIRC, it's basically in two parts:

"Error reported by file upload server" - it's the upload component of the BOINC server suite that's telling you there's a problem.

"[error] can't open file" - but the problem is that *your computer* can't open the file, so there's nothing that can be uploaded.

I think that it is server problem. Server can't open file to write the upload to (or log file). How can the serve know that client can't open file? The same happened on MindModelling forum

All of your services are on the same host (status page): don't use VM or Docker ?

How VM would help? Remember there is overhead for every VM. And linux is quite good at isolating programs if configured properly. And if the VM host crashes all VMs go down too.

What could help is to move the download and or upload service to another host and better network.

---

I think that it is server problem. Server can't open file to write the upload to (or log file). How can the serve know that client can't open file? The same happened on MindModelling forum

No, that results in a different and more explicit error message, like for example "[error] Error reported by file upload server: [1x81-GERARD_CXCL12_LIG10-4-5-RND5536_0_9] locked by file_upload_handler PID=17978" (from message 39407 and surrounding discussion). That one was certainly a problem on the server, but I think in the case of "can't open file", the server is simply echoing back a message received from the reporting client. But I'll check some more.

What could help is to move the download and or upload service to another host and better network.

That would certainly help with the huge CUDA runtime files, because they are created by NVidia and redistributed without alteration by the project. I actually downloaded the whole CUDA 8.0 development toolkit (1.2 GB) during testing over the weekend - it only took about five minutes - and confirmed that the cufft_80.dll it contains is identical to the one distributed here.

I think that BOINC should develop (perhaps should already have developed) a more efficient way of handling shared library files between projects, but that won't happen overnight. Instead, we could ask somebody else to host the files and allow GPUGrid to use a download url pointing to their servers: maybe another BOINC project with ample resources - Einstein comes to mind - or even NVidia themselves as a project sponsor could provide the download facility.

Of course, the CUDA 8.0 files only need to be downloaded once by each user - either on first joining the project, or on purchasing their first GTX 10xx GPU. Subsequent machines can be provided with the file by copying locally, and I'd advise backing these files up for re-use before making major changes like re-installing an operating system.

But that won't help us with the files required for an individual task: I think each one is distinct, and would need to be uploaded separately to a mirror server before then could be downloaded by the end user - that is likely to be just as problematic. We still need to solve the underlying problem for those files.

Hi, first time here on the forum.
Since 1st of December my GPU is number crunching as always, unlike what I read in this thread. But, I don't get any points anymore... Is this a related issue to the server update?

Link to screenshot of stats graph: https://www.dropbox.com/s/w9z8vu0riq3x2r1/Schermafdruk%202016-12-07%2018.22.15.png?dl=0

Best regards, Ernst

You haven't completed any work units, or even been sent any new ones, since 2 December.
http://www.gpugrid.net/results.php?hostid=387577

Thanks,
So does it mean that my GPU might be too slow? See screenshot here: https://www.dropbox.com/s/w6y5vv2d6gudjme/Schermafdruk%202016-12-07%2019.53.30.png?dl=0
Anything I can do?

All of your services are on the same host (status page): don't use VM or Docker ?

How VM would help? Remember there is overhead for every VM. And linux is quite good at isolating programs if configured properly. And if the VM host crashes all VMs go down too.

What could help is to move the download and or upload service to another host and better network.

Because is most easier for modify a service without impact others, have different versions of a prog/API/lib and scalable.
We don't talk about a lambda user use but for professional use. It is quite different ! 15 years as an Admin Sys, I know a few... And of course, if you just have one host and it crash..., but it is already the case. For eleminate a SPOF (Single Point of failure) you need money and brains...

So does it mean that my GPU might be too slow?

You card, a GTX 650 is not the fastest in the world these days, but should be able to complete a work unit in a few days I would guess. Is the percentage complete increasing? If so, I would just let it run.

But I notice you have VirtualBox projects also. They have sometimes caused problems for me, so if your work unit is stuck (not increasing), I would reboot and see what happens. It might un-stick it, or it might error out.

EDIT: Also, be sure to reserve a CPU core for the GTX 650, or it will be slow for sure. You can do that by setting the BOINC Manager preferences to "use at most 90%" of the CPUs" for example, or whatever it takes. And some VirtualBox projects (the multi-core ones) like to grab all the CPU cores that they can, so that may be why the GTX 650 is running slowly.

What could help is to move the download and or upload service to another host and better network.

That would certainly help with the huge CUDA runtime files, because they are created by NVidia and redistributed without alteration by the project. I actually downloaded the whole CUDA 8.0 development toolkit (1.2 GB) during testing over the weekend - it only took about five minutes - and confirmed that the cufft_80.dll it contains is identical to the one distributed here.

The gridrepublic project is already providing somewhat similar services for the Fight Neglected Diseases project; you might want to contact them.

https://compute.gridrepublic.org

I think that BOINC should develop (perhaps should already have developed) a more efficient way of handling shared library files between projects, but that won't happen overnight. Instead, we could ask somebody else to host the files and allow GPUGrid to use a download URL pointing to their servers: maybe another BOINC project with ample resources - Einstein comes to mind - or even NVidia themselves as a project sponsor could provide the download facility.

In other words, modify BOINC to allow specifying a URL to download a file from, as well as the name of the file to download, on the list of files to download. What I've seen so far indicates that this is currently possible only by specifying one alternate URL to download ALL of the files from and upload ALL of the files to.

I assume that you're aware of the boinc_dev mailing list as the place to request changes to BOINC. If not, see:

http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev

That would require developing a way to avoid file name conflicts between shared files used by different projects. One way to do that would be to extend the system currently used for sharing files used for sharing files between multiple tasks from the same BOINC project to allow specifying which project to look for the file under, then create a new type of BOINC project that does no more than download large files needed by some other BOINC project.

The forwarding services already available, such as gridrepublic and World Community Grid, only need to download one copy of the files specific to each workunit, regardless of how many wingmates tasks for that workunit will be sent to. So such forwarding already provides some server relief, just not as much as you want.

World Community Grid, however, requires a code review for all applications they forward; I'm not sure if you'll consider that fast enough to be worthwhile.

No work coming down...what's up?

Hi,

dont have really time to debug. But I resetted the project, I deleted it, I have added it again. Download stucks at alsowas 0%.

Kind regards
Dirk

I'm getting tasks on my laptop, my desktop won't get them though. It might have t do with its wifi signal because WGC does the same thing. I'll try using my laptop as an access point.

Everything fine for me download/upload and running WU.

But don't always get work to do, I'm internet connected 50% of the time via cell phone hotspot. So when a reconnect, sometime, no more WU, hope more WU to come available.

Using Folding@Home as backup for work.

Copy&Paste from other post:

I think there is value looking into the "inside Spain vs outside Spain" angle. Many of these traceroutes have an increase in ping time starting when the connection gets from some ISP or backbone to the Spanish ISP or backbone. Also the user who reports no issues at all is the one inside Spain. I would be interested in seeing if the internal network computers running alpha, beta, regular, or testing units are seeing any issues like we are seeing. I would also be interested in seeing if any of the students or members of staff (etc) have PCs at home in Spain that could test the "inside Spain" theory for us outside of the local network. And perhaps it is the ISP within Spain that is just not good at the handoff to other ISPs through the backbones or some filtering method being used on the Spanish ISP networks for packet capture and analyzation. I would think that would more prevalent in places like Turkey, China, and Pakistan where it is a known issue that the government is tracking all the traffic in and out of the country to censor it, but maybe Spain has something in place that does something for some reason having this effect on our connections to the project???

I also do see different sounding reports from users within the same country, but on different ISPs or areas of that country. Some Comcast users in remote areas seeing increased issues than Comcast users in populated ares, but still reporting the issues are only with this project. I would wonder if some of those with the worst download and upload issues are also getting a greater number of failed tasks due to files that had dropped packets and seemed complete, but had some issue that caused the failures.

I will try the traceroute and downloading that file in both Firefox and BitCommet to see if there is any difference, failures, or interruptive delays.

One added thing I will add as an observation though is the increased reports of tasks that are listed for machines that do not have those tasks. I have one that has a delivery date of today on it to a computer that does not have that task on it. http://www.gpugrid.net/workunit.php?wuid=12186647 I wonder if this and the connection issue is also related, as a "tried to send" issue that one side (server) thinks it was sent and received while the other side (client) knows it got no such WU?

---

1 Corinthians 9:16 "For though I preach the gospel, I have nothing to glory of: for necessity is laid upon me; yea, woe is unto me, if I preach not the gospel!"
Ephesians 6:18-20, please ;-)
http://tbc-pa.org

I'd also say that apparently minor variations in local networking seem to be significant - possibly the route to Spain is peculiarly sensitive to timing errors in the networking protocols?

I have a combined ADSL modem/router downloading at 40 Mbit/sec and uploading at 10 Mbit/sec (the ADSL link only runs as far as the nearest street cabinet, where it's connected to an optical fibre feed). The router is connected to a 16-port gigabit switch.

From there, my four computers running GPUGrid are connected as follows:

A) 43362 and 43404 are close to the switch and connected to the switch by a gigabit patch cable.

B) 45218 is in another room, connected via trunk Cat5e cable and a secondary 100 Mbit switch and patch cable. The secondary switch is also a wireless access point for:

C) 388426 connected to the access point by 802.11'n' WiFi.

The two computers at location 'A' can both download the 140MB cufft64_80.dll in about 30 seconds, either by http or by https, consistently - I tried over the weekend, and I tried again just now.

The computer at location 'B' won't download cufft64_80.dll at all today, although it's my 'daily driver', and all other internet tabs in my browser (at least a dozen) are refreshing normally as I type. Other days, it's at least started the download, but normally stalls after less than 20 MB.

The computer at location 'C' is a new testbed, which I can use either wired or wireless, running Windows 7 or Windows 10. 'wired' means another run of Cat5e back to the gigabit switch, and no secondary switch. Under optimal conditions (wired, Win 7) it can also download the fft file in 30 seconds using my normal Chrome browser, but that fails completely in Windows 10. I can, however, download the file using the wired connection and Microsoft Edge.

The two computers at 'A' rarely stall when downloading a task datafile, but the two at 'B' and 'C' usually stall on at least one file, but complete the download at the second or third attempt.

So, big variations even within my house in northern England, before I even consider routes to Spain.

Tracing route to www.gpugrid.net [84.89.134.145] over a maximum of 30 hops:
  1     8 ms     8 ms     8 ms  bdl1.rdl-cbr2.phdl-rdl.pa.cable.rcn.net [10.25.216.1]
  2    11 ms     9 ms     9 ms  bdle25-sub212.aggr2.phdl.pa.rcn.net [207.172.196.219]
  3     8 ms     8 ms    10 ms  xe-4-1-0.bar2.Philadelphia1.Level3.net [4.78.154.89]
  4   105 ms   105 ms   105 ms  ae-1-3101.bar1.Madrid2.Level3.net [4.69.210.222]
  5   112 ms   104 ms   105 ms  ae-1-3101.bar1.Madrid2.Level3.net [4.69.210.222]
  6   104 ms   105 ms   109 ms  213.242.113.78
  7   116 ms   112 ms   111 ms  TELMAD.AE4.uv.rt1.val.red.rediris.es [130.206.245.89]
  8   117 ms   119 ms   119 ms  anella-val1-router.red.rediris.es [130.206.211.70]
  9     *        *        *     Request timed out.
 10   151 ms   112 ms   121 ms  grosso.upf.edu [84.89.134.145]
 11   115 ms   118 ms   116 ms  grosso.upf.edu [84.89.134.145]
 12   119 ms   113 ms   117 ms  grosso.upf.edu [84.89.134.145]
Trace complete.

As expected, the time in ms jumps significantly on the handoff from the US side of the Level3 backbone to the Spanish side of it. Then all the traffic within Spain is just as high a difference while the USA based connections are almost single digit and the Spanish based and handoff ones are triple digit ms times.

The cufft64_80.dll file stops at 14,557 packets 20,167,320 bytes received over https on a Firefox browser.

Over http on the Firefox browser it stops at around the same place at 14,480 packets 19,993,440 bytes.

On Bitcommet (a torrent and http downloader) it took 7 min 11 sec to complete across 20 connections to the file. There were errors in the logs. Some of the connections ended in

2016-12-08 17:25:20 Error occurred: Server disconnected unexpectedly (error code: 10060)

Most of the logs looked like this though. (This is a copy of "Connection 1" as listed in the below connections picture.)

2016-12-08 17:23:36	Content-Type: application/octet-stream
2016-12-08 17:23:36	Start receiving data...
2016-12-08 17:23:36	Received 13352 bytes.
2016-12-08 17:23:36	Connection closed.
2016-12-08 17:23:36	Connecting to www.gpugrid.net:80...
2016-12-08 17:24:17	Conntecting to www.gpugrid.net:80 succeeded.
2016-12-08 17:24:17	GET /download/cufft64_80.dll HTTP/1.1
2016-12-08 17:24:17	Host: www.gpugrid.net
2016-12-08 17:24:17	Connection: close
2016-12-08 17:24:17	Accept: */*
2016-12-08 17:24:17	Range: bytes=122332803-122994701
2016-12-08 17:24:17	User-Agent: Mozilla/4.0 (compatible; MSIE 11.; Windows NT 6.2; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)
2016-12-08 17:24:17	Pragma: no-cache
2016-12-08 17:24:17	Cache-Control: no-cache
2016-12-08 17:24:18	HTTP/1.1 206 Partial Content
2016-12-08 17:24:18	Date: Thu, 08 Dec 2016 22:24:11 GMT
2016-12-08 17:24:18	Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
2016-12-08 17:24:18	Last-Modified: Thu, 27 Oct 2016 15:15:15 GMT
2016-12-08 17:24:18	ETag: "8b04238-53fda356846c0"
2016-12-08 17:24:18	Accept-Ranges: bytes
2016-12-08 17:24:18	Content-Length: 661899
2016-12-08 17:24:18	Content-Range: bytes 122332803-122994701/145769016
2016-12-08 17:24:18	Connection: close
2016-12-08 17:24:18	Content-Type: application/octet-stream
2016-12-08 17:24:18	Start receiving data...
2016-12-08 17:24:18	Received 661899 bytes.
2016-12-08 17:24:18	Connection closed by server.
2016-12-08 17:24:18	Connecting to www.gpugrid.net:80...
2016-12-08 17:24:59	Conntecting to www.gpugrid.net:80 succeeded.
2016-12-08 17:24:59	GET /download/cufft64_80.dll HTTP/1.1
2016-12-08 17:24:59	Host: www.gpugrid.net
2016-12-08 17:24:59	Connection: close
2016-12-08 17:24:59	Accept: */*
2016-12-08 17:24:59	Range: bytes=112686094-112832152
2016-12-08 17:24:59	User-Agent: Mozilla/4.0 (compatible; MSIE 11.; Windows NT 6.2; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)
2016-12-08 17:24:59	Pragma: no-cache
2016-12-08 17:24:59	Cache-Control: no-cache
2016-12-08 17:24:59	HTTP/1.1 206 Partial Content
2016-12-08 17:24:59	Date: Thu, 08 Dec 2016 22:24:53 GMT
2016-12-08 17:24:59	Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
2016-12-08 17:24:59	Last-Modified: Thu, 27 Oct 2016 15:15:15 GMT
2016-12-08 17:24:59	ETag: "8b04238-53fda356846c0"
2016-12-08 17:24:59	Accept-Ranges: bytes
2016-12-08 17:24:59	Content-Length: 146059
2016-12-08 17:24:59	Content-Range: bytes 112686094-112832152/145769016
2016-12-08 17:24:59	Connection: close
2016-12-08 17:24:59	Content-Type: application/octet-stream
2016-12-08 17:24:59	Start receiving data...
2016-12-08 17:24:59	Received 13353 bytes.
2016-12-08 17:24:59	Connection closed.
2016-12-08 17:24:59	Connecting to www.gpugrid.net:80...
2016-12-08 17:25:38	Conntecting to www.gpugrid.net:80 succeeded.
2016-12-08 17:25:38	GET /download/cufft64_80.dll HTTP/1.1
2016-12-08 17:25:38	Host: www.gpugrid.net
2016-12-08 17:25:38	Connection: close
2016-12-08 17:25:38	Accept: */*
2016-12-08 17:25:38	Range: bytes=39287322-39393920
2016-12-08 17:25:38	User-Agent: Mozilla/4.0 (compatible; MSIE 11.; Windows NT 6.2; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)
2016-12-08 17:25:38	Pragma: no-cache
2016-12-08 17:25:38	Cache-Control: no-cache
2016-12-08 17:25:38	HTTP/1.1 206 Partial Content
2016-12-08 17:25:38	Date: Thu, 08 Dec 2016 22:25:32 GMT
2016-12-08 17:25:38	Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
2016-12-08 17:25:38	Last-Modified: Thu, 27 Oct 2016 15:15:15 GMT
2016-12-08 17:25:38	ETag: "8b04238-53fda356846c0"
2016-12-08 17:25:38	Accept-Ranges: bytes
2016-12-08 17:25:38	Content-Length: 106599
2016-12-08 17:25:38	Content-Range: bytes 39287322-39393920/145769016
2016-12-08 17:25:38	Connection: close
2016-12-08 17:25:38	Content-Type: application/octet-stream
2016-12-08 17:25:38	Start receiving data...
2016-12-08 17:25:38	Received 13355 bytes.
2016-12-08 17:25:38	Connection closed.
2016-12-08 17:25:38	Connecting to www.gpugrid.net:80...
2016-12-08 17:26:15	Conntecting to www.gpugrid.net:80 succeeded.
2016-12-08 17:26:15	GET /download/cufft64_80.dll HTTP/1.1
2016-12-08 17:26:15	Host: www.gpugrid.net
2016-12-08 17:26:15	Connection: close
2016-12-08 17:26:15	Accept: */*
2016-12-08 17:26:15	Range: bytes=121705404-121772270
2016-12-08 17:26:15	User-Agent: Mozilla/4.0 (compatible; MSIE 11.; Windows NT 6.2; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729)
2016-12-08 17:26:15	Pragma: no-cache
2016-12-08 17:26:15	Cache-Control: no-cache
2016-12-08 17:26:16	HTTP/1.1 206 Partial Content
2016-12-08 17:26:16	Date: Thu, 08 Dec 2016 22:26:09 GMT
2016-12-08 17:26:16	Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_auth_gssapi/1.3.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
2016-12-08 17:26:16	Last-Modified: Thu, 27 Oct 2016 15:15:15 GMT
2016-12-08 17:26:16	ETag: "8b04238-53fda356846c0"
2016-12-08 17:26:16	Accept-Ranges: bytes
2016-12-08 17:26:16	Content-Length: 66867
2016-12-08 17:26:16	Content-Range: bytes 121705404-121772270/145769016
2016-12-08 17:26:16	Connection: close
2016-12-08 17:26:16	Content-Type: application/octet-stream
2016-12-08 17:26:16	Start receiving data...
2016-12-08 17:26:16	Received 13354 bytes.
2016-12-08 17:26:16	Connection closed.
2016-12-08 17:26:16	Connecting to www.gpugrid.net:80...
2016-12-08 17:26:39	Received 0 bytes.
2016-12-08 17:26:39	Connection closed.
2016-12-08 17:26:39	Connection stopped.

This is what the Summary of the download looked like:


This is what the connections looked like:


Here is the xml info from how the download went and what its internal settings were to download it on via Bitcommet. I am not sure if this is relevant so I am including it.

DownloadInfo
DownloadBytes="145769016"
CreateDate="2016-12-08 17:19:27.940523"
FinishDate="2016-12-08 17:26:39.663576"
MaxConnection="20"
downloaded_filesize="0"
no_size_file_complete="false"
support_set_range="true"
filesize="145769016"
know_filesize="1"
block_list="22 serialization::archive 3 0 0 0"
file_created="true"
rate_download="0"
save_file_name_original="cufft64_80.dll"
enable_p2sp="false"
enable_dl_from_mirror="false"
file_hash="badf879aa99fdd138bc2d362bd661ef76ba98431"

So, big variations even within my house in northern England, before I even consider routes to Spain.

Truly remarkable. I have been fighting a long battle to get my PC to record videos from my TV cable that are converted by HDHomeRun into a standard 100 Mbps Ethernet format. To eliminate the various glitches, I had to turn off flow control. Perhaps that is counter-intuitive, but the various protocols are known to fight against each other; I am not the first to discover that. Whether than has anything to do with Spain I don't know, but it might account for local variations. There are other LAN settings (Interrupt Moderation, Energy Efficient Ethernet, etc.) that are known sources of trouble too, not to mention WiFi signal problems of course.

I'm getting tasks on my laptop, my desktop won't get them though. It might have t do with its wifi signal because WGC does the same thing. I'll try using my laptop as an access point.

If you use BOINC Manager (Advanced View) on your desktop, click on Activity, then make sure you do not have it set to Suspend network activity or Suspend GPU or Suspend. Also click on Projects, then GPUGRID, and make sure you do not have it set to No new tasks (listed in the Status column).

If you use Simple View, click on View, then Advanced View, to switch views. To switch it back, click on View, then Simple View.

Also click on Projects, then on one of the BOINC projects listed, then Home page. You should then see a web page for that project.

By the way, what GPU (or graphics card) does that desktop use? GPUGRID can use only some of the graphics card models.

Any idea when new WUs will be available again?

As expected, the time in ms jumps significantly on the handoff from the US side of the Level3 backbone to the Spanish side of it. Then all the traffic within Spain is just as high a difference while the USA based connections are almost single digit and the Spanish based and handoff ones are triple digit ms times.

There's a simple explanation for the triple digit (in miliseconds) round-trip times for US residents: the signal travels at the speed of light (no matter if it's on copper or glass fibre), and Philadelphia is 6289km (3908mi) away from Barcelona, it takes ~21ms for the signal to travel that far, so ~42ms adds to the round-trip times because the speed of light is so low :). US residents in the west coast should add ~26ms latency.