What is the ghost recovery procedure on this project?

I've tried the way it works for SETI, but it didn't work here.
Luckily GPUGrid has a much shorter deadline than SETI, so it's not a big problem.

the present supply (171,016) will last for about 4 days from now.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

Ghost tasks are on GPUGRID's server side.
After 5 days deadline is past, server will automatically clear ghost tasks on original host, and resend to another one.

[Clarification]

We call "Ghost task" to that the server counts as sent to a Host, but for any reason, it was not really received.
It doesn't interfere at the host side, as BOINC Manager will not see these ghost tasks, and it will continue asking for new tasks until tasks buffer is full, or maximum "2 tasks per GPU" is achieved.
On the server's side, ghost tasks are wrongly being counted as "In process" tasks, while really they are not.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

What is coming next, is a mystery...

What is the ghost recovery procedure on this project?

I've tried the way it works for SETI, but it didn't work here.
Luckily GPUGrid has a much shorter deadline than SETI, so it's not a big problem.

Thanks Zoltan, I tried my Seti ghost recovery protocol and it didn't work either.
I managed to pick up 10 ghosts and wanted to clear them.
Good thing the deadline here is so short compared to Seti.

These ghost tasks seem to occur after the server runs out of disk space. Are they somehow related to that? 🤔
_____________________________

An unrelated item: Anybody else getting this error?

(unknown error) - exit code 195 (0xc3)</message>
<stderr_txt>
01:29:38 (6776): wrapper (7.9.26016): starting
01:29:38 (6776): wrapper: running acemd3.exe (--boinc input --device 0)
EXCEPTIONAL CONDITION: src\mdio\bincoord.c, line 193: "nelems != 1"
01:29:40 (6776): acemd3.exe exited; CPU time 0.015625
01:29:40 (6776): app exit status:

It apparently signals that the WU is bad- when you track them. After getting 6 of them I'm curious what the bug might be. Bad code?

yes, I saw a bunch of bad WUs. checking the resends, they are all erroring out also on different hosts.

---

Looks like a lot of tasks lost their file references on the storage. Can't pull the correct data for the tasks.

<core_client_version>7.17.0</core_client_version>
<![CDATA[
<message>
ERROR: /home/user/conda/conda-bld/acemd3_1570536635323/work/src/mdsim/trajectory.cpp line 135: Simulation box has to be rectangular!
07:01:16 (1119448): acemd3 exited; CPU time 0.557061
07:01:16 (1119448): app exit status: 0x9e
07:01:16 (1119448): called boinc_finish(195)

I'm interpreting that message as "file is present, but contains bad contents".

On another aspect of the 'error task' problem. I'm using a very ancient predecessor of BoincTasks. It (and I think BoincTasks itself), retains the concept of "CPU efficiency", which was withdrawn from BOINC Manager several years ago.

What I'm seeing for Windows tasks is that the ACEMD worker app crashes seconds after launch, but the Wrapper app doesn't notice for some time - the task as a whole is seen by BOINC as continuing to run. This shows up as a CPU efficiency of 0.0000 (helpfully colour coded) - no CPU time is being measured for the task as a whole, instead of the usual 96% - 97%.

That low efficiency warning prompts me to look at the workunit on the website, and see if there are any previous failures (the replication number is a good hint, as well). If it's a bad workunit, I can abort and move on with less wasted time overall.

It's a technique which some users might find helpful.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

What is coming next, is a mystery...

And what we're finishing now is a complete and utter mystery as well.

And what we're finishing now is a complete and utter mystery as well

I've only been able to glean that it is a vigorous attempt at mapping the simulation environment which is meant to improve (or simplify?) future modeling methods.

If one of the admins would want to comment, we're all ears...
👂👂👂👂👂🦻👂😉

New version of ACEMD: 73,631 Unsent tasks left

⏳️

the present supply (171,016) will last for about 4 days from now.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

3 days passed, there are 11.806 workunits left, this supply will last for another 6~7 hours.

They're all gone, so what now?

Our poor GPUs start getting hangry!! :)

And I was pushing so hard for the magic 100m milestone. :(

They're all gone, so what now?

I liked this expresion:

...is a complete and utter mystery...

Familiar?
(I took note for such a moment like this)

Now that unsent tasks have reached and stuck on zero, the topic of this thread recovers full sense: Unsent tasks decreasing much more slowly
(Unless negative values are permitted, who knows?)

They're all gone, so what now?

It will take at least 5-10 days (or more) until all the workunits out in the field are finished (or timed out, and finished on another host).
I don't expect that another batch will be queued until then.
Exam period is coming, then the summer break is coming, so perhaps there won't be much work queued soon.
Unless Toni prepared some COVID-19 related work. Or perhaps we could help out the Acellera drug design people doing their job.

the difference between the tasks of the current series in contrast to all the others before is:
whereas, before, tasks still could be downloaded once in a while, as long as there were enough tasks "in process", here this seems not to be the case.
Once the "unsent" queue is dry, no more tasks can be downloaded.

I picked up 4 resends after the RTS buffer had hit zero today.

I picked up 4 resends after the RTS buffer had hit zero today.

Were they from the 'instant crashing' batch? I've had a few of those recently, though I haven't checked to see if I got any while I was asleep.