I've noticed that the number of Unsent Tasks is decreasing at a much slower rate even though the number of tasks in progress is growing and the Current GigaFLOPS is approaching record levels.

Tasks in progress had decreased from 300,000 to 250,000 in a few weeks, but now it is taking several days to decrease by only 1,000.

What changed? Are additional new tasks being added or are the tasks being crunched now more difficult?

Toni prioritized some batches before, those have run out. That made the number of unsent task to decrease more rapidly.
Now it's back to the "normal" (almost 0) rate. It means that when these will run out, the decrease will be 100 times faster than the previous faster rate.

Thanks!

On March 10th 2020 | 17:39:16 UTC Retvari Zoltan wrote at message #53884:

I'm receiving many tasks which are the last one of their batch:

1nkvA00_450_0-TONI_MDADpr4sn-9-10-RND4090_0

Or near the end of their batch:
1gaxA04_348_0-TONI_MDADpr4sg-8-10-RND1850_0

Total number of tasks in the batch
The sequential number of the given task within the batch (starting number is 0)

I expect the number of unsent tasks in the queue will drop significantly during the next days.
There are 305.826 unsent tasks as I wrote this.

As soon as series arrives 9-10 ones, it is predictable that unsent tasks will decrease again at a higher rate... (?)

As soon as series arrives 9-10 ones, it is predictable that unsent tasks will decrease again at a higher rate... (?)

All my received WUs today are this kind.
Current reading is 242.563 unsent tasks. We will be soon confirming or discarding this assumption.

Current reading is 242.563 unsent tasks. We will be soon confirming or discarding this assumption.

Current reading is 242.563 unsent tasks. We will be soon confirming or discarding this assumption.

Current reading is 222 460 that is -20 103 (8.28%) drop in 12h 20m = 27.17 / minute
If this rate is constant, the present supply will last for 5 days 16 hours 28 minutes and 50.8 seconds. :)

The current reading is 200,361 that is 42,202 (17.4%) decrease in 24h 10m = 29.10 / minute
The rate is slightly increased. According to this new rate, the present supply will last 4 days 18 hours 44 minutes 6.94 seconds from now .:)

The current reading is 200,361 that is 42,202 (17.4%) decrease in 24h 10m = 29.10 / minute
The rate is slightly increased. According to this new rate, the present supply will last 4 days 18 hours 44 minutes 6.94 seconds from now .:)

At the moment of writing this, scheduler is stopped.
I guess that this high rate in returning results has caused a new momentary buffer disk overflow...

Note that the return rate was this high all along hence there are frequent disk buffer overflows. As new tasks created from the returned tasks the number of unsent workunits remain constant, so the return rate remain hidden from us, until the batches reach their final sequence number.

It looks like the server status page needs something added - free disk space - at least for this disk areas that receive uploads.

That seems to be the current bottleneck in the project's resources.

One more conclusion that could be drawn:

- Taking Retvari Zoltan's current calculation: 29,1 average returned WUs per minute
- Taking some calculations coming from this previous outage: 6,367 MB average per returned WU
This results in 185,28 MB coming from finished WUs data returned to server per minute.
That is: 260,55 GB of data to manage per day, counting only returned WU's data. (About 1 TB every 4 days)

What we don't know - at least, I certainly don't know, and I've not seen it described here, ever - is what exactly the processing path of that data is after our raw results are returned to the server.

We do know that each of our tasks forms part of a sequential sequence of (currently) 10 tasks making up the entire job, and that at least some of our returned data is used to assemble the starting data for the next task in the sequence.

Is it all used in that way? Once it's been used, does it need to be kept? If so, how long? Can it (any of it) be discarded once the next task in sequence has been created? Has been completed? Once the whole 10-task job has been completed?

People in other threads have mentioned SETI as a comparison. There, the process is that the scientific data returned by each task is assimilated into a gigantic, 20-year, scientific database. And that once assimilation has taken place, our raw, returned, data is erased (usually within 24 hours).

If we knew for certain that our returned data needed to be retained in quick-access online storage, say until the final paper had been accepted for publication following peer review, then I'd be prepared to contribute to a fundraising drive for additional disk spindles and a chassis to mount them in. But if the daily data is simply transferred over a slow link to an offsite backing store, then spindles aren't the answer: more drives would simply delay the need for an outage from a 5 day to a 10 day interval, and then extend that outage when it eventually arrived.

Project's scheduler is just up again, with 174.874 tasks left ready to send!

All my stacked WUs have been reported as finished, and all (but one 8-10) the new WUs I've received are of the kind 9-10.
So this topic is still on fire 🔥🔥🔥

I have a couple of ghost tasks, so I suppose that many other ghost tasks are waiting to pass their deadline, so some 8-10 tasks will be re-send to other hosts.
However the present supply (171,016) will last for about 4 days from now.

What is the ghost recovery procedure on this project?

Ghost tasks are on GPUGRID's server side.
After 5 days deadline is past, server will automatically clear ghost tasks on original host, and resend to another one.

What is the ghost recovery procedure on this project?

the present supply (171,016) will last for about 4 days from now.

Ghost tasks are on GPUGRID's server side.
After 5 days deadline is past, server will automatically clear ghost tasks on original host, and resend to another one.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

What is the ghost recovery procedure on this project?

I've tried the way it works for SETI, but it didn't work here.
Luckily GPUGrid has a much shorter deadline than SETI, so it's not a big problem.

These ghost tasks seem to occur after the server runs out of disk space. Are they somehow related to that? 🤔
_____________________________

An unrelated item: Anybody else getting this error?

(unknown error) - exit code 195 (0xc3)</message>
<stderr_txt>
01:29:38 (6776): wrapper (7.9.26016): starting
01:29:38 (6776): wrapper: running acemd3.exe (--boinc input --device 0)
EXCEPTIONAL CONDITION: src\mdio\bincoord.c, line 193: "nelems != 1"
01:29:40 (6776): acemd3.exe exited; CPU time 0.015625
01:29:40 (6776): app exit status:

yes, I saw a bunch of bad WUs. checking the resends, they are all erroring out also on different hosts.
____________

Looks like a lot of tasks lost their file references on the storage. Can't pull the correct data for the tasks.

I'm interpreting that message as "file is present, but contains bad contents".

On another aspect of the 'error task' problem. I'm using a very ancient predecessor of BoincTasks. It (and I think BoincTasks itself), retains the concept of "CPU efficiency", which was withdrawn from BOINC Manager several years ago.

What I'm seeing for Windows tasks is that the ACEMD worker app crashes seconds after launch, but the Wrapper app doesn't notice for some time - the task as a whole is seen by BOINC as continuing to run. This shows up as a CPU efficiency of 0.0000 (helpfully colour coded) - no CPU time is being measured for the task as a whole, instead of the usual 96% - 97%.

That low efficiency warning prompts me to look at the workunit on the website, and see if there are any previous failures (the replication number is a good hint, as well). If it's a bad workunit, I can abort and move on with less wasted time overall.

It's a technique which some users might find helpful.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

What is coming next, is a mystery...

And what we're finishing now is a complete and utter mystery as well

New version of ACEMD: 73,631 Unsent tasks left

⏳️

the present supply (171,016) will last for about 4 days from now.

The rate of decline seems to stabilize around 30/minute, so the supply will last for about 3 days from now.
(exactly 2 days 22 hours 39 minutes and 42.9 seconds)

They're all gone, so what now?

Our poor GPUs start getting hangry!! :)

And I was pushing so hard for the magic 100m milestone. :(

They're all gone, so what now?

...is a complete and utter mystery...

They're all gone, so what now?

the difference between the tasks of the current series in contrast to all the others before is:
whereas, before, tasks still could be downloaded once in a while, as long as there were enough tasks "in process", here this seems not to be the case.
Once the "unsent" queue is dry, no more tasks can be downloaded.

I picked up 4 resends after the RTS buffer had hit zero today.

I picked up 4 resends after the RTS buffer had hit zero today.

No,3 in fact were original issue -1's from yesterday. Must have been the very last issued. One was a -2 resend from an aborted user. None were from the badly formatted task run. I got lucky.

I was just surprised to see the cache increase after I had seen the RTS count on the SSP go to zero.

At this project, only the _0 are original issue. _1 is already a replacement, unlike projects which use comparison validation.

But I'm glad you got some meat off the bones.

Thanks for correcting me Richard. I forgot about the workunits on this project with quorum of 1.

I figure what we see trickle out for a while will be timed-out tasks that are recycled by Grosso. I'm curious as to how many tasks expire on how many hosts by the end of a run the size of this one.

Friends, don't dismiss the amount of raw computing power that the project we all have just accomplished represents! Even if Grosso isn't always as awesome as its nickname suggests, the DC network it supports is truly awesome in its potential power. That power finds its source in every one of us and our individual support.

And beside that, where else can we accrue BOINC cobblestones this fast? 😉

Hey, look at this little hiatus as "recess", where you get to go find out what the kids in the other classes are crunching. You might get a little case of hardware lust.

Just remember to practice virtual social distancing! 💻🌎🖥🌏🖥🌍💻😎

Looks like I'm participating in that trickle-down, too. Somebody let WU 19993861 slip past it's deadline, so they tossed it back for me.

folding@home has many GPU work units now against Corona Virus. So that would be a good option.

folding@home has many GPU work units now against Corona Virus. So that would be a good option.

I signed up for folding@home at least a week ago, and then enabled GPU work for them. No GPU work downloaded so far, only CPU work.

Also, so far, I've been unable to log into their forums.

OK, that site offers a more recent version.

I hope that updating it will not disturb work in progress - there doesn't seem to be a way to tell the previous version to finish any work in progress, but not start more.

Robert, I had to adjust the slider to full power to get my GPUs to engage. It will take a while to catch some available work the first time as I recall. Once you have GPU tasks you can run at any speed.
The app is great alongside of BOINC from my recent experience. I can multi-task my GPUs that way. Just remember twice the tasks means half the speed for each task.

Pop Piasa,

Thanks. Than started my first Folding@Home GPU task.

Here we are again in the same situation:
My last received task is named 1nh8A02_348_1-TONI_MDADex2sn-49-50-RND4451_0
That is, we have arrived to the last tasks of current batch.
If repeating similar past behaviors, unsent tasks will start decreasing more quickly until they get exhausted.
With our effort, a new milestone at GPUGrid is likely to be reached...
It's somehow exciting: What will come the next?

What will come the next?

August is next.

maybe tasks related to Corona?

There are so many anti-virals and vaccines under testing now, and probably available in a few months, that any additional computer project would get lost in the noise. I think they are better in focusing on more general subject matter, where they can make a difference in the longer term. The databases or whatever it is that they are establishing look worthwhile to me.

About two days of tasks left to think about it...

⏳️

There are so many anti-virals and vaccines under testing now, and probably available in a few months, that any additional computer project would get lost in the noise. I think they are better in focusing on more general subject matter, where they can make a difference in the longer term. The databases or whatever it is that they are establishing look worthwhile to me.

All true, but that is not the point. It is not what GPU work the projects can provide, but what we can do for the projects, and hence the science. Even if you handed them a new recipe to test now (there seem to be hundreds waiting in the wings to be tested), they could not do it for several more months. By then at least a first round of anti-virals will be ready, with the vaccines not far behind.

As you know, there has been a huge effort by labs all over the world. Just because it is a GPU project does not guarantee any special status for the results. This video, posted by Jim Slade on WCG, shows the effort underway in the U.S. to test them all, and figure out how to produce them in time.
I don't think we can do more here.
https://www.youtube.com/watch?v=uEgfwVy9w8A&feature=youtu.be

If repeating similar past behaviors, unsent tasks will start decreasing more quickly until they get exhausted.

If repeating similar past behaviors, unsent tasks will start decreasing more quickly until they get exhausted.

I love GPUGrid. It is a surprises box...
Suddenly, behavior has given a quantum jump, and it somehow differs from precedents:
Unsent tasks started to decrease more slowly today, in coincidence with receiving new tasks as:
e1s241_17gen-PABLO_UCB_NMR_KIX_CMYB_8-0-5-RND1858_1
e1s321_villin_100ns_6-ADRIA_VillinAdaptive100ns-0-1-RND7450_0
1b5fC00_320_4-TONI_MDADpr4sb-0-10-RND9302_2
1ac5A00_320_0-TONI_MDADex7sa-0-50-RND9726_2
...
To be continued... (?)

If repeating similar past behaviors, unsent tasks will start decreasing more quickly until they get exhausted.

About two days of tasks left to think about it...

Suddenly, behavior has given a quantum jump, and it somehow differs from precedents:

At the present rate, I roughly estimate that current batch of TONI_MDAD available WUs will start to decrease at a significant higher rate at around next Sunday (January 3rd 2021)
After that, they will be extinguishing in about one more week.
If nothing changes in between, we faithfull crunchers and project's Team will be achieving another milestone at GPUGrid...

At the present rate, I roughly estimate that current batch of TONI_MDAD available WUs will start to decrease at a significant higher rate at around next Sunday (January 3rd 2021)
After that, they will be extinguishing in about one more week.
If nothing changes in between, we faithfull crunchers and project's Team will be achieving another milestone at GPUGrid...

All gone. We're down to the resends.

All gone. We're down to the resends.

No more GPUGRID work? For how long?

No telling. Could be a couple days. Could be a couple weeks. Could be months.
____________

No telling. Could be a couple days. Could be a couple weeks. Could be months.

https://www.dictionary.com/e/slang/take-for-granted/

WHAT DOES TAKE FOR GRANTED MEAN?

The expression to take for granted means "to accept without question or objection," and often implies a lack of appreciation or gratitude. (E.g., "Many of us may take for granted the fact that we have access to clean drinking water.")

When it come to people, to take someone for granted means to take advantage of, show no appreciation for, or undervalue them.