Hello GPUGRID!

Here Adrià. I'll be recovering the ACEMD3 application again, and sending new jobs of standard MD simulations
(We've been testing it these past weeks to make sure it worked well for both Windows and Linux)

The main goal of these new batch of simulations will be to validate further our capacity to predict the binding mode of ligands using simulations and adaptive sampling methods. Those of you that have been around for some time here might already be familiar with these simulations, such as the Benzamidin-Trypsin system (https://www.pnas.org/doi/abs/10.1073/pnas.1103547108) or the Dopamine D3 receptor with an antagonist ligand (https://www.nature.com/articles/s41598-018-19345-7#Ack1), which we were able to simulate thanks to GPUGRID and all your effort!

Now, we are revisiting this method, which we call in-silico binding assay (ISBA). During drug discovery campaings, it's common that you know of ligands that bind to your target, but you don't know their binding mode, the exact conformation and structure that both the ligand and the protein have when bound. Knowing the binding mode is critical for further development of the molecule into a potent and usable drug.

The most precise way of discovering the binding mode is with crystallization. However, that can take too much time or be directly impossible, depending on the protein. Therefore, we want to optimize and refine ISBA for binding mode prediction, so it can be usable during drug discovery projects. To summarize a bit our objectives, we want to predict binding modes for larger molecules than Benzamidin, with the same precision, but with less simulation time that was needed for the D3 receptor system.

To do so, we'll be using the latest version of adaptive sampling that we developed, AdaptiveBandit (https://pubs.acs.org/doi/abs/10.1021/acs.jctc.0c00205). The objective of these new simulations I'll be sending will be to benchmark AdaptiveBandit in an ISBA scenario, improve the algorithm if required and fine-tune its hyperparameters.

Let me know if there's any issue with the simulations. I'll be sending 100ns trajectories for the most part, divided in two steps.

It does not make any difference to Quico. The task will be completed on one or another computer and his science is done. It is our very expensive energy that is wasted but as Quico himself said, the science gets done who cares about wasted energy?

Sorry for missing out for a while. We were testing ATM in a setup not available for GPUGRID. But we're back to crunching :)

I've seen that more or less everything is running fine. Albeit for some crashes that can happen everything seems to come back to me fine.

Is there anything in specific I should look into it? I already know about the progress reporting issue (if it persists) but there's not much more I can do on my end. If they plan to update the GPUGRID app at some point I'll insist.

I have question regarding the minimum hardware requirements (i.e. Amount, speed, type of RAM, CPU speed and type, motherboard speed and requirements, etc.) for the computer to be able to complete successfully, these units for either windows and linux OS?

One of my computers has been running these units successfully, the other has not. They both have the same OS, but have different hardware. I just want to know the limits.

I had 14 errors in a row last night, between about 18:30 and 19:30 UTC. All failed with a variant of

Running command git clone --filter=blob:none --quiet https://github.com/raimis/AToM-OpenMM.git /hdd/boinc-client/slots/5/tmp/pip-req-build-_q32nezm
fatal: unable to access 'https://github.com/raimis/AToM-OpenMM.git/': Recv failure: Connection reset by peer

Is that something you can control?

Let me see if I can find the other few that crashed and cancel those WU.

FileNotFoundError:
https://www.gpugrid.net/result.php?resultid=33507881
Energy is NaN:
https://www.gpugrid.net/result.php?resultid=33507902
https://www.gpugrid.net/result.php?resultid=33509252
ImportError:
https://www.gpugrid.net/result.php?resultid=33504227
https://www.gpugrid.net/result.php?resultid=33503240

The difference between 0-5 and n-5 has been consistent throughout - there hasn't been a "fix and revert". Just new data runs starting from 0 again.

"ValueError: Energy is NaN" is back quite often :-(

FileNotFoundError: [Errno 2] No such file or directory: 'TYK2_m42_m54_0.xml'

https://www.gpugrid.net/result.php?resultid=33509037

:-(

OK, confirmed - it is still the Apache problem.

Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: HTTP/1.1 413 Request Entity Too Large
Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: Date: Tue, 09 May 2023 14:06:15 GMT
Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_auth_gssapi/1.5.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5

File (the larger of two) is 754.1 MB (Linux decimal), 719.15 MB (Boinc binary).

At this end, we have two choices:

1) Abort the data transfer, as Ian suggests.
2) Wait 90 days for somebody to find the key to the server closet.

Quico?

That's weird. I'll get a look.
But this shouldn't happen so cancel the BACE (uppercase) runs. I'll have a look on how to do it from here.
With the last implentation there shouldn't be such related file-size issues.

All bad/bug jobs should be cancelled by now.

this is not something new from this project. this has been a recurring issue from time to time. seems to pop up about every year or so whenever the result files get so large for one reason or another. so don't feel bad if you are unable to find the setting to fix the file size limit. no one else from the project has been able to for the last several years.

why are the result files so large? 500+MB. that's the root cause of the issue. do you need the data in these files? if not, why are they being created?

OK, confirmed - it is still the Apache problem.

Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: HTTP/1.1 413 Request Entity Too Large
Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: Date: Tue, 09 May 2023 14:06:15 GMT
Tue 09 May 2023 15:06:15 BST | GPUGRID | [http] [ID#15383] Received header from server: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_auth_gssapi/1.5.1 mod_auth_kerb/5.4 mod_fcgid/2.3.9 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5

File (the larger of two) is 754.1 MB (Linux decimal), 719.15 MB (Boinc binary).

At this end, we have two choices:

1) Abort the data transfer, as Ian suggests.
2) Wait 90 days for somebody to find the key to the server closet.

Quico?

And a similar batch configuration error with today's BACE run, like

BACE_m24_m7e_5-QUICO_ATM_Sage_xTB-0-5-RND7993_0

08:05:32 (386384): wrapper: running bin/bash (run.sh)
bin/bash: run.sh: No such file or directory

(five so far)

Edit - now wasted 20 of the things, and switched to Python to avoid quota errors. I should have dropped in to give you a hand when passing through Barcelona at the weekend!

Thanks. Now I know what I'm looking for (and when), I was able to watch the next transition.

Task PTP1B_new_20669_2qbr_23472_T1_2A-QUICO_TEST_ATM-0-1-RND5753_3 started with a couple of 0.1% initial steps (as usual), but then jumped to 50.983%. It then moved on by 0.441% every five minutes or so.

The run.log shows the same figures as before: a pre-existing run of 114 samples, then the real work starts with sample 115, and should proceed to a max_sample of 341. The progress jumps match the completion of samples 115 - 120.

The %age intervals match the formula in Emilio Gallicchio's post 60160 (115/(341-114)), but I can't see where the initial big value of 50.983 comes from.

Also, I don't follow the logic of the resend explanation. Mine is replication _3, so there have been 3 previous attempts - but none of them got beyond the program setup stages: all failed in less than 100 seconds. So who did the first 114 samples?

OK, it's the same story as yesterday. This task:

PTP1B_23486_23479_4_2A-QUICO_TEST_ATM-0-1-RND5081_2

downloaded at 15:26:54 UTC yesterday, and started running at about 16:30 UTC.

As before, the run.log shows a MAX_SAMPLES: 114, with timings that don't match my machine. The 16:30 run has MAX_SAMPLES: 341, and starts running with sample 115.

The machine downloaded a new task at 3:50:47 UTC: that normally happens around 85 - 90% progress, with an hour to run - but the existing one is still only at sample 308, so maybe three hours to go. And it's another PTP1B_new_ resend, so we may have to go round the cycle again.

I've read some people mentioning that the reporter doesn't work or that it goes over 100%. Does it work correctly for someone?

It varies from task to task - or, I suspect, from batch to batch. I mentioned a specific problem with a JNK1 task - task 33380692 - but it's not a general problem.

I suspect that it may have been a specific problem with setting the data that drives the progress %age calculation - the wrong expected 'total number of samples' may have been used.

I can't answer immediately on the termination question, but it's all open-source and I can look through it. In this case, it's more complicated, because BOINC will talk to the wrapper, and the wrapper will talk to the science app.

But the basic idea is that BOINC will send a request to terminate over the API, and wait for the application to close itself down as it sees fit. Actual signals will only be used to force termination in the case of an unconditional quit, such as an operating system closedown.

Right, probably the wrapper should send a termination signal to AToM.

We have of course access to AToM's sources https://github.com/Gallicchio-Lab/AToM-OpenMM and we can make sure that it checkpoints appropriately when it receives the signal.

However, I do not have access to the wrapper. Quico: please advise.

progress reporting is still not working.

instead of halting progress at 75%, it now halts at 0.19%. the weights help prevent the task from jumping to 75%, but there is still something missing.

Python tasks are able to jump to about 1% after the extraction phase due to the weights, and then slowly creeps up over time as the task progresses. 2%, 3%, 4%, etc until it hits 100% in a natural and linear way. The ATM tasks do not do this at all. they sit at 0.19% for hours and hours with no indication of when they will complete. is it 4hrs? is it 20hrs? there's no feedback to the user. when it's done it just jumps to 100% without warning.

makes it very difficult to tell is a task is stuck or working.

-Edit-

The "BACE" tasks do seem to be reporting progress now. but the earlier tasks from yesterday ("T_p38") do not.