There are projects giving out VMs instead of normal applications. We are considering it. How is it? Does it work?

thanks.

gdf

The people to ask would be the guys at CERN. They developed the technology and the hooks into the BOINC framework. See projects like Virtual LHC@home, ATLAS@Home, and CMS-dev.

Their particular solution (I'm sure others are possible)

1) Runs a single BOINC meta-task for a fixed period, typically 24 hours. This task launches the virtual machine, and the VM fetches the tasks to be computed and returns the results via IP tunneling. This requires specialist project servers and schedulers, and if they run out of work, the computing resources can't be released to other projects and remain idle. I find this mode of working less satisfying to the interested, involved, volunteer.

2) The CERN infrastructure (the only one, AFAIK, supported by BOINC) relies on the Oracle VirtualBox technology. This is open source (i.e. free), but it doesn't - yet - support 'VGA passthrough': in other words, GPUs can't be utilised inside the virtual machine. That may be a showstopper for this project.

With the limitations Richard has noted, my personal experience is:
1) The volunteer must commit to running VMs when they create an account.
2) VMs run normal priority so they tend to crowd out other polite project tasks (and the user) if running near CPU or memory limits.
3) VMs, of course, reserve resources up front (particularly memory) and don't share it with the user.
4) VM's use resources just to build the VM. It would be nice if a constructed VM could be used over and over once allocated.

None the less, I've committed to running for CERN because I'm interested in their research.

I briefly crunched over at Atlas@home ("Atlas") with the BOINC Virtual Box ("VB") and IMHO like all software emulation packages, this has its limitations. The software emulation is good, however when you are dealing with very complex and/or demanding tasks it usually does not work as well and is subject to higher than normal errors and a certain amount of crunching frustration. Specifically with Atlas, one had to run those tasks with hyper-threading turned off, the CPU must be powerful, it was difficult to run another projects tasks side-by-side, tasks were prone to higher than normal failure rates especially when the task says it is at 100% completed and is still running for over 24 hours for an non hyper-threaded CPU (Intel 3930k, 16gig RAM running four (4) Atlas tasks concurrently. That one lasted over 48 hours before I aborted). This is besides what I believe is Atlas project very poor point yields for completed tasks that are unacceptably low (less than 20 points per hour and I pull the computer from the project. Others usually will have different standards, if any).

VB does sometimes work well, however software emulation just does not hold up well with very demanding and complex tasks as Atlas has proven. VB is well suited for more light work type of tasks. The best analogy I have is this, running VB (BOINC version) in any OS environment is almost the same as running a high definition native PS 4 graphic game (non-ported or optimized version for that OS) using a Windows OS (7,8, or 10) emulation software. Yes it will run, but not all that well and over time you may get very frustrated with the overall performance.

I personally will not run any BOINC VB task and have pointed out the risks to other cruncher's about these types of tasks, however their are many who will still accept and crunch these tasks for the sake of the science being generated.

I believe that Cosmology@Home has recently also added a VM version on to their project, running the original single thread CPU application along side the VM application, so they might be a useful to call.

Also RNAWorld has been using VMs for a long time now.

CPU only of course.

But although I have not run VMs before I also read (same as Richard and others) that the GPU is ignored by the VM, you can use all CPU cores (multi-thread) but not the GPU.

Conan

I have tried to crunch some VM projects without any success. VM client didn't receive any work due to LAN, modem and firewall configuration. Standard Boinc applications runs without any problem.

I have never had any success attempting to run projects which use a VM.

A CERN authored 2012 paper, "BOINC service for volunteer cloud computing" may be requested through researchgate.net which broadly outlines the integration of CernVM with BOINC:

https://www.researchgate.net/publication/258668690_BOINC_service_for_volunteer_cloud_computing

Regarding the general issues of hypervisors (VB or other) used for distributed computing,the drawbacks mentioned in the thread (GPU pass through, performance, resource usage and overhead etc.) might to some degree be addressed by using containers rather than full visualization hypervisors. I might be interesting to explore delivering the BOINC client as a container with GPU pass though.

I believe that Cosmology@Home has recently also added a VM version on to their project, running the original single thread CPU application along side the VM application, so they might be a useful to call.

Cosmology uses VM and
Docker

I will let others more wise in the ways of the VM environment speak to the technicalities. I will comment on being an end-user. I have a very low success rate of using VM for projects. They, more often than not, seem to hang with messages like failed to enter in a timely fashion, etc. I haven't seen much help in the forums. Every once in a while one runs to completion yet I do not know what is different in my computing environment. I do run a lot of projects so maybe it has something to do with the allocation of resources - but I am uncertain at this point. I usually abort and restart the programs. Sometimes I can get them to run (longer) if I exit BOINC and/or restart my PC but that is a little bothersome.

Thought I would go ahead and add my 2 cents worth.

The advantage of a virtualbox based application is that the researchers only have to make their application work with one guest operating system. CERN uses a version of Linux that they optimized for their type of scientific computing.

And here are the disadvantages that I see:

Running a task under an operating system on a virtual machine suffers about a 15% performance degradation.

For each task running, there has to be a copy of the virtualbox guest operating system and the scientific task loaded into memory. CERN tasks take between .5 GB and 2.5 GB memory for each task. Most typical PC's can only run one or two tasks at a time because of the memory requirements.

The BOINC task in the host operating system invokes a vboxwrapper that in turn initiates the guest operating system in virtualbox. Finding a working combination of the technology stack (host operating system, vboxwrapper, version of virtualbox, guest operating system and research task seems to be challenging. There seems to be quite a bit of time spent getting everything to work together again when any of the components of the technology stack upgrade their software to a new major release.

The VM application seems to suffer when the host hardware stops abruptly, for example a electrical outage. Often the BOINC task has trouble communicating with virtualbox to get the VM started up again. This quite often leads to aborted tasks and starting up another task.

Because of the challenges in getting everything to work together properly, the CERN projects that use VM get a low participation rate. The Test4theory (vLHC@home) project usually has between 500 and 700 participants per day. And because of the memory restrictions, most of those users are only running a few tasks at a time. I have 36 threads available in my crunching farm and usually run 3-4 VM tasks at a time.

Having said all that, I do run tasks for CERN. But I sure don't get it to run by being a first adopter whenever one of the components in the technology stack comes out with a new version. I wait until the researchers have some time to experiment and get everything to work together properly.

Hope that helps.

I do not like the tasks submitted to the virtual box,
sometime the task cannot start (syncro issue with the linux tasks).

I have two Haswell machines running Win7 64-bit and VirtualBox 5.0.10 (BOINC 7.6.21), and generally like it a lot. As captainjack mentioned, it allows the researchers to optimize their application on whatever version of Linux (or anything else I suppose) that they want, and it all works more or less the same on each client. Much of the hassles on each project involve trying to optimize the software for some particular OS/Hardware combination from what I can see.

I have one machine on CERN (Atlas, vLHC, CMS-dev) and have no problems. With the new wrapper for Atlas, my error rate is now zero, as it has been for the other projects from the beginning. However, you do need a LOT of memory in some cases, so don't go into it unprepared.

The other machine is on Cosmology, and as noted above, that is multi-threaded. That can cause some problems in sharing cores among projects, since Cosmology wants to grab them all. I can devote an entire machine to it (reserving a couple of CPU cores for GPU work), but most people will get rather annoyed at it and leave the project, so I don't really recommend it for most projects.

As for GPUs, I was under the impression that the new Version 5 of VBox allowed for that, but maybe it is not implemented yet. I think it is supposed to allow for SSE and AVX also, but have not seen that used yet. I would try it if possible.

http://www.cosmologyathome.org/forum_thread.php?id=7348&postid=20629

Works for me, Atlas and LHC run well, even on antique laptops, regards mikn

Hi GDF,

I'm running both Atlas and LHC@Home (CERN) as some people below do as well.
My experience with VM's is that they are very stable as also pointed out by Jim1348, once they are stabilized.

I've run them now for nearly 8 months nearly uninterrupted and did not encounter major issues so far. (knocking on ...)

A couple of drawbacks that I see are:
1) Consuming a lot of memory (Atlas is now running 5 simultaneous tasks and EACH task takes up between 5.7 GB and 7.2 GB) I got 32 GB of RAM so I'm still fine, but on other systems with lower memory, that could become fast prohibitive.

2) Atlas and LHC tend to be VERY disk intensive with up to 80 GB per day being written. I have raised the point multiple times, but without satisfactory answer.

3) As you're working with VM's, you need to reserve a core only for the VM ware (whether it's Oracle VirtualBox or VMWare, or whatever)
If you want to use your PC still, you'll need to take off another core for your applications as well as the BOINC Manager itself.

4) For each core, you'll also take a 15 % hit (depends on processor type & speed) on performance due to the emulation going on.

So, it definitely has it's advantages, but it also has some important drawbacks, so you'll need to carefully weigh your options and think of your target audience.

Hope this helps ?

Kind Regards,
K.

2) Atlas and LHC tend to be VERY disk intensive with up to 80 GB per day being written. I have raised the point multiple times, but without satisfactory answer.

Good point. I use a large (12 GB) write cache (PrimoCache) to protect my SSD, and it apparently helps to reduce the error rate also, and reads can come out of that cache also.

With a GPU project like GPUGrid, writes may not be a problem though.

As for GPUs, I was under the impression that the new Version 5 of VBox allowed for that, but maybe it is not implemented yet. I think it is supposed to allow for SSE and AVX also, but have not seen that used yet. I would try it if possible.

Searching https://www.virtualbox.org/wiki/Changelog for either VGA or passthrough doesn't reveal much. AVX and AVX2 have been disabled (at v5.0.2) and not yet restored. Webcams are supported. And that's about it.

Another problem with VM projects: They don't play nicely with Hyper-V. I cannot/will not run any Virtualbox project on my machines that are Hyper-V hosts. This means that I have one or two isolated computers (both laptops) that will crunch the occasional Vbox project. Furthermore, I have experienced a ridiculously high rate of problems (most of which are mentioned already in this thread) from those projects, so I often set them to "no new work" until I am bored enough to troubleshoot them.

Please don't go there.

There are projects giving out VMs instead of normal applications. We are considering it. How is it? Does it work?

thanks.

gdf

I've tried, but find it uses too much memory and is very CPU intensive, so quit. Some searches will verify this for you. Personally, I think GPUs is the way to go, because of efficient energy use.