Hi all,

I know a few BOINC projects deploy apps in virtual machines. Are any of you crunching for any of those? Would be obliged if you could you give me a quick run-down of the common problems, please.

Cheers

Matt

I have been crunching on Test4Theory for just over 2 years. It is a project that does work for CERN and the large hadron collider.

The theory is that by using a virtual machine, they are only required to get their application to work on one operating system.

The setup requires a host operating system (either Windows, OSX or Linux), Virtualbox, a guest operating system (CERN uses their own flavor of LINUX), BOINC, and the scientific application. And that my friend is an awful lot of moving parts to try to keep working together.

Any time there is an upgrade in any of the components, there is a chance of compatibility problems. Most of the time, problems occur when Oracle "improves" Virtualbox and the "improvements" break something in the BOINC -> Virtualbox -> guest O/S interface.

Another area of problems is getting permissions set up correctly so that Virtualbox and the guest O/S have the permissions that they need to function properly, especially in Linux. That may have improved some with the latest release of the Virtualbox wrapper, but caution is still advised.

If you cruise on over to the Test4Theory web site, you will see that they have a relatively low level of participation. It is probably because it is challenging to set up the BOINC/Virtualbox environment and keep up with all the upgrades and problem resolutions.

You might want to spend some time browsing the message boards at Test4Theory and get a feel for they challenges that they face.

If you want more information from me, let me know.

Hope that helps,
CaptianJack

The Test4Theory project has been one of the projects at the forefront of developing the "app in a VM" approach. Working closely with Rom Walton and Oracle, they've eliminated most of the technical problems though I think they haven't quite finished recompiling and bundling all the fixes together and releasing a new vboxwrapper. Jacob Klein is more familiar with that than I am so I'll leave that topic for him. I would rather describe to you what I think has been the source of past problems and may well be the source of future problems.

In the traditional model BOINC communicates with and controls the project app directly. By "app control" I mean suspending, resuming, stopping, starting the app. When the app is in a VM the traditional BOINC <-> app comms lines don't exist because the app is sandboxed in the VM. App control is achieved through the vboxwrapper developed mostly by Rom Walton. Vboxwrapper calls vbox-manager (a convenience app Oracle ships with the VBox package) with appropriate parms, vbox-manager controls the VM, and that is mostly where past problems have occurred. Bugs in either vbox-manager or the VM itself or perhaps Rom's use of vbox-manager, have caused sporadic comms "disconnects" between BOINC client and vbox-manager which have resulted in, among other things, the VM not stopping/suspending when it should. As I mentioned above, that problem as well as others appear to have been eliminated recently. There may be other problems I am not aware of.

Today the main deficiency, though it may never cause a problem that becomes apparent to the user, is that controlling the VM via vbox-manager is like controlling BOINC client via boinccmd. It works but it's clunky, for lack of a technical term. Oracle has developed an API for the control interface which vboxwrapper could use (but does not) to control the VM directly. Perhaps one day BOINC client itself will use the API and eliminate the wrapper.

---

BOINC <<--- credit whores, pedants, alien hunters

Here is another factor to consider, not a problem per se, but just something to think about. When running a BOINC task under a VM, each task that runs requires a virtual machine, a copy of the guest O/S, a copy of the scientific application and all the data that is required to perform all the calculations. That all amounts to quite a bit of memory that is required to run a task.

The Test4Theory task that I am running right now in Linux is using 348MiB of memory.

Test4Theory is currently set up to only run 1 task per computer. If a project/person wants to run more than one task per machine, memory requirements will go up proportionally.

Thanks guys, all useful info.

On a practical level, do I take it that the user needs to download and install Virtualbox separately, or is there a combined BOINC+VM installer?

http://boinc.berkeley.edu/download_all.php

Read the BOINC v7.2.28 has been released! thread.

So far as I'm aware, there are no GPU projects using a VM.

If you google Boinc, VM and VirtualBox you should find some details, including publications/development work.

- RNA World also have an app; cmsearch VM (VirtualBox) 1.0.2
They use a 782MB virtual disk image (vdi), rnaWorld2GB_x64_2.vdi
and the wrapper, vboxwrapper_26028_windows_x86_64.
RNA World use the VM as it allows them to checkpoint (in a way). Some/most of their tasks take months (~4moths on high end processors).
Problems include the VM not being removed when a WU fails (would use up hard drive space).

http://www.rechenkraft.net/phpBB/viewtopic.php?f=75&t=13127

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Problems include the VM not being removed when a WU fails (would use up hard drive space).

That has been a problem at Test4Theory (T4T) as well. It's been a while since I played with vbox-manager but if memory serves correctly it is possible to query vbox-manager to see if there is any garbage laying about and follow with commands to clean it up. If devs at projects employing VBox are not doing that then it might be out of concern they might receive bad intel from the query and delete some other VM running on the host.

As far as VBox not virtualizing the GPU... maybe someone needs to ask Oracle if they could add that capability. Maybe it's just a matter of Oracle not perceiving a demand for it? If so then perhaps that perception can be changed?

---

BOINC <<--- credit whores, pedants, alien hunters

I'll chime in.

I've only been doing VM tasks for about 3-4 weeks. I wish I had been doing it earlier, since I'm a BOINC alpha tester (and could have helped find/fix the problems), but alas, for whatever reason, I didn't. But now, I am doing work for Test4Theory, RNA World, and Climate@Home (when they have work).

The main problems I've seen have been:

1) Oracle problems resuming from snapshots. 4.2.16 was great, and didn't have problems, but 4.2.18 introduced a bug, 4.3.0 fixed it but introduced another, and 4.3.2 [current] fixed it but introduced another. The most recent unfixed bug does not affect RNA World, but does affect T4T. Details can be found at these 2 links:
http://lhcathome2.cern.ch/test4theory/forum_thread.php?id=1364&postid=15560
https://www.virtualbox.org/ticket/12291
I expect Oracle to get this resolved in the next maintenance release, so VirtualBox itself doesn't fail BOINC tasks like it does now when resuming T4T tasks.

2) Any BOINC applications that use the VM, go through that vboxwrapper. It too has had some bugs that I've worked with Rom to solve. Most recently, we found a heartbeat bug where tasks for applications compiled with API code within the past few months, when ran on Windows service installs, would simply fail due to a problem with the heartbeat check. It's fixed now, but it requires a newer vboxwrapper, which means a newer application version. Also, there were some issues with trickle messages not working properly, but I believe that's fixed too. Long story short: The vboxwrapper is not quite matured yet, but it is maturing, and I'd recommend any application that uses it to have a team that is "nimble" enough to easily update to newer application versions.

3) Communication/control doesn't always work. It works almost all the time, but during BOINC upgrades and a few other scenarios, the VMs are not paused appropriately. Then when BOINC later tries to "attain a lock" to control it, the lock can fail, and it causes problems. Rom is trying to get better control on this with the wrapper, but it is an ongoing issue that does occur.

4) VirtualBox being installed. This is a problem for some. For me, I encounter network problems during LAN parties, whenever VirtualBox is installed (because it does something with some "VirtualBox Host-Only Network" network adapter that can cause problems when gaming or sharing files.) In general, it works just fine, but it does monkey with the network, and that right there is a reason that I don't install it on my wife's laptop, because I cannot risk her having any sort of crazy network issue when she's away at the office doing work. And so that's a "showstopper" for some machines/users.

5) VM Management. If you are in a non-service-install, it's neat that you can open the "VirtualBox Manager" program to see BOINC managing (pausing, snapshotting, resuming, shutting off) your VMs. However, for service-installs, you cannot. So, you lose out on seeing that info. Not a big deal, but it is a problem, in the sense that the user can't ever manually see/control/cleanup the VMs. Note: T4T actually uses the "Remote Display" capabilities of VirtualBox, and the BOINC client has support for it too, such that in BOINC I can click a "Show VM Console" button, that uses Window Remote Desktop to actually connect to the VM and show me some text. It's neat. :)

6) Cleanup. Most times, when a task fails, BOINC uses the vboxwrapper to clean the slot up. But sometimes it doesn't do that. (I haven't thoroughly tested) But Rom told me that, the next time that application runs in that slot, it'll do a cleanup before copying data into the slot. I find that a bit sloppy, especially since we're dealing with big files, but it is what it is.

7) Memory. The RNA World uses a 64-bit VM with a Base Memory of 4 GB. So, BOINC will make sure your system is allowed to use 4 GB before giving you the task. But, while that task is running, because Windows doesn't have an effective way to see how much memory the VM is using, and because the VirtualBox manual states that the host shall grant the Base Memory amount as free memory and not expect to use any of it... We recently put in a client fix (made it into 7.2.28) where, when any VM app is running, the <rsc_memory_bound> value will be used as the "this amount of memory is allocated for this running app" value. Non-VM apps still query the Windows processing list to get that information. So... If you want to run an RNA World app, BOINC [correctly] considers 4 GB memory as "used", even if the client OS is using a fraction of that. Thus, it's crucial to set "Base Memory" to an appropriate value, and to make sure <rsc_memory_bound> matches it, and for those projects that "might use x amount", BOINC will consider "x amount used" while the task is running.

8) Host responsiveness. I have found that, when running 2 VMs (that use the "Enable VT-x/ADM-v" hardware acceleration option) at the same time, my music sometimes skips, my mouse cursor's acceleration isn't always registered properly, so my cursor skips around when I move it, the UI sometimes refreshes panes/grids slowly, and my machine is difficult to control. We're still looking into this issue, but I'm pretty sure it's related to that hardware acceleration option, and perhaps Intel doesn't handle multiple VMs too well on my 2009 i7 965XE CPU. But it is an issue. So, I make sure to only run 1 VM at a time, by manually suspending tasks, until I go to bed, and then I run multiples.

Whew, I'm beat. I'm sure there's more "problems" out there, but that's all I can think of off the top of my head. I'm still cutting my teeth on this VM stuff, but am pressing T4T admins, RNA World admins, and Oracle devs, to all fix the bugs so it all runs smoother. You'll see my name on a lot of posts, mainly because I care. Eventually, BOINC devs may ONLY offer the "BOINC+VirtualBox" installer; they want users to be capable of running these new VirtualBox tasks. And I get that. But, at the same time, we're still working out the kinks.

By the way, I should mention that, for RNA World, their VirtualBox application is a blessing! I restart my machines often, and so for the longest time (years?), I refused to run their tasks, because they could take 2-60 days, with no checkpointing! But that issue is solved! So, right now, I've got one of their VM tasks that is estimated to take 60 days, I've been working on it for 3 weeks, I'm on "CPU-processing-day 11", and so far it is working well! I've restarted numerous times, and it resumes form snapshot just fine. Here's a little more info on how it works: I believe that, if the user has "save to disk" at a value that is less than 10 minutes, vboxwrapper takes a snapshot every 10 minutes (since a lot of data may need to be saved to store the snapshot which is a "dynamic difference disk"). So, any time I need to restart my PC, I only lose up-to-10 minutes of work. That's AWESOME! That means I can actually finish this task that is estimated to take 8 weeks! (I'm excited).

Hope you find this useful!
- Jacob

Jacob Klein wrote:

8) Host responsiveness. I have found that, when running 2 VMs (that use the "Enable VT-x/ADM-v" hardware acceleration option) at the same time, my music sometimes skips, my mouse cursor's acceleration isn't always registered properly, so my cursor skips around when I move it, the UI sometimes refreshes panes/grids slowly, and my machine is difficult to control. We're still looking into this issue, but I'm pretty sure it's related to that hardware acceleration option, and perhaps Intel doesn't handle multiple VMs too well on my 2009 i7 965XE CPU. But it is an issue. So, I make sure to only run 1 VM at a time, by manually suspending tasks, until I go to bed, and then I run multiples.

This not a shameless Linux plug. My intention is more to applaud VirtualBox than to praise Linux. If Windows were as configurable as Linux I believe one could get results on Windows just as impressive as I am about to describe.

On my i7-2600K, no OC, with VT-x enabled, I was able to run simultaneaously:

- the 1 VM associated with the T4T BOINC task
- plus 6 other VMs I created, each with BOINC installed and running T4T which of course means a T4T VM in each of those VMs (nested VMs)

That's a total of 13 VMs running simultaneously, all Linux. That was with the old cernwrapper which was as bare bones as it could be, less than 200 lines of C, IIRC. It didn't do any snapshots which reduces load on vbox-manager somewhat... but... the system ran almost flawlessly, with very little cursor jerking and slow acceleration. VMs started, stopped, paused and resumed flawlessly, only 1 in about 30 VMs had to be cleaned up manually. I could interrupt any of the VMs at any time and it would resume no problem. It survived power outages with no problem. Then I turned off all the eye candy in the desktop and switched the OS installed in the first tier VMs from 64-bit to 32-bit and that restored responsiveness to normal. Oh, I adjusted some process priorities (nice levels) too and that actually helped a lot. After I had it all tuned she was a beast of rare beauty! I was thoroughly amazed and very impressed with VirtualBox.

Rom's vboxwrapper changed all that for me. As the size and complexity of the wrapper grew, the performance of my i7 system with 13 VMs running deteriorated considerably and I had to cut back to just the 1 VM running on the real machine. No ill will toward Rom as I was always amazed that I could run even 3 VMs simultaneously let alone 6 nested VMs plus a normal one. I always considered it flying too close to the sun and new I would plummet back to Earth one day.

On the positive side, and this is a big positive, Rom's wrapper fixed a lot of problems for Windows users, problems that made it impossible for quite a few of them to run T4T.

---

BOINC <<--- credit whores, pedants, alien hunters

Just out of curiousity, when you were running multiple VMs, are you ABSOLUTELY POSITIVE that multiple running VMs had the "Enable VT-x/ADM-v" option checked? The reason I ask is that, right now, T4T uses a version of the wrapper where it leaves that option unchecked. (T4T uses a single-CPU 32-bit VM, and the old version of the wrapper left the option unchecked in that scenario, and on side note, it suffers from the server-install heartbeat problem as well. T4T admins are looking to upgrade to latest wrapper in beta, sometime soon.)

... and I've only noticed the OS responsiveness issues when multiple tasks with "Enable VT-x/ADM-v" were running. (For T4T, I'm currently using anonymous platform with the newest version of the wrapper, for testing purposes, which checks/enables that option even for single-CPU 32-bit tasks, which is why I'm having the issue currently.)

So, again... are you sure your T4T tasks were using it?

Jacob,

Thanks very much for taking the time to write such a comprehensive summary. It's very useful!
One question - am I right to understand that the VM image is supplied in the same way that the application is for a standard BOINC project? That's to say that it is downloaded once and re-used and shared for all WUs?

Matt

So, again... are you sure your T4T tasks were using it?

Yes, they were. That was over a year and several VBox updates ago so keep that in mind.

---

BOINC <<--- credit whores, pedants, alien hunters

am I right to understand that the VM image is supplied in the same way that the application is for a standard BOINC project? That's to say that it is downloaded once and re-used and shared for all WUs?

Yes.

---

BOINC <<--- credit whores, pedants, alien hunters

Jacob,

Thanks very much for taking the time to write such a comprehensive summary. It's very useful!
One question - am I right to understand that the VM image is supplied in the same way that the application is for a standard BOINC project? That's to say that it is downloaded once and re-used and shared for all WUs?

Matt

You're welcome.
Regarding the VM image, yes, I believe that is the case; it is only downloaded once.
My turn to ask a question: Why are you gathering this information - do you have any sort of plans for a VM app or maybe a new VM project?

Matt said he wanted to make a VM image weeks ago...

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Ah, thanks, I must have missed that.

I assume this is meant for a possible multi core app. But: if the GPU could be used under Virtual Box without a performance hit, this could provide a much more stable GPU-Grid environment! You could reduce the software complexity to just one driver under one OS, which could help a lot. Or, if running virtualized caused a performance hit (which I expect to happen, if possible at all), this could be used as an optional "reference platform", which people could use when in doubt about their system and software, or when they can't get it to work otherwise.

MrS

---

Scanning for our furry friends since Jan 2002

A lot of projects don't behave themselves when it comes to task settings (priority, GFlops estimate, estimated runtime, deadline...). So when work arrives for projects like Physics, GPU work stops and Physics tasks run on the CPU. I find this very annoying to say the least. This is one reason I use Linux virtual images to run CPU work, but I install Boinc within the VM.

Last time I tried to get the drivers to work for my GPU's I forgot I didn't have the monitor plugged in, and ended up at a black screen with an X for an arrow. It might be easier to go headless but I might have to build an image outside of the VM and then import it, some how. Next time I try the drivers I'll remember to clone first and plug in a monitor!

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Last time I tried to get the drivers to work for my GPU's I forgot I didn't have the monitor plugged in, and ended up at a black screen with an X for an arrow. It might be easier to go headless but I might have to build an image outside of the VM and then import it, some how. Next time I try the drivers I'll remember to clone first and plug in a monitor!

I've never tried headless but have heard others discussing it. For headless I believe you need to install a virtual monitor to fool X into thinking a monitor is attached, otherwise X won't boot. I believe X initializes the driver so if X doesn't boot, BOINC client won't see the GPU.

I still can't get BOINC to see the GPU in virtual Ubuntu. The driver installer definitely checks to see if a supported GPU is installed but even if there is no GPU it allows you to install the driver anyway. Even after an OS reboot, BOINC client doesn't see the GPU. Now I really am out of ideas so it's time to Google and ask for help in nvidia and vbox forums.

---

BOINC <<--- credit whores, pedants, alien hunters

You can force X to start even in the absence of an attached monitor by adding:

Option "ConnectedMonitor" "crt"

to the Device section of the xorg.conf file.

But you don't even need to be running X. You can simply load the nvidia module with "sudo modprobe nvidia", then run the BOINC command line client.

Matt