Dear crunchers,

I'm starting this topic to collect information and feedback on the HERG workunits, all in a single place. The idea (under test) is to provide a quick-to-find reference for both those of you curious about the purpose of the WU they are crunching, and a place to report issues.

This post, and the one below, may be updated from time to time.


Scientific rationale.

First of all, some background information on the experiment: we are doing various studies on the so-called "hERG channel". You can find a (longish) description on Wikipedia's hERG page.
This complex of four proteins (tetramer) is found in many of the body cells, and most notably the heart tissue, where it plays a very important role: it conducts charged particles (potassium ions), which flow through it cyclically, ultimately governing the heart beat.

The molecule is of especial interest because interferences with its functioning, e.g. unintentional side effects of drugs, and congenital mutations, cause potentially fatal alterations in the cardiac rhythm, including the long QT syndrome.

The curious ones may find an image of the tetramer on our Flickr photostream.

Crunching issues.

The TONI_HERG workunits use the same parameters as many others. As far as we know, they have the same failure rate as other workunits, but I am trying to get some sounder statistics. If you see more HERG failures, it could be that there are many of those WU out right now.


[This post reserved for future updates]

http://www.gpugrid.net/forum_thread.php?id=1506

So, from what I understand, these WUs sometimes fail on older cards? I'm trying to collect statistics on non-overclocked cards.

From what I see in SKGiven's task list for host 51279, he had at least three TONI_HERG successfully completed, as well 1572466, 1606985 and 1558388. BTW, isn't the card overclocked at 1.85 GHz?

So, from what I understand, these WUs sometimes fail on older cards? I'm trying to collect statistics on non-overclocked cards.

From what I see in SKGiven's task list for host 51279, he had at least three TONI_HERG successfully completed, as well 1572466, 1606985 and 1558388. BTW, isn't the card overclocked at 1.85 GHz?

My GTS250 managed to complete one! http://www.gpugrid.net/result.php?resultid=1625604

The success percentage of these HERG tasks for anything less than a GTX260 seems to be poor, with the older cards being less reliable.

Just because an NVidia card is new does not mean there is any new technology inside!

We are keeping eyes on the failure rate wrt card types (in absence of overclock). As said, the matter is puzzling because there should be no major difference with other WU types. For now, I reduced the number of HERG WUs out, and possibly I'll reduce their length a bit in order to increase the chances of correct termination.

Almost all of the failures seem to be related to the infamous CUDA FFT bug, on which we have little to no control (i.e., errors in "pme" or "fft" kernels).

Definitely, thanks for bearing with us.

It's a long standing issue that hits older cards especially hard. Please see here or here. For what concerns FFT being ok with SETI, in fact there are many types of FFT, and it's not surprising that the bug only manifests for some of them.

Perhaps the FFT bug is being compounded by a mixture of G92/65nm cores and old firmware?

Reducing the work length would help, as the tasks that failed on my systems seemed to do so randomly, in terms of time. If they fail after 10sec its not really a problem that effects turnover, but after 6h is not good.

Ultimately if you could match cards to work units it would resolve this issue. It might even be better than card pairing, though both could be done.

No hERG tasks for G92 cards would soon sort a lot of problems out.

I can just repeat what I have already said somewhere in the forum.
We have furnished a reproducer of the bug to Nvidia. We contacted them back several times. They say that there they are looking at it. Another time, they said that technical stuff is trying to find the problem and the are discussions on what to do. But then nothing. This is common with Nvidia, we have sent several bug reproducers but they only fixed once another other bug with their FFT which we have sent. In my experience, they use bug reports to fix bugs on new chips not older ones. It also makes some sense given the rate at which new GPUs are produced. So we have stopped reporting bugs for older cards.


GDF

Had two TONI_HERG's fail. They were run on a GTX295 (single PCB variety, so the newer model).

WU 1
WU 2

Both say "Cuda error: Kernel [pme_fill_charges_overflow] failed in file 'fillcharges.cu' in line 97 : unknown error".

I know there isn't much you can do if nvidia don't want to fix their software.
____________
BOINC blog

Jari Pyyluoma,
I broadly agree with your concerns; I too am annoyed by wasting some of my efforts:

Although my GTS250 has recently faired a bit better (RAC 8255), largely due to the changes made by the Techs, my 8800GTS 512MB is all but lost – RAC is about 900!
I know it is old kit, but there are lots of people that have old kit.

However, on a positive note, my recent reviewing has shown that some of the new cards, although not magnificent, can still contribute substantially.
The GT 240 in particular is a worthy card. Very reliable.
So I would suggest to anyone that wants to continue participating, Sell your old cards and buy a new one.
A GT 240's can be purchased from between £60 an £80. The running costs are about one third of top end Compute Capable 1.1 cards, so over 6months crunching you will save:

If you run a 9800 GTX for 6 months the running cost = 180W * £1.20 per Year Watt * ½ year = £108
Sell your card for £25 minimum. Buy a new GT 240 for £65 and you spend a total of £40
Run a GT 240 for 6 months = 60W * £1.20 per Year Watt / 2 = £36
Total cost of buying a new card and running it = £36 + £40 = £76
So, over 6months crunching time you would save £108 - £76 = £32

Over a year of crunching that works out at a saving of £216-36-36-40 = £104
Oh, plus you get a better card!

From a network managers point of view, the fact that you would break even within 4months is Very attractive!

Under full load a GT 240 will use about 50 or 60 Watts and give you about 6900points per day. About 125 Points per Watt day.
Under full load a GTS 250 will use about 184Watts and give you about 8250points per day (perhaps partially due to failures). About 44 Points per Watt day!
Even my GTX 260 sp216 (55nm version) only gets 14000points per day, and eats up about the same Watts as a GTS 250. About 76 Points per Watt day.
Given that Three GT 240 cards would use less electric than One GTS 250 and do more than twice the work, these cards are very efficient!

In terms of Points per Watt, the GT 240 IS BY FAR the most efficient card available to GPUGrid supporters!

It will also do TEN times the work of an overclocked i7-920 running at 3.8GHz and using over 300W. It is a NO BRAINER!

Ref:
http://benchmarkreviews.com/index.php?option=com_content&task=view&id=423&Itemid=72&limit=1&limitstart=11
http://www.guru3d.com/article/msi-geforce-gt-240-review-test/5
My Stats!

Sony apparently stopped allowing the use of Linux on the PS3, required to run GPUGrid. Lots of G92 cards are struggling due to a number of things outside the control of the project, including the reliance on NVidia for code. If there is a bug in their code and the project team are not allowed to correct it, there is nothing they can do. If the project team was larger I am sure they would be better equipped to make more changes, perhaps even write tasks for the older cards, but as is things appear to be tight.
This is a good project and needs support. To me it makes sense to sell on old parts and replace them with new parts that are better and actually work well.

I agree that manually aborting WUs should not be necessary. In any case, BOINC does not currently foresee a mechanism for letting people chose WUs. As already said, some classes of WUs have higher probability of triggering bugs in some cards, but to the best of our knowledge this is not as simple as fixing a bug in our code.

We are working on the problem, of course; in the meantime, I've suspended the generation of the last HERG workunits (most of them were stopped before Christmas), even though this is not really a "solution": any bugs it triggers will not go away.

I'm sending a few more workunits of the HERG type. In the meanwhile, if you want to see images of what you are crunching, have a look at the flickr page!

Well, bugs may come and go with cards, drivers, and versions of the application. We try to make all WUs run equally well, rather than fork (and maintain) separate queues.

As soon as the new application will be available, I'll migrate the workunits to it, crossing fingers it will improve the situation.

Feel free to see the new molecular images on flickr...

The new HERGqext are out (note the middle "q"). I'm trying a variation of the FFT parameters, using a slightly longer computation than necessary, to see if they run more stably on more cards. Thanks for your support and patience...

Which of the two hosts?

BTW, for those of you crunching beta, the L*_TONI_TEST WUs are the same as the HERG and HERGext ones.

Does not seem to be HERG specific, you also had an error on task 1816245 of another batch.

Good!

Thanks, well spotted. I tried to replace a parameter on the fly. Please post if it happens again.

I also noticed the increase, and that was higher than expected. This is what I was trying to fix...

The new ones should be back to the norm.

I think the only way round this sort of problem is for the server to identify cards abilities to complete the various types of work unit and allocate accordingly. If there is more than say a 25% chance of failure then dont allocate the task, unless there are no other tasks.

This bad one seems to have been created by some file transfer error. It should fail immediately.

A GTS250 is very similar (almost identical) to a 9800 GTX+
So it is probably not that important, unless you are getting lots of failures. The half a 1GB vs 500MB does not make any difference here.