Get your votes in early ...

I just installed an ATI HD4870 to match my existing HD4870 though the memory sizes are different ... I posted a note regarding the way BOINC reports this. And Rom replied thusly:

Asymmetric GPUs from the same vendor is currently not supported.

The reason both or your GPUs are showing up instead of just the most
capable is because you have the <use_all_gpus/> flag specified in your
cc_config.xml file.

At this time we do not know if/when we'll support the Asymmetric GPU
from the same vendor scenario.

I suggested that this is a bad idea:

You are going to have to eventually for the simple reason that people, particularly for the wider rigs are not going to be able to upgrade 3-4 GPUs necessarily at the same time...

Even worse, models are dropped and in essence you are going to be telling people that when they want to upgrade and fill a slot and the old model is no longer available that they have to junk a perfectly good card? Not a good move ...

Understand clearly what this means, if you have a 3 GPU system their expectation is that all three will be same make, model, and internals... In that GPU cards in the same model line don't always stay the same from one year to the next this is an impossible standard to meet ... unless you have way more money than I and can afford to buy your GPU sets all on the same day ...

I pointed out that this asymmetry was inevitable last year, that GPU installations are more than likely going to be asymmetric than not for many people ... but UCB in cloud land is going to expect that if you want to run GTX295 cards or HD5870s you are going to buy them in batch lots ... I don't know about you ... but I don't always have $1,500-$2,000 to spend all in one swell foop ...

Of course, you can just let the least capable, that is all those GPUs that don't match the top one, sit idle ... won't that be fun ...

Anyway, please make your objections known on the Alpha list or you are going to be stuck ... forewarned ...

Anyway, please make your objections known on the Alpha list or you are going to be stuck ... forewarned ...

I can't speak for an ATI config, but with nVidia cards it's not quite as simple as just enabling this functionality as anyone who ever tried to fold with a 200 series card and a previous gen card (in the same mb) will tell you. Some of the issues have been resolved in the latest series drivers, but I still don't think it is a case of enable by default.

It's one thing having 2 cards with the same GPU but one has more memory on board, quite another when the shader count differs. This functionality should not be enabled (for Linux BOINC and nVidia GPU's at least) until the current stable driver release is >= 190! 185.xx and CUDA 2.2, I guarantee problems with non asymmetrical hardware on the same mb. At best, the fastest card will run as slow as the slowest card, at worse both will error out when you start a CUDA kernel on them. By all means vote, but know what you're voting for! ;)

Sorry Paul, it always seems like I'm trying to rain on your parade ......

---

Crunching on Linux: Fedora 11 x86_64 / nVidia 185.18.36 driver / CUDA 2.2

Jack,

I love the rain ...

I will agree there are points at which it may be impractical. I stress MAY! But, in that the basic model of BOINC is one task per processing unit (with some coming exceptions) I am not at all convinced that even your example is not a problem.

I ran a GTX280 and 9800GT with a factor of 3 difference in speed side by side for most of last year with no issues. I know UCB does not want to touch this, heck what issue do they want to touch ... but ... with the one task per GPU I will be honest with you ... I cannot conceive of why it should be an issue. The tasks on the 9800 took 17 plus hours to run and on the 280 about 6 ... so I have had a different experience than you ... and with at least three driver "generations" because I have had several cycles of upgrades (though the 9800 is now on the sidelines) ...

I know that their option is easier ... but if it were easy anyone could play ...

Though the accusation is that I think only I know what is best ... the truth is that I want the best overall options for all ... I think that the draconian policies adopted by UCB are bad and this is just another case where the cure may be worse than the problem ... so, by all means raise your points ... and be sure to also rebut me on the mailing list or UCB will not know that you support their position ... just be sure that you want to never be able to run cards with any differences between them in your rigs ... because that is what their rule is ... no differences of any kind ... and good luck finding that exact card a year or two from now ... :)

{edit}

Last note, silence on this matter UCB will take as consent ... or that no one objects, besides me, to this policy ... stay silent now and you will have to live with it the next time you want to upgrade your GPUs ...

Paul,

Unfortunately, I know more than I can say. As I mentioned in another thread, I'm under a NDA, which is why I gave a (third hand) folding example, rather than first hand experience.

There are/were 2 issues.

Mismatched shader count. (Which caused an issue even on same gen hardware, eg. between 2x GTX260's in the same machine. ie. 192 shader count model and later revision with 216 shader count. At best 216 shader count 260 would run at lower shader count card speed, at worst unexplained errors.)

Mixing hardware with different compute capability. 200 series and 1st CUDA gen 8800 is a classic failure example.

---

Crunching on Linux: Fedora 11 x86_64 / nVidia 185.18.36 driver / CUDA 2.2

Paul,

Unfortunately, I know more than I can say. As I mentioned in another thread, I'm under a NDA, which is why I gave a (third hand) folding example, rather than first hand experience.

There are/were 2 issues.

Mismatched shader count. (Which caused an issue even on same gen hardware, eg. between 2x GTX260's in the same machine. ie. 192 shader count model and later revision with 216 shader count. At best 216 shader count 260 would run at lower shader count card speed, at worst unexplained errors.)

Mixing hardware with different compute capability. 200 series and 1st CUDA gen 8800 is a classic failure example.

And I am not asking for you to violate the NDA ...

But when Rom indicates that the plan and the code they are putting in place is intended to support one ATI card and one Nvidia card and or some other combinations (not sure if they would support one ATI card with a pair of GTX260s or not) ... but if that is not supporting mismatches I don't know what is ...

But when Rom indicates that the plan and the code they are putting in place is intended to support one ATI card and one Nvidia card and or some other combinations (not sure if they would support one ATI card with a pair of GTX260s or not) ... but if that is not supporting mismatches I don't know what is ...

Where is this discussion taking place, the dev list?

---

Crunching on Linux: Fedora 11 x86_64 / nVidia 185.18.36 driver / CUDA 2.2

But when Rom indicates that the plan and the code they are putting in place is intended to support one ATI card and one Nvidia card and or some other combinations (not sure if they would support one ATI card with a pair of GTX260s or not) ... but if that is not supporting mismatches I don't know what is ...

Where is this discussion taking place, the dev list?

boinc_alpha, thread "Memory, oh the memories"

boinc_alpha, thread "Memory, oh the memories"

Thanks. I had a quick squint at the dev list archive but I couldn't find anything relevant. ;)

---

Crunching on Linux: Fedora 11 x86_64 / nVidia 185.18.36 driver / CUDA 2.2

Get your votes in early ...

I just installed an ATI HD4870 to match my existing HD4870 though the memory sizes are different ... I posted a note regarding the way BOINC reports this. And Rom replied thusly:

Asymmetric GPUs from the same vendor is currently not supported.

The reason both or your GPUs are showing up instead of just the most
capable is because you have the <use_all_gpus/> flag specified in your
cc_config.xml file.

At this time we do not know if/when we'll support the Asymmetric GPU
from the same vendor scenario.

I suggested that this is a bad idea:

You are going to have to eventually for the simple reason that people, particularly for the wider rigs are not going to be able to upgrade 3-4 GPUs necessarily at the same time...

Even worse, models are dropped and in essence you are going to be telling people that when they want to upgrade and fill a slot and the old model is no longer available that they have to junk a perfectly good card? Not a good move ...

Understand clearly what this means, if you have a 3 GPU system their expectation is that all three will be same make, model, and internals... In that GPU cards in the same model line don't always stay the same from one year to the next this is an impossible standard to meet ... unless you have way more money than I and can afford to buy your GPU sets all on the same day ...

I pointed out that this asymmetry was inevitable last year, that GPU installations are more than likely going to be asymmetric than not for many people ... but UCB in cloud land is going to expect that if you want to run GTX295 cards or HD5870s you are going to buy them in batch lots ... I don't know about you ... but I don't always have $1,500-$2,000 to spend all in one swell foop ...

Of course, you can just let the least capable, that is all those GPUs that don't match the top one, sit idle ... won't that be fun ...

Anyway, please make your objections known on the Alpha list or you are going to be stuck ... forewarned ...

I usually agree with you, but not sure on this issue.

You typically would not put two dissimilar CPUs in a dual-socket mobo and I think SLI wants to see two virtually identical GPUs.

"They" have enough problems fixing BOINC for simple systems.

I usually agree with you, but not sure on this issue.

You typically would not put two dissimilar CPUs in a dual-socket mobo and I think SLI wants to see two virtually identical GPUs.

"They" have enough problems fixing BOINC for simple systems.

Well, Rom's explanation of the intention it is to support Nvida / ATI mixes and I cannot see how you can get more complex than that or more dissimilar in GPU architecture ...

I saw that on the test msgs.

Still doesn't make sense to me to try for a more complex integration when "they" haven't got it right with two of the same.

In my mind it is better to limit configuration possibilities. I mean we don't expect mobo mfgs to accommodate a RAM config of three disparate types of memory chips.........you use two or three of the same chip for memory channels. I haven't seen any recent autos that allow a motorcycle tire and a tractor tire on one side of the car and a pair of airplane landing tires on the other side and it doesn't make sense to try.

So far my installs are just single GPUs and I'd like to see the efforts expended towards getting the basic CPU/GPU config working first.

I saw that on the test msgs.

Still doesn't make sense to me to try for a more complex integration when "they" haven't got it right with two of the same.

Well, there is this failure of imagination ...

Knowing the history of GPUs over time especially when multiple GPUs came out was that people upgrade a little bit at a time. In the VERY early days GPU cards more often than not as an upgrade replaced a GPU on the MB that was just enough to draw dots on the screen.

Hi Paul,

It would be a nice to have, but as the other guys have said I think they need to get things working properly to start with, before we try and handle different cards from the same vendor.

My personal opinion is they should treat gpu tasks the same as cpu tasks and swap out as needed, honour TSI and so on. We haven't got to that point yet.

---

BOINC blog

It would be a nice to have, but as the other guys have said I think they need to get things working properly to start with, before we try and handle different cards from the same vendor.

A dreamer ... we have a dreamer on aisle 5! :)

My personal opinion is they should treat gpu tasks the same as cpu tasks and swap out as needed, honour TSI and so on. We haven't got to that point yet.

Sadly, the history is that operational experience does not color design or implementation much ... and sometimes when it does it hatches new major issues like Strict FIFO breaks Resource Share with MW because of its restricted queue rule.

My point is that they are saying, in effect, that they will be supporting the more difficult case first (getting the two different cards working correctly has been, historically, a far more difficult proposition - ATI and Nvidia have for years assumed that you want only their product and so make little allowance to run them together - that said, some have managed to get the cards to work together - I have not for what that is worth). But that is my point, if they think BOINC can support GPUs that are so different that they use different drivers, architectures, and programming models; then why is supporting cards from the same MFGR with minor differences so out of the question ...

That is the joke, they are saying that the ATI / Nvidia mode is supported but not two cards from the same MFGR with a difference in memory size ... it is looney toons ...

That is the joke, they are saying that the ATI / Nvidia mode is supported but not two cards from the same MFGR with a difference in memory size ... it is looney toons ...

Paul, you do realize that the cuda_compare function already has a test that allows 30% memory size difference between otherwise identical cards?

    if (loose) {
        if (c1.prop.totalGlobalMem > 1.4*c2.prop.totalGlobalMem) return 1;
        if (c1.prop.totalGlobalMem < .7* c2.prop.totalGlobalMem) return -1;
        return 0;
    }

If you don't like the (0.7/1.4) multipliers, change them and recompile! ;)

Alternatively, why not just set the 'use_all_gpus' config option?

---

Crunching on Linux: Fedora 11 x86_64 / nVidia 185.18.36 driver / CUDA 2.2

That is the joke, they are saying that the ATI / Nvidia mode is supported but not two cards from the same MFGR with a difference in memory size ... it is looney toons ...

BOINC already automatically handles a 30% memory size difference if using the same type of card. Same card with more extreme memory differences are already supported with the <use_all_gpus> flag. Where you run into issues are where the cards are more different (eg shaders/chip).

Lets explore this one shall we. To support same brand cards of different types we'd (roughly) need BOINC to:

1. Maintain array with card details (mem/shaders/compute capability etc)
2. Need to tweak scheduler to split work to the appropiate card
3. Tweak scheduler to maintain a benchmark/estimated speed so we can tell in relative terms how fast/slow it is (needed for work fetch/est run times)
4. Server side changes (for work requests, display of details, etc)
5. Some ability to exclude the card from boinc using it (optional)

As you can see its more work than simply cloning the existing code for Nvidia cards to their ATI equivilivent, which is basically what has been done to get ATI cards added.

Also consider that ATI has around a 50% market share (or whatever percentage it really is) to Nvidia in the graphics card arena. To include the other 50% makes sense as you are effectively doubling your capability as you now have both brands of gpu supported instead of just one. Adding support for mixed cards doesn't double the capability.

Thats not to say it won't happen. I think it will given time.

---

BOINC blog

I am aware of the "Use All" flag and use it now.

I am aware of the potential issues of having to schedule the resources separately. In fact, LAST YEAR I was warning UCB that this was an issue ... another case where I was told that it was not a problem. WHen small issues came up because UCB wrote the initial CUDA code with no regard to these issues that was the start of the if they are not identical use only the best one concept.

Sorry guys, another case of "I don't want to hear it because I know better itis ..." and as usual the first reaction to a mistake on their part is to cripple the sources that might give rise to the issues.

Had they bothered to listen, the infrastructure for different size/class cards would already exist. I would mind less bugs in that then this pretense that this was not an issue that could be anticipated.

Secondly, the mechanisms to handle cards from two MFGRs are basically the same as needed to handle two cards from the same MFGR that have different capabilities. As to the things that needs to be tracked... well, that is what they get paid for...

MarkJ is correct that there are significant differences and that there is need to program for them. Some work is more appropriate to run on one card vs another. If anything I would comment that he missed a couple possible additional features like memory bandwidth, bus bandwidth, memory speed, linked (or not, assuming that linked cards can be both applied as linked to a single task to improve speed) ... but that is not the point. The point is that before we even got to adding ATI cards we could have been debugging those issues that might arrive with the already existing asymmetric CUDA installations (I had one myself)...

Tackling the within family problem first is also more logical because it is far more likely to be the normal situation. User buys card ... user buys faster card and links it to have better gaming ... user thinks that he can use this with BOINC ... user thinks UCB are idiots because thy did not think of this ...

Tackling the within family problem first is also more logical because it is far more likely to be the normal situation. User buys card ... user buys faster card and links it to have better gaming ... user thinks that he can use this with BOINC ... user thinks UCB are idiots because thy did not think of this ...

If I was looking at it from a cost/benefit point of view I think they made the right choice. I have a product (BOINC) that supports one brand of gpu (Nvidia). If I want to improve my reach I add support for the other brand (ATI) for a relatively small cost because I am basically cloning the code. It is also less risky because I already have code that while not great does the job. I have also reduced my dependancy on one brand remaining in business.

If on the other hand I could sit down and tweak the code so it behaves better what is my cost/benefit? Next to nothing. I haven't improved my reach a great deal because I still don't support the other brand. Sure my product is better behaved and I can support more of the same brand, but there is less in that than getting the other brand supported.

---

BOINC blog

If on the other hand I could sit down and tweak the code so it behaves better what is my cost/benefit? Next to nothing. I haven't improved my reach a great deal because I still don't support the other brand. Sure my product is better behaved and I can support more of the same brand, but there is less in that than getting the other brand supported.

Well, I can cite some cost benefit thoughts too ...

It is far better to do it right the first time ...

I am not arguing for the plan or how they implemented it to do one then the other ... though we made several suggestions about that too to make it smoother ... all ignored ... I am suggesting that the concept that BOINC can support one or more cards from one MFGR and one or more cards from another MFGR as long as each set is identical is no more difficult than supporting two cards from the same MFGR that may have differences.

Well I just found this thread... I have been running a machine with 1 9800gtx+ and 1 8800 gt for a couple of months now and as of last week I can only get 1 wu at a time, for the 9800. The 8800 no longer runs. Bummer. Since I can't run it anymore I will retire it. I just can't afford to buy $1800 of cards at a time. My car is worth less than that......