Nov2010> New ACEMD application

Message boards : Graphics cards (GPUs) : Nov2010> New ACEMD application
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 7 · Next

AuthorMessage
Betting Slip

Send message
Joined: 5 Jan 09
Posts: 670
Credit: 2,498,095,550
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19166 - Posted: 1 Nov 2010, 8:55:43 UTC

GT240 really a lot slower with new app. I can't hit the 1 day deadlin anymore and computer freezes periodically. i can't say this is due to the new app and card driver but it started after I changed.
Radio Caroline, the world's most famous offshore pirate radio station.
Great music since April 1964. Support Radio Caroline Team -
Radio Caroline
ID: 19166 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19169 - Posted: 1 Nov 2010, 9:32:59 UTC - in response to Message 19166.  
Last modified: 1 Nov 2010, 10:35:45 UTC

Betting Slip, you have only ran the slower IBUCH tasks since moving to CUDA 3.1. On my Fermi’s these are basically half speed tasks, using between 45% and 55% of the GPU. Wait until you have run a few faster WU's and you might want to test the new app when it comes out, just in case it turns out to be faster for 200series cards.

Sutaru Tsureku, you seem to have found a rare and endangered species - a working GTX260-192 (65nm)
one of its tasks

lkiller123, your clock speeds for both of those GTX260's is high, up from 1.24 to 1.51 and 1.49, and they are in the same rig. What are the GPU temperatures? All 3 failures came from device 1, device 0 has however not returned any completed tasks. Device 1 is at 1.49 GHz; is this the 65nm version? I would be happier to see those cards running in different systems. Putting a second card into a system usually increases the temperature of both cards, especially if you are crunching on your X4 940 as well. You might want to increase the fan speeds of both (I use EVGA Precision to do that).

All 4 of my GT240's (in the one rig) are still at the correct clocks, but it may be tomorrow before they return tasks that were running on a stable rig. So I will not be able to confirm performance differences of that Vista x64 system before then.
On my GTX470's the change in driver to 260.99 made no difference in terms of task performance; completion times were identical for same task types.

Anyone with Vista or Win7 and experiencing speed (frequency) problems should right click on the desktop, click NVidia Control Panel, on the left pane click Manage 3D Settings, on the right pane click the Program Settings Tab, make sure the Power Management Mode is set to Prefer Maximum Performance. If you had to change this you will need to restart the system to make it take effect.

On Win XP, there is no option to change from Adaptive Power Management to Prefer Maximum Performance. If anyone knows of an alternative method to force this please post it up. While this does not effect Fermi's (GTX470's at least) it does mean that some 200series cards go into power saving mode. Firmware updates might solve this, but just try getting the hold of one. I suspect the absence of a Maximum Performance option is a deliberate ploy by NVidia to undermine older generations of cards (they have done this in the past). It's like going into a garage to have your car checked over, and finding they topped up the windscreen washer reservoir but let two tyres down.
ID: 19169 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19176 - Posted: 1 Nov 2010, 16:52:48 UTC - in response to Message 19166.  
Last modified: 1 Nov 2010, 17:00:31 UTC

GT240 really a lot slower with new app. I can't hit the 1 day deadlin anymore and computer freezes periodically. i can't say this is due to the new app and card driver but it started after I changed.

Betting Slip, all my GT 240 cards slowed at least 60% with the 6.11 app.

Betting Slip, you have only ran the slower IBUCH tasks since moving to CUDA 3.1. On my Fermi’s these are basically half speed tasks, using between 45% and 55% of the GPU. Wait until you have run a few faster WU's and you might want to test the new app when it comes out, just in case it turns out to be faster for 200series cards.

SK, don't know why you're covering up this performance loss but here's your own GT 240 machines:

http://www.gpugrid.net/results.php?hostid=51747&offset=0&show_names=1&state=3

(71,990.56 seconds for TONI_KKi4 in 6.05 versus 128,498.32 seconds for TONI_KKi4 in 6.11)

http://www.gpugrid.net/results.php?hostid=33731&offset=0&show_names=1&state=3

(66,627.82 seconds for IBUCH__pYEEI in 6.05 versus 104,038.66 seconds for -IBUCH__pYEEI in 6.11)

SK, your machines are showing even more than a 60% slowdown. Your 3rd GT 240 machine is also showing a huge slowdown but has no EXACTLY comparable WUs to post. To make matters even worse, with 6.11 they are almost always missing the 1 day bonus deadline so the credit/day is approximately (even more than) cut in half :(

Betting Slip, to get the GT 240 cards running 6.05 again I had to revert to older drivers. So far 197.45 seems to be working the best for a combination of stability and reasonable speed. Would like to hear results from others. I tried 160.38 and it was fast at first but maybe because it doesn't officially support the GT 240 it started having strange problems after running for 6-8 hours.

Had to move one machine to Collatz as it had to have newer drivers to support another application and 6.11 is too slow on the GT 240 to even bother with.
ID: 19176 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19179 - Posted: 1 Nov 2010, 18:09:53 UTC - in response to Message 19176.  

On my four GT240 card system I have 3 cards at 1599MHz and one at stock, 1340MHz. So I have to be careful to make sure I am comparing the same card.
The problem I have on comparing tasks on that system is that one of the cards dropped back to 405MHz, and I can't be sure the others did not at some stage. I also got 2 of the slower IBUCH tasks. I was not getting any of these slow IBUCH tasks with 6.05. Until now only Fermi's ran the 3.1 app, so the longer tasks are harder to finish in time for the full bonus. I expect you are right about the GT240's being slower, certainly unable to finish for the full bonus, but I just don’t have enough to go on to be sure about the performace; I have only completed 4 tasks. In a day or two I will know, and I want to wait and see how the 3.2 app works out for GT240's.

ID: 19179 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lkiller123

Send message
Joined: 24 Dec 09
Posts: 22
Credit: 15,875,809
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 19180 - Posted: 1 Nov 2010, 19:20:44 UTC - in response to Message 19169.  
Last modified: 1 Nov 2010, 19:23:40 UTC

Sometimes the WUs will just error out by itself. But most of the errors come from Windows when it says "ACEMD2 6.11....... has stopped working." When I just ignore that message and let it kept crunching, the WU just hangs. So I will be forced to abort the WU through Windows.

By the way, most errors are from the 65nm card.

Maybe that might help a bit for troubleshooting.


lkiller123, your clock speeds for both of those GTX260's is high, up from 1.24 to 1.51 and 1.49, and they are in the same rig. What are the GPU temperatures? All 3 failures came from device 1, device 0 has however not returned any completed tasks. Device 1 is at 1.49 GHz; is this the 65nm version? I would be happier to see those cards running in different systems. Putting a second card into a system usually increases the temperature of both cards, especially if you are crunching on your X4 940 as well. You might want to increase the fan speeds of both (I use EVGA Precision to do that).


I can assure you that the clock speed will be okay to crunch with, I've done Folding@Home and Collatz Conjecture with those clock speeds for about a month already, no problems. However I will try to lower the clock a bit to see what will happen.

The temperature is fine. The cards run at about 60 degrees under GPUGRID load.

I will try to swap the cards around to see what happens.


I will try to test it for another day to see what happens.
ID: 19180 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19181 - Posted: 1 Nov 2010, 19:28:50 UTC - in response to Message 19179.  

I expect you are right about the GT240's being slower, certainly unable to finish for the full bonus, but I just don’t have enough to go on to be sure about the performace; I have only completed 4 tasks. In a day or two I will know, and I want to wait and see how the 3.2 app works out for GT240's.

Given the experience of the 3 of us (Betting Slip, you and I: with a total of over 10 GT 240 cards) I'd say the conclusion is clear. For some reason 6.11 is EXTREMELY slow on GT 240 cards. One strange thing is that GPU usage can stay above 90%, and the core / shader clocks remain high, yet the completion times are abysmal.

ID: 19181 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lkiller123

Send message
Joined: 24 Dec 09
Posts: 22
Credit: 15,875,809
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 19182 - Posted: 1 Nov 2010, 19:40:18 UTC

More findings:

Started out with two brand new WUs.

Check out the weird pattern of GPU Usage.

GPU1 is the GTX260-216 65nm
GPU2 is the GTX260-216 55nm

GPU1 is the one that had been giving out errors.

ID: 19182 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19184 - Posted: 1 Nov 2010, 22:11:48 UTC - in response to Message 19182.  

More findings:

Started out with two brand new WUs.

Check out the weird pattern of GPU Usage.

I've seen this pattern before depending on driver version and what CPU projects are running. The solution for me is to boost the GPUGRID priority to high. I use eFMer Priority, works great:

http://www.efmer.eu/forum_tt/index.php?topic=198.0
ID: 19184 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lkiller123

Send message
Joined: 24 Dec 09
Posts: 22
Credit: 15,875,809
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 19185 - Posted: 2 Nov 2010, 0:26:57 UTC

But the weird thing is, only one GPU happens to be like that. Anything wrong there?
ID: 19185 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19186 - Posted: 2 Nov 2010, 0:39:34 UTC - in response to Message 19185.  

But the weird thing is, only one GPU happens to be like that. Anything wrong there?

Try my solution above, see if it works. Let us know.
ID: 19186 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lkiller123

Send message
Joined: 24 Dec 09
Posts: 22
Credit: 15,875,809
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 19187 - Posted: 2 Nov 2010, 1:07:59 UTC
Last modified: 2 Nov 2010, 1:10:47 UTC

Well, I aborted that particular WU and it went fine for all other WUs.

After a while of crunching, the WU which the 65nm was crunching failed. Like I said above, it crashed and Windows wants to end it.
GPU1 is the 65nm GTX260-216

ID: 19187 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19188 - Posted: 2 Nov 2010, 1:30:42 UTC - in response to Message 19187.  

It looks like the cuda bug still effects your 65nm card, so you should stop crunching with it on GPUGrid. If the 3.2app fairs no better I think it is safe to say your 65nm card will never work here.

Sometimes a system restart can recover from that Windows error. Just keep the error message open and power down or restart the system.

Beyond, I would say I'm seeing a 10% drop in GPU utilization (Vista x64), but I would still like these tasks to finish before I compare them to my previous 6.05 tasks.
ID: 19188 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lkiller123

Send message
Joined: 24 Dec 09
Posts: 22
Credit: 15,875,809
RAC: 0
Level
Pro
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 19189 - Posted: 2 Nov 2010, 1:42:27 UTC - in response to Message 19188.  

Will a BIOS flash help with the situation?
ID: 19189 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19191 - Posted: 2 Nov 2010, 8:20:31 UTC - in response to Message 19189.  

Not that we know. Apparently there's been sopme subtle hardware changes along with the die shrink.

MrS
Scanning for our furry friends since Jan 2002
ID: 19191 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist

Send message
Joined: 14 Mar 07
Posts: 1958
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 19194 - Posted: 2 Nov 2010, 13:18:08 UTC - in response to Message 19191.  
Last modified: 2 Nov 2010, 13:40:06 UTC

ACEMD beta 6.37 cuda2.2 for Windows and Linux is now out. Of course, these should be for g200 cards only and should not work for fermi.

Please accept beta work.

gdf
ID: 19194 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19195 - Posted: 2 Nov 2010, 13:18:19 UTC - in response to Message 19188.  

Beyond, I would say I'm seeing a 10% drop in GPU utilization (Vista x64), but I would still like these tasks to finish before I compare them to my previous 6.05 tasks.


Well here ya go, on your Vista machine with a single GT 240:

http://www.gpugrid.net/results.php?hostid=84713&offset=0&show_names=1&state=0

Your first 6.11 WU took 63% longer than a virtually identical IBUCH_xx_PQpYEEIPI 6.05 WU. To add insult to injury you lost the 1 day bonus and received only 9,556.61 credits for the 6.11 WU compared to 11,467.93 credits for the 6.05 WU. You MUST be getting the picture by now?
ID: 19195 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19197 - Posted: 2 Nov 2010, 15:18:26 UTC - in response to Message 19195.  
Last modified: 2 Nov 2010, 15:25:58 UTC

For GT240's the picture is simple; the CUDA3.1 app is slower than the CUDA2.2 app. and the 260.99 drivers are crap for XP. So people with GT240s should stay with the CUDA2.2 app, and not update to the latest drivers.

That ref. card was left at Adaptive power performance (now changed to Max). The reason for leaving it was to see if the frequency would drop or stay high; I wanted to get an idea of which cards dropped the frequency under which Operating System. On my four GT240 system one cards frequency dropped, but not the others. It also occasionally went back up to full performance only to drop again (a messy picture), and sometimes on a restart stayed high for a while (same as you observed). I wanted to work out if this was also happening on single card Vista systems and W7, but this is now irrelevant as it is clear that the CUDA3.1 app is simply too slow for GT240s.

Not sure why there are beta WUs out for CUDA2.2, thought tasks for the CUDA3.2 app were going to Beta instead, hence I kept the 260.99 drivers on? It will be tomorrow before I can go back to old drivers and get CUDA2.2 tasks again.
ID: 19197 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist

Send message
Joined: 14 Mar 07
Posts: 1958
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 19199 - Posted: 2 Nov 2010, 16:09:45 UTC - in response to Message 19197.  

There was a problem with the server that it was distributing beta cuda2.2 to every cards as there was no cuda3 app. Now it's fixed.

gdf
ID: 19199 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile GDF
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Volunteer tester
Project scientist

Send message
Joined: 14 Mar 07
Posts: 1958
Credit: 629,356
RAC: 0
Level
Gly
Scientific publications
watwatwatwatwat
Message 19200 - Posted: 2 Nov 2010, 21:44:38 UTC - in response to Message 19199.  

Linux and Windows app seem to work well for cuda2.2.
So we can think of updating the official ones.

gdf
ID: 19200 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
fractal

Send message
Joined: 16 Aug 08
Posts: 87
Credit: 1,248,879,715
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 19204 - Posted: 2 Nov 2010, 23:11:03 UTC

Will this release address the Linux CPU utilization bug reported so many months ago http://www.gpugrid.net/forum_thread.php?id=2118 ?
ID: 19204 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 7 · Next

Message boards : Graphics cards (GPUs) : Nov2010> New ACEMD application

©2026 Universitat Pompeu Fabra