New Gianni tasks take loooong time... a warning (8-12-16)

Message boards : Graphics cards (GPUs) : New Gianni tasks take loooong time... a warning (8-12-16)
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
Profile caffeineyellow5
Avatar

Send message
Joined: 30 Jul 14
Posts: 225
Credit: 2,658,976,345
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwat
Message 44148 - Posted: 12 Aug 2016, 12:57:13 UTC
Last modified: 12 Aug 2016, 12:59:18 UTC

Just a warning for those with weaker/slower GPUs than maybe a 770, the new batch of Gianni is going to take around 30 hours on my 980 TI Classified. That means one of my 980 standards would crunch it in about or less than 47 hours. You can go way down from there. My 730 would not finish one in time. My Quadro K2100M would not finish one in time. Some place between the K2100M and the 980, it the cut-off point on these in cards. Depending on settings and configuration, some cards may just make it or fail it wasting time.

To compare, the 980 TI Classy will do a single task in 8 hours or so, and 2 tasks in 11-13 hours. This Gianni has taking 20 hours for the first 68.5% and I turned the card from 2 tasks to 1 task 12 hours ago when it was looking to try to hit 36 hours or more. The other tasks, even the one Gerard_FXCXCL12RX that was running for those first 8 hours with it did normal time and was scheduled to finish in about 12.5 hours. It has since cycled through the other cards on the system and finished.

If you are running a card that struggles for the wall of time given it on normal Gerard_FXCXCL12RX tasks and occasionally misses or makes it on the MO_MOR or MO_TRV tasks, do not attempt these current GIANNI_D3C36bCHL tasks. Again, I am thinking based on configuration and settings and assuming 24 hour a day crunching, somewhere between a 970 and a 770 might not make it and lower than a 770 it would not.
1 Corinthians 9:16 "For though I preach the gospel, I have nothing to glory of: for necessity is laid upon me; yea, woe is unto me, if I preach not the gospel!"
Ephesians 6:18-20, please ;-)
http://tbc-pa.org
ID: 44148 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile caffeineyellow5
Avatar

Send message
Joined: 30 Jul 14
Posts: 225
Credit: 2,658,976,345
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwat
Message 44150 - Posted: 12 Aug 2016, 13:34:15 UTC
Last modified: 12 Aug 2016, 13:37:12 UTC

I also just had one lock my system up and error out on a dual 980 system after 13 hours of running. It took 2 reboots, second one being a hard boot, to get it back up and running aborting the task naturally (didn't hit abort task).

https://www.gpugrid.net/result.php?resultid=15233074
ID: 44150 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44151 - Posted: 12 Aug 2016, 14:42:44 UTC - in response to Message 44148.  

If you are running a card that struggles for the wall of time given it on normal Gerard_FXCXCL12RX tasks and occasionally misses or makes it on the MO_MOR or MO_TRV tasks, do not attempt these current GIANNI_D3C36bCHL tasks. Again, I am thinking based on configuration and settings and assuming 24 hour a day crunching, somewhere between a 970 and a 770 might not make it and lower than a 770 it would not.

Sounds like they should be put in a separate queue.
ID: 44151 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Logan Carr

Send message
Joined: 12 Aug 15
Posts: 240
Credit: 64,069,811
RAC: 0
Level
Thr
Scientific publications
watwatwatwat
Message 44152 - Posted: 12 Aug 2016, 17:46:09 UTC - in response to Message 44151.  

Earlier this morning I've seen the error rates for gianni around 90% and now it's in the 80's.

I have a 960 that completes tasks just fine though in under 24 hours, but I got an error last night due to a power outage.

If I get a gianni task, I'll report how it goes here if you want.


Cruncher/Learner in progress.
ID: 44152 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile caffeineyellow5
Avatar

Send message
Joined: 30 Jul 14
Posts: 225
Credit: 2,658,976,345
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwat
Message 44154 - Posted: 12 Aug 2016, 20:41:32 UTC - in response to Message 44152.  

Sounds good. I was just warning about the length, not really the error rate til I noticed that. Often times new tasks get errors when first run and a bug is fixed for the rest of them, so that isn't a concern of mine. the length on some cards not being able to do them in time was what the warning was really about. But thanks.
ID: 44154 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Logan Carr

Send message
Joined: 12 Aug 15
Posts: 240
Credit: 64,069,811
RAC: 0
Level
Thr
Scientific publications
watwatwatwat
Message 44156 - Posted: 12 Aug 2016, 20:59:48 UTC - in response to Message 44154.  

Oh, I didn't know the errors got fixed liked that, thanks for letting me know!

And no prob, will gladly keep my eye out.
Cruncher/Learner in progress.
ID: 44156 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile caffeineyellow5
Avatar

Send message
Joined: 30 Jul 14
Posts: 225
Credit: 2,658,976,345
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwat
Message 44157 - Posted: 12 Aug 2016, 22:35:40 UTC - in response to Message 44156.  
Last modified: 12 Aug 2016, 22:40:37 UTC

Looks like the one I was running that was going to go longer than 30 hours as 2 tasks per card was able to be reduced to 28.2 hours after it ran the last 10 or so hours as a single task on that card. Still stand by anything 770 or below would not finish and 770 to 970 might come close depending on config and settings. I think this as 2 per card would have been 32-34 hours on the 980 TI Classified clocked at 3005mem and 1430mhz clock speed.

https://www.gpugrid.net/result.php?resultid=15233076

439,250.00 credit by the way, so good accurate recompense for reward compared to those done in half the time for just more than half the credit.

12 have finished. Does anyone have feedback on what card and how long it took? As long as I am making a warning for these, I'd like to make it accurate with feedback. Especially if they are going to continue to come out and be this large 0computationally. Thanks.
ID: 44157 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bedrich Hajek

Send message
Joined: 28 Mar 09
Posts: 490
Credit: 11,731,645,728
RAC: 51
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44163 - Posted: 14 Aug 2016, 13:50:26 UTC

I just finished one of these units on my windows 10 computer:

e2s7_e1s51p0f618-GIANNI_D3C36bCHL1-0-1-RND2166_0 11693869 13 Aug 2016 | 19:38:10 UTC 14 Aug 2016 | 13:35:27 UTC Completed and validated 63,577.01 63,348.98 527,100.00 Long runs (8-12 hours on fastest card) v8.48 (cuda65)


http://www.gpugrid.net/result.php?resultid=15235031


These units seem to be very CPU dependent. The GPU and power usage are slightly lower than the GERARD_FXCXCL12RX units.


ID: 44163 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Retvari Zoltan
Avatar

Send message
Joined: 20 Jan 09
Posts: 2380
Credit: 16,897,957,044
RAC: 0
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44164 - Posted: 14 Aug 2016, 14:26:23 UTC - in response to Message 44163.  
Last modified: 14 Aug 2016, 14:50:02 UTC

I have three running on my hosts.
On a GTX980Ti the task properties show 6.48% per hour, which estimates 15h 25m 56s, but just now I've reduced the number of CPU tasks to 1, so it's a bit faster from now.
ID: 44164 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Retvari Zoltan
Avatar

Send message
Joined: 20 Jan 09
Posts: 2380
Credit: 16,897,957,044
RAC: 0
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44165 - Posted: 14 Aug 2016, 14:42:33 UTC - in response to Message 44156.  
Last modified: 14 Aug 2016, 14:48:55 UTC

Earlier this morning I've seen the error rates for gianni around 90% and now it's in the 80's.
Often times new tasks get errors when first run and a bug is fixed for the rest of them...
Oh, I didn't know the errors got fixed liked that, thanks for letting me know!

Guys, this could be the case, but usually this is a "natural phenomenon" coming from the way the performance / reliability stats work:
As a valid result takes 8-15~24-48 hours to process, a failed one takes only seconds (or maybe just a few hours), so right after the release of a new batch there are only failed tasks in the stats, which can be ignored. Then the stats "normalize" themselves when valid results have returned, but it takes at least as much time as it takes to finish a WU (plus the overhead of the data transmission).
The only way to know that a batch has a bug if it is failing even on the most reliable hosts. This is very rare at GPUGrid.
ID: 44165 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Logan Carr

Send message
Joined: 12 Aug 15
Posts: 240
Credit: 64,069,811
RAC: 0
Level
Thr
Scientific publications
watwatwatwat
Message 44166 - Posted: 14 Aug 2016, 15:54:13 UTC - in response to Message 44165.  

I just downloaded a new task and it's a gianni task. (first one)

I have still a bit to go on my current Gerard task, though.

Would you like me to link you all to my result once it's complete? Maybe that can help others?

Your choice.






Cruncher/Learner in progress.
ID: 44166 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Retvari Zoltan
Avatar

Send message
Joined: 20 Jan 09
Posts: 2380
Credit: 16,897,957,044
RAC: 0
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44167 - Posted: 14 Aug 2016, 16:48:31 UTC - in response to Message 44166.  

Would you like me to link you all to my result once it's complete?
There's no need for that as your computers are not hidden, so anyone can see and browse your hosts and results.
But you can do it by courtesy if you want to make our job easier :)
ID: 44167 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Logan Carr

Send message
Joined: 12 Aug 15
Posts: 240
Credit: 64,069,811
RAC: 0
Level
Thr
Scientific publications
watwatwatwat
Message 44168 - Posted: 14 Aug 2016, 17:49:57 UTC - in response to Message 44167.  

Would you like me to link you all to my result once it's complete?
There's no need for that as your computers are not hidden, so anyone can see and browse your hosts and results.
But you can do it by courtesy if you want to make our job easier :)


Sure! I'll post back when it's complete. (1 or 2 days)
Cruncher/Learner in progress.
ID: 44168 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44170 - Posted: 14 Aug 2016, 23:06:26 UTC - in response to Message 44168.  

Expecting these Gianni tasks to take ~32h on a GTX970 (W10/WDDM).
Noticed the ~65% GPU usage, higher than usual clocks and lowish power usage.
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help
ID: 44170 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bedrich Hajek

Send message
Joined: 28 Mar 09
Posts: 490
Credit: 11,731,645,728
RAC: 51
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44171 - Posted: 15 Aug 2016, 1:49:29 UTC - in response to Message 44163.  

I just finished one of these units on my windows 10 computer:

e2s7_e1s51p0f618-GIANNI_D3C36bCHL1-0-1-RND2166_0 11693869 13 Aug 2016 | 19:38:10 UTC 14 Aug 2016 | 13:35:27 UTC Completed and validated 63,577.01 63,348.98 527,100.00 Long runs (8-12 hours on fastest card) v8.48 (cuda65)


http://www.gpugrid.net/result.php?resultid=15235031


These units seem to be very CPU dependent. The GPU and power usage are slightly lower than the GERARD_FXCXCL12RX units.




Here is an example of this unit type running on a computer with an older and slower CPU and motherboard:

e2s4_e1s51p0f710-GIANNI_D3C36bCHL1-0-1-RND6774_0 11693866 13 Aug 2016 | 19:40:23 UTC 15 Aug 2016 | 1:36:12 UTC Completed and validated 104,399.86 100,946.40 439,250.00 Long runs (8-12 hours on fastest card) v8.48 (cuda65)

http://www.gpugrid.net/result.php?resultid=15235028


ID: 44171 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44172 - Posted: 15 Aug 2016, 3:16:31 UTC - in response to Message 44171.  

Here is an example of this unit type running on a computer with an older and slower CPU and motherboard:

e2s4_e1s51p0f710-GIANNI_D3C36bCHL1-0-1-RND6774_0 11693866 13 Aug 2016 | 19:40:23 UTC 15 Aug 2016 | 1:36:12 UTC Completed and validated 104,399.86 100,946.40 439,250.00 Long runs (8-12 hours on fastest card) v8.48 (cuda65)

http://www.gpugrid.net/result.php?resultid=15235028

Yikes, and that's on a 980Ti. Found 2 running on my boxes. Aborted the one on the 650Ti, left the one running on the 750Ti. Will report when done, if I don't die of old age first.

BTW, CONGRATS on kicking my tukus getting to 3,000,000,000! :-)

ID: 44172 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Bedrich Hajek

Send message
Joined: 28 Mar 09
Posts: 490
Credit: 11,731,645,728
RAC: 51
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44173 - Posted: 15 Aug 2016, 10:40:35 UTC - in response to Message 44172.  



BTW, CONGRATS on kicking my tukus getting to 3,000,000,000! :-)



Thanks, that's what happens if you hang around here long enough! You do lots of crunching.

By the way, I am (have been) (and will not be for long) keeping the number 6 position in total credit warm for you!


ID: 44173 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Retvari Zoltan
Avatar

Send message
Joined: 20 Jan 09
Posts: 2380
Credit: 16,897,957,044
RAC: 0
Level
Trp
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44174 - Posted: 15 Aug 2016, 14:52:38 UTC

e4s30_e1s26p0f463-GIANNI_D3C36bCHL1-0-1-RND6365_0 15h 6m 47s (54.407s) 980Ti/XP
e3s119_e1s26p0f620-GIANNI_D3C36bCHL1-0-1-RND0388_0 14h 58m 29s (53.909s) 980Ti/XP
e3s65_e1s13p0f646-GIANNI_D3C36bCHL1-0-1-RND3014_0 22h 18m 2s (80.282s) 980/XP

I have the feeling of that the length of these workunits is set to the performance level of the GTX 1080.
ID: 44174 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Beyond
Avatar

Send message
Joined: 23 Nov 08
Posts: 1112
Credit: 6,162,416,256
RAC: 0
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 44175 - Posted: 15 Aug 2016, 16:22:06 UTC
Last modified: 15 Aug 2016, 16:24:14 UTC

These long WUs have an extra caution: if there's any kind of power glitch, the app has a very good chance of causing the WU to error out. The app really needs to be fixed, but wonder if it's ever going to happen. Looks like the Gianni will finish on my super-clocked (factory) 750Ti in about 60 hours. Obviously too late for any bonuses and also at risk of power glitches due to the faulty app. I've sadly started aborting the rest of the Giannis. :-(

As previously requested, a separate queue would be nice. Also probably not going to happen.
ID: 44175 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile caffeineyellow5
Avatar

Send message
Joined: 30 Jul 14
Posts: 225
Credit: 2,658,976,345
RAC: 0
Level
Phe
Scientific publications
watwatwatwatwatwatwatwat
Message 44177 - Posted: 15 Aug 2016, 19:10:43 UTC - in response to Message 44163.  
Last modified: 15 Aug 2016, 19:17:31 UTC

These units seem to be very CPU dependent. The GPU and power usage are slightly lower than the GERARD_FXCXCL12RX units.

Yes, and I was off on which cards could finish them on time. I was not estimating the time-out time correctly when I said it (TWICE!) And it seems the CPU has more to do with the length than the GPU on these. Across my systems the same NVIDIA cards on different mobo/CPU combos makes a big difference on how fast they are completing. All my main cards are 980 or 980 TI but the CPUs vary from i7-4960X to i7-4790K to AMD A10-7700K Radeon R7 and they step down in time as the processor gets weaker. The i7-4960X was 101,521 (28 hours). The i7-4790K is looking to be around 111,600 (31 hours). And the AMD A10-7700K Radeon R7 is looking at around 152,500 (42 hours). The i7-4790K and the AMD A10-7700K Radeon R7 are both running 980 standard cards and the settings are the same for both cards (card model and memory and core clock speeds) and external factors (like the 3d settings in the NVIDIA control panel, little usage of other processes, etc), but the finish time is a big difference. The fact that the i7-4960X and the i7-4790K are similar in time but the cards are different (i7-4960X has 980TI Classys)(i7-4790K has 980s) tells me the CPU is making up for the GPU in that the i7-4790K is running at 4Ghz and the i7-4960X at 3.6Ghz. Also the i7-4960X is running CPU tasks and the i7-4790K is not (because of heat which cannot be changed because of its low air movement location). BTW, they are all set to run 2 tasks per card and that 28 hour reading was done with some of the time done as single task cards (as noted in my previous comments), so I suspect they would be even closer in time had I not run them single card (all things being equal and all). Also noteworthy is that the i7-4960X is on Windows 10 but the i7-4790K and AMD are on Windows 7.

So still a warning, but not really card defendant, but GPU, CPU, OS, and mobo all play a part in the total time.

...the new batch of Gianni is going to take around 30 hours on my 980 TI Classified. That means one of my 980 standards would crunch it in about or less than 47 hours.

I was basing that statement on the times of other work units which do stretch out at that proportion. The end result seems drastically different because of the factors I just described.

...15h 6m 47s (54.407s) 980Ti/XP
...14h 58m 29s (53.909s) 980Ti/XP
...22h 18m 2s (80.282s) 980/XP

Maybe all things being equal the card does have more to do with it as well. Though I am noticing that the longest time was on an i7 CPU 870 @ 2.93GHz, the shortest is on an i7-4930K CPU @ 3.40GHz, and the one similar in length to the short one is on an i3-4160 CPU @ 3.60GHz. Is there a difference in settings, usage of other processes, or whatever else that is different between the i7-4930K and the i3-4160 that would make the 3.6Ghz slightly slower than the 3.4Ghz one both on 980TIs (like pcie speed on the mobo, etc)?

I have the feeling of that the length of these workunits is set to the performance level of the GTX 1080.

I am not sure if the GTX10 has much to do with planning the length of time to complete. It may be just the case, but I would think if they were planning length to completion they would keep them at or slower than the current GERARD_FXCXCL12RX series. Many of the cards still in common use (the GTX 7 series and above) should be able to do a task in 24 hours or less in my opinion. Having the 9 or 10 series should make the long units as they are stated, "under 8 hours" and the slower cards up to 24 hours with the laptop GPUs and slower cards being able to do them in the time allotted to time out. I have seen many comments here and on my team forum stating they stopped crunching GPUGRID altogether because they just could not finish tasks in time, either they have the older cards or they can't keep the PC on 24 hours a day.

I do understand that the longer they can be "out on the field" the less bandwidth is needed constantly on the servers and college, so I am not ruling out the need for it if that is the case. I just think there has to be a better balance of practicalities between the needs of the project infrastructure and the needs of the project volunteers to complete the work. This would/might be a bigger concern if the tasks were needed quickly, as sometimes they are, for a deadline or if there were so many tasks that the users could not grab and crunch them fast enough for the amount of work to be done. In the current state (and I am talking about at least since this time last year or earlier), the WUs available are zero most of the day most of the time and when they add 200 or 500 they are gone in about an hour or so if that. And I know that is dependent on the amount of students/staff that need work done and the need of the papers and science those students are doing related to what can/has to be done via distribution.

That is the downside of working out of a school for student needs though and not out of a science lab for scientific research like other projects. The upsides far outweigh the downsides though, as the work helps students get their degrees, papers published, and thesis and the actual work concluded by the findings. Back in the United Devices days we did HMMR, Markov modelling, and then moved to straight cancer/protein binding and even finished the Anthrax cure in 30 days and people were mad that UD was a for profit company. Even though the work done through the volunteers was donated to those who could actually use the research for the science, the company itself was using the distributed projects to complete and test their own distributed platform for corporate customers looking to complete large tasks across their in house networks and then sold as such. It was unfortunate that when they sold the company the projects ended without finishing, the work done for over a year was very valuable to Oxford labs and the National Foundation for Cancer Research here in the states. I am sure that the work completed there (as well as the work since then at F@H and other BOINC projects) led to finding the markers that my eventual cancers would be reacted upon by my current chemotherapy drug. When my doctor told me that it might have an effect on my cancer he told me, without me asking, that computer modeling was what found the reaction and not trials on actual people and that this particular drug was not used for my cancer until that was found in the distributed projects. And low and behold, it did reduce the tumor and its activity. So as I said, the science far outweighs the methods and needs for those methods.

Obviously too late for any bonuses

It looks like these are getting bonuses as their credit is high enough to not need the bonus as an extra bonus is added into the task based on its run-time. Even the 35 hour ones award 439,250 credit which is a bonus in relation to the credit that would be awarded to any other WU in current production for that time. It does seem though that the ones done in less 24 hours are getting 527,100, so... normal bonus on top of extra bonus?. According the the Performance page, only the top 10 (that allow their tasks to be seen publicly)(of which I should be listed in the 18 spot for that 28 hour one and am not for some reason) have been under 24 hours.
ID: 44177 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · 4 · Next

Message boards : Graphics cards (GPUs) : New Gianni tasks take loooong time... a warning (8-12-16)

©2025 Universitat Pompeu Fabra