Advanced search

Message boards : Graphics cards (GPUs) : Tesla K40

Author Message
Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 33987 - Posted: 23 Nov 2013 | 11:04:39 UTC

nvidia launches tesla k40, overview by Ryan Smith.

Ignoring reality (the hiked up $5500 off the shelf price tag), the design expands potential; Boost is 19.5% faster than a K20 (no boost), it has twice the memory (12GB), faster GDDR, an additional 192 shaders and with the help of the new B1 stepping and binning, it's still a 235W model.
The performance boost in apps over the K20 seems to range from 11% to 25% but could be as high as 28%. The K40 is PCIE3 (unlike the K20 which is PCIE2). This eliminates some bottlenecks for some applications, but unless it scales much better on the newer Xeon processors there is still going to be some drop off.
Obviously the 5TFlops GTX780Ti is the best card for here, but the K40 offers versatility at a high performance level and opens up more R&D avenues.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 33992 - Posted: 23 Nov 2013 | 18:41:29 UTC - in response to Message 33987.

Since the K40 is significantly lower clocked than GTX780Ti, it's no wonder it's "only" got a 235 W TDP.. considering GTX780Ti has a 250 W TDP. Aprart from the additional memory and it being ECC (which can mean the world to some) the feature that really sets these cards apart is the massive FP64 performance.

MrS
____________
Scanning for our furry friends since Jan 2002

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 34000 - Posted: 24 Nov 2013 | 0:10:16 UTC - in response to Message 33992.

Yes, when you compare the K40 to a 780Ti (the best GPU for here), but when you compare the K40 to its predecessor (the K20) what's noteworthy is that it's PCIE3 complaint, the potential performance improvement is 28%, and what's outstanding is the 12GB GDDR5 (Rendering, Fluid Dynamics and Seismic Analysis...). Considering the GPGPU has more cuda cores, is clocked higher, has Boost, faster RAM and twice the amount of RAM, being able to stick to the 235W TDP is very nice, and for business/research this is very important - it's a potential upgrade.
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

VT
Send message
Joined: 28 Nov 13
Posts: 17
Credit: 153,786,987
RAC: 0
Level
Ile
Scientific publications
watwatwatwatwatwatwatwatwatwat
Message 34193 - Posted: 10 Dec 2013 | 21:44:19 UTC - in response to Message 34000.

yes, 12Gb and 64fp performance is hard to replace for some tasks.
I was very happy they released a 1k$ 6gb card a few months ago, as even current 3gb 780ti is significantly slower for my tasks due to a lot of memory swapping between GPU and system.

12Gb would be even better, but I can not justify 5k$ just yet :)

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35435 - Posted: 1 Mar 2014 | 20:14:46 UTC

I'm using two GTX Titans and two Tesla K40 and I have 305,000 average work. I don't know if that is ok or should be higher.

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35436 - Posted: 1 Mar 2014 | 20:25:55 UTC - in response to Message 35435.

It is bad
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35438 - Posted: 1 Mar 2014 | 22:38:07 UTC

My GTX660Ti does ~320k RAC, had even reached 330k. And that's under Win 8.1, i.e. without XP/Linux speed boost.

MrS
____________
Scanning for our furry friends since Jan 2002

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35441 - Posted: 2 Mar 2014 | 1:21:46 UTC
Last modified: 2 Mar 2014 | 1:23:36 UTC

Any Ideas whats wrong with the setup?? Running Boinc as is with Nvidia driver 332.50. Shows 4 GPU crunching.
I am now installing on Windows 2008 R2 to see if the tesla k40 work better. also using single precision on all cards

Thanks
____________

Profile Mumak
Avatar
Send message
Joined: 7 Dec 12
Posts: 92
Credit: 225,897,225
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 35442 - Posted: 2 Mar 2014 | 7:57:42 UTC - in response to Message 35441.

Post some info about your GPU stats - load %, temperatures, power %, etc.

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35443 - Posted: 2 Mar 2014 | 9:24:33 UTC - in response to Message 35442.

Hello, under GPU=Z the load on both titans is 66%, temp about 56C, power 66%.

The tesla k40 show nothing under GPU-Z but they are crunching. Wrote to Nvidia and they are working on a driver fix so that it shows up under GPU-Z

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35444 - Posted: 2 Mar 2014 | 10:00:22 UTC - in response to Message 35443.
Last modified: 2 Mar 2014 | 10:24:52 UTC

Stop running Short tasks and just run Long tasks. The credit/h is higher.
It will take time for your RAC to rise. Going by the runtime of your last Long task, if your computer is on 24/7 your potential RAC is about 1.6Million/day. However, I suspect a better configuration could improve that to somewhere between 1.8M/day to 2M/day. Running short tasks it's would only be about 700K/day.

I suggest you stop running CPU tasks on that system, and just run GPU work. Use software such as MSI Afterburner to set the fan high enough to prevent the cards from getting too hot. Check that your GPU clocks don't fall.

I see that several tasks failed or were Abandoned (presumably when you changed OS). Odd that the GPU's are failing when the temp is only 62C.

Suggested reading,
FAQ - Best configurations for GPUGRID
____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

biodoc
Send message
Joined: 26 Aug 08
Posts: 183
Credit: 6,493,864,375
RAC: 2,796,812
Level
Tyr
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35445 - Posted: 2 Mar 2014 | 11:00:13 UTC

I would think you would want to move the Tesla cards to a different computer and install drivers that are designed specifically to support them. I think the latest win 8.1 drivers for Tesla are 332.50 WHQL, which interestingly also supports cuda 6.0.

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35450 - Posted: 2 Mar 2014 | 13:48:13 UTC

Thanks for the replies. How do i specify only long tasks?

Thanks

ExtraTerrestrial Apes
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 17 Aug 08
Posts: 2705
Credit: 1,311,122,549
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35455 - Posted: 2 Mar 2014 | 16:45:32 UTC - in response to Message 35450.

You can set it in your GPU-Grid settings in "my account", for each venue separately if needed. You can assign hosts to venues in the host details under "my computers".

(I'm assuming you're familiar with how this owrks under pretty much all BOINC projects, if not I/we could be more specific)

MrS
____________
Scanning for our furry friends since Jan 2002

TJ
Send message
Joined: 26 Jun 09
Posts: 815
Credit: 1,470,385,294
RAC: 0
Level
Met
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35456 - Posted: 2 Mar 2014 | 16:47:04 UTC - in response to Message 35450.
Last modified: 2 Mar 2014 | 16:47:58 UTC

Thanks for the replies. How do i specify only long tasks?

Thanks

On the web site in "Your account" then "GPUGRID Preferences". You can choose for long, short, beta. If you only thick the long runs, you get only those.

Edit: while typing this ETA was faster :)
____________
Greetings from TJ

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35459 - Posted: 2 Mar 2014 | 19:29:10 UTC

Okay. All set as long runs only. Let's see what happens!

Thanks

Profile Mumak
Avatar
Send message
Joined: 7 Dec 12
Posts: 92
Credit: 225,897,225
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 35460 - Posted: 2 Mar 2014 | 19:35:29 UTC

It's strange, that GPU-Z doesn't show info about Tesla.
Can you maybe try HWiNFO if it reports sensor information about them ?

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35461 - Posted: 2 Mar 2014 | 19:53:27 UTC - in response to Message 35460.
Last modified: 2 Mar 2014 | 19:58:18 UTC

Installed HWinfo and it show tesla K40 at 63% load on one of the tesla. The other tesla shows up but the load information is not shown or is not listed on the screen to show up.

Same for the Titan, 63% load and shows complete info on one only.

Profile Mumak
Avatar
Send message
Joined: 7 Dec 12
Posts: 92
Credit: 225,897,225
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 35465 - Posted: 2 Mar 2014 | 20:37:25 UTC - in response to Message 35461.
Last modified: 2 Mar 2014 | 20:38:11 UTC

I could have a look at why/what it reports in more detail if you send me the HWiNFO Debug File. But I think the most likely reason for not showing all GPUs is that the drivers don't have proper NVAPI support (yet).

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35466 - Posted: 2 Mar 2014 | 21:34:35 UTC - in response to Message 35465.

That is what nvidia told me about the Teslas not showing up under GPU-Z.

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35467 - Posted: 2 Mar 2014 | 21:56:18 UTC - in response to Message 35465.

How do you want me to send the debug file? Thanks for your time.

Profile Mumak
Avatar
Send message
Joined: 7 Dec 12
Posts: 92
Credit: 225,897,225
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 35468 - Posted: 2 Mar 2014 | 22:02:33 UTC

I have sent you a PM about that

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35470 - Posted: 2 Mar 2014 | 22:12:52 UTC - in response to Message 35468.

Done

Profile Mumak
Avatar
Send message
Joined: 7 Dec 12
Posts: 92
Credit: 225,897,225
RAC: 0
Level
Leu
Scientific publications
watwatwatwatwatwatwatwatwatwatwat
Message 35471 - Posted: 2 Mar 2014 | 22:38:56 UTC

After checking the detailed data, it's indeed an NVIDIA driver problem. NVAPI doesn't return information about Teslas at all.
Moreover, due to a bug in addressing, HWiNFO thinks the 2nd adapter is the Tesla, but in fact that information should belong to the 2nd TITAN.
NVIDIA needs to fix this.

Despite that, I think that HWiNFO should display at least the temperature for all GPUs even if not currently properly supported by NVAPI.

Wdethomas
Send message
Joined: 6 Feb 10
Posts: 38
Credit: 274,204,838
RAC: 0
Level
Asn
Scientific publications
watwatwatwatwatwat
Message 35479 - Posted: 3 Mar 2014 | 11:22:58 UTC - in response to Message 35471.

This is what Nvidia says:

"Checking with Driver team.

Tesla customers use the nvsmi or nvml for getting the gpu statistics.

Nvapi is more for geforce and quadro. I have asked engg if the nvapi extends to Tesla."

Thanks

Profile skgiven
Volunteer moderator
Volunteer tester
Avatar
Send message
Joined: 23 Apr 09
Posts: 3968
Credit: 1,995,359,260
RAC: 0
Level
His
Scientific publications
watwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwatwat
Message 35554 - Posted: 7 Mar 2014 | 9:11:28 UTC - in response to Message 34193.
Last modified: 7 Mar 2014 | 9:12:01 UTC

yes, 12Gb and 64fp performance is hard to replace for some tasks.
I was very happy they released a 1k$ 6gb card a few months ago, as even current 3gb 780ti is significantly slower for my tasks due to a lot of memory swapping between GPU and system.

12Gb would be even better, but I can not justify 5k$ just yet :)


CUDA6 will introduce Multi-GPU scaling (cublasXT); should allow you to use the GDDR memory of up to 8 cards on one app.
NVidia,

    a new BLAS GPU library that automatically scales performance across up to eight GPUs in a single node, delivering over nine teraflops of double precision performance per node, and supporting larger workloads than ever before (up to 512GB). The re-designed FFT GPU library scales up to 2 GPUs in a single node, allowing larger transform sizes and higher throughput.


____________
FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Post to thread

Message boards : Graphics cards (GPUs) : Tesla K40

//