I have serious problems with my new card, and after trying to make it work with GPUGrid for a whole day - quite frankly - I've run out of ideas.
Does anybody crunch with this type of card (GV-N780TOC-3GD) for GPUGrid?
I have a standard Gigabyte GTX 780 Ti in this host (beside an ASUS GTX 670 DC2 OC). These cards are doing very well their job. Originally I wanted a Gigabyte GTX 780 Ti OC, so after waiting for this card a couple of weeks it became available, and I've ordered and received one. I've put this card beside its little brother (the non OC Gigabyte GTX 780 Ti), and my problems immediately emerged: the new card failed every task almost immediately.
There is only one task which was run for 2000s, because the card was downclocked to 450MHz, bur after a restart the downclock was gone, and it has failed too.
Since then I've tried the following, none of them helped:
1. uninstall NV driver, restart, install latest NV (331.93 beta) driver
2. install a fresh Win7 x64 on this host, with the latest beta driver
3. Put the failing card in different PCIe slots of the GA-Z87X-OC motherboard, remove the original card, and put the failing card in it's place to be the only card in the system
4. test the card with different applications: Heaven's benchmark, Furmark, Primegrid, GPU memory stress test, G80 memtest - all of them were OK
5. put the card to a different MB (Intel DH87RL), under different OS (Win8.1 x64)
6. Further tests: NVidia human face demo (it's quite stunning actually), 3DMark13, NVidia design garage, NVidia Islands - all of them OK.
7. downclock the card with MSI Afterburner
8. increase the GPU voltage to the maximum of 1212mV with Kepler BIOS tweaker 1.26
9. uninstall the 331.93 driver, restart, download and install the card's original driver (331.60) from Gigabyte
10. decrease the GPU clocks, power limits, boost clocks, etc. to the settings of the working (non-OC) card with Kepler BIOS tweaker 1.26
11. decrease the GPU clocks further down to 800MHz
12. decrease the GPU memory clock to 3.4 GHz
13. decrease the GPU memory clock to 3.3 GHz
14. decrease the GPU memory clock to 3.3 GHZ and the GPU clocks etc. to the settings of the working card.

The really annoying aspect of this is that only the GPUGrid tasks are failing on this card - the tests, benchmarks, and demos are not. I had a faulty GTX 580 some time ago, but that card also showed some heavy artifacts with Heaven's benchmark. I would appreciate if you could recommend any further testing tool which can detect any defect on this card (while it shows no problem on the other GTX 780 Ti), because I'm not convinced that this card is faulty.
Tomorrow I'll try to test this card with folding@home (I couldn't reach Stanford's webpage today).

ok already done on Point 4 sry ^^

---

DSKAG Austria Research Team: http://www.research.dskag.at

I also doubt the card is defective but if it can't run GPUgrid tasks with the numerous tweaks you've tried then something is wrong somewhere.

I've Googled terms like "GV-N78TOC-3GD problem", "problem GV-N78TOC-3GD", "fubar GV-N78TOC-3GD", "GV-N78TOC-3GD fail", etc. and haven't found any reports or reviews speaking of problems.

Do any of the diagnostic programs you've run test with CUDA or in CUDA mode? I ask because if you can demonstrate that it consistently fails on CUDA but not OpenGL or alternatively fails OpenGL and CUDA but nothing else then you have something definite that you can present to Gigabyte and/or NVIDIA if you want to motivate them to investigate and perhaps issue a driver update or BIOS update.

The point is you've more or less eliminated clock and voltage settings, mobo and drivers as the problem. The only other possible things I can think of to experiment with are:

1) Win XP or Linux versus Win 7/8
2) CUDA vs. OpenGL vs. "game mode" or whatever the correct term is
3) localized over-heating which I explain below

We know there are different ways of measuring/detecting GPU core temperature as was discussed recently in another thread here. Maybe certain portions of the GPU core are not cooling properly due to one or more of the following causes:

1) a curved heat spreader on the GPU
2) a curved surface on the heat sink
3) improper application of thermal grease

Maybe high temps in affected areas are not showing up in temp reports and only GPUgrid tasks are exercising those poorly cooled areas. (Yes, that does require a number of factors to align properly but nothing happens by accident, there is always a cause, you have eliminated most of the causes already and this one is one of the few remaining reasons.)

If you have good vision you can spot badly curved/warped surfaces easily with what I call "the straight edge and light test" which is a very common test. You probably are familiar with how it works but don't rely on it unless you know your vision is good. A much better way of measuring flatness is to use a dial gauge on a pivot as it will easily show defects as small as .005 inches if used properly. A top quality dial gauge and pivot are expensive but there are less expensive models that are accurate enough for the job we're talking about. Or you can take the GPU and heatsink to a machine shop and pay them to check the flatness.

If it turns out one or both of the 2 surfaces are curved then you can correct it by lapping if the mismatch is small or by machining and lapping/polishing if the mismatch is large.

Some people think they can detect curved surface(s) by seeing if they can rock the heatsink side-to-side. That test can reveal a convex-to-flat or convex-to-convex situation but cannot reveal a concave-to-flat or concave-to-concave situation.

---

BOINC <<--- credit whores, pedants, alien hunters

I am sorry to hear you have problems with you new card Zoltan. I can't help you, as you don't know the answer yourself then I certainly don't know. The only thing I can think of is that these cards are to powerful but that can't be true?
Good luck, I am sure over time you will find the answer.

But I have two questions.
1. The host you are showing has awesome time compared to mine and you have now Win7 installed as well? What have you done then to get these results.
2. I downloaded and installed Kepler BIOS Tweaker as you suggested in the thread I started, but all I see are empty fields and clicking in it does not work, I can not put any values in. What did I do wrong there?

---

Greetings from TJ

I also doubt the card is defective but if it can't run GPUgrid tasks with the numerous tweaks you've tried then something is wrong somewhere.

Errrr.....

I've Googled terms like "GV-N78TOC-3GD problem", "problem GV-N78TOC-3GD", "fubar GV-N78TOC-3GD", "GV-N78TOC-3GD fail", etc. and haven't found any reports or reviews speaking of problems.

That's pioneer's fate.

Do any of the diagnostic programs you've run test with CUDA or in CUDA mode? I ask because if you can demonstrate that it consistently fails on CUDA but not OpenGL or alternatively fails OpenGL and CUDA but nothing else then you have something definite that you can present to Gigabyte and/or NVIDIA if you want to motivate them to investigate and perhaps issue a driver update or BIOS update.

Primegrid PPS Sieve & GeneFer is CUDA (I'm not sure about its version though). As I further tested with PrimeGrid, the GeneFer CUDA client was stuck once, and anonther task has failed - I'm not sure about why.
FurMark is OpenGL.
Heaven's Benchmark is DX11 & tessellation is OpenGL4.0.

The point is you've more or less eliminated clock and voltage settings, mobo and drivers as the problem. The only other possible things I can think of to experiment with are:

1) Win XP or Linux versus Win 7/8

The card first failed under WinXPx64, I've switched to Win7x64 and Win8.1x64 only for further testing the card without interrupting the crunching.

2) CUDA vs. OpenGL vs. "game mode" or whatever the correct term is

My guess is that the GPU Boost 2.0 make mistakes, or the algorithm in the GPUGrid client which detects when the simulation becomes unstable, or I've got a very tricky error in my card.

3) localized over-heating which I explain below

We know there are different ways of measuring/detecting GPU core temperature as was discussed recently in another thread here. Maybe certain portions of the GPU core are not cooling properly due to one or more of the following causes:

1) a curved heat spreader on the GPU
2) a curved surface on the heat sink
3) improper application of thermal grease

Maybe high temps in affected areas are not showing up in temp reports and only GPUgrid tasks are exercising those poorly cooled areas. (Yes, that does require a number of factors to align properly but nothing happens by accident, there is always a cause, you have eliminated most of the causes already and this one is one of the few remaining reasons.)

I got that. As you say, it's very unlikely that this is the source of my problems. However, I'll check the heatsink if I can remove it without voiding warranty.

If you have good vision you can spot badly curved/warped surfaces easily with what I call "the straight edge and light test" which is a very common test. You probably are familiar with how it works but don't rely on it unless you know your vision is good. A much better way of measuring flatness is to use a dial gauge on a pivot as it will easily show defects as small as .005 inches if used properly. A top quality dial gauge and pivot are expensive but there are less expensive models that are accurate enough for the job we're talking about. Or you can take the GPU and heatsink to a machine shop and pay them to check the flatness.

If it turns out one or both of the 2 surfaces are curved then you can correct it by lapping if the mismatch is small or by machining and lapping/polishing if the mismatch is large.

Some people think they can detect curved surface(s) by seeing if they can rock the heatsink side-to-side. That test can reveal a convex-to-flat or convex-to-convex situation but cannot reveal a concave-to-flat or concave-to-concave situation.

From my experience it's much more common error that one or two corners of the heatsink is not fastened well, so it's touching the chip only on one (or two) edge(s), not on its entire surface.

I have two questions.
1. The host you are showing has awesome time compared to mine and you have now Win7 installed as well? What have you done then to get these results.

I'm still crunching under WinXPx64 on that host. I've installed Win7 only to have DX11 and some other fancy stuff for the graphical tests.

2. I downloaded and installed Kepler BIOS Tweaker as you suggested in the thread I started, but all I see are empty fields and clicking in it does not work, I can not put any values in. What did I do wrong there?

The previous (1.25) version can extract / flash the BIOS from the card if nvflash.exe is located in its folder. The latest one (1.26) can't, it can manipulate the firmware image in a file, so you have to extract / flash it manually with nvflash.exe (GPU-Z can extract the BIOS from the card through GUI, but it also uses a built-in copy of nvflash.exe)

I've flashed the working card's BIOS to the OC card (they have different vendor and PCI subsystem IDs, so I was a little concerned about doing it). The card was working ok, but the GPUGrid client still fails. I'm afraid that I have to sell this beautiful card to a gamer...

From my experience it's much more common error that one or two corners of the heatsink is not fastened well, so it's touching the chip only on one (or two) edge(s), not on its entire surface.

Yep, that happens frequently and bit me once a few years ago on a CPU. The temperatures were a little high but still reasonable. It took me a long time to figure it out.

---

BOINC <<--- credit whores, pedants, alien hunters

Zoltan, you have already tried a lot, and as you have tried the GPU on different rigs and with different operating systems, my guess is that the card is a dud, most likely GDDR/some capacitor. Alternatively the GPUGrid app/s simply don't work with it due to some obscure oddity in the app/bespoke card (slim chance).
Suggest you check the GPU physically (loose anything, burnt smell, dodgy soldering...), sunk PCIE power connector pins (on the card), does the GPU seat fully?

Some loose/desperate suggestions (possibly already covered):
Try short tasks, if you haven't already.
What was installed with the NVidia drivers; all the 3D and sound crap or just the drivers?
Have you tried lowering the power target (MSI Afterburner...)?
What about System Power settings?
NVidia control panel settings? Prefer max performance, PhysX pointing where - specific GPU or CPU (not sure that even matters though)?
Have you tried to drop the GPU memory to 3000MHz?
Motherboard Bios upgrade (long shot).
CPU drivers from Intel (might be messing with the bus)?
Chipset update?
Different versions of Boinc (perhaps even completely uninstalling Boinc and then reinstalling.

Tried Linux?

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

Having just completed a new Haswell build, I have learned that a motherboard that is quite stable with its own internal graphics can go bananas once I insert a GTX 660 and try to run BOINC/GPUGrid. As soon as BOINC starts (barely having time to reach the desktop, if that), I get BSODs. That itself is not so unusual, but like you I had already underclocked/overvolted the card sufficiently that it should have worked fine.

The solution for me lay in the motherboard DDR3 memory; I had manually set the speed to 1600 MHz and 8-8-8-24, which is the rated speed of the Crucial Ballistix 2 GB modules (4 of them). But by returning to the default motherboard value of 1333 MHz and 9-9-9-24 timings (as set by SPD), the problems seem to have disappeared. Maybe you have something similar.

1) a curved heat spreader on the GPU
2) a curved surface on the heat sink
3) improper application of thermal grease

Yesterday I've had the nerves to dismount the heatsink from the failing card. There were 2 things that surprised me:
1. there are only 7 screws fixing the whole heatsink assembly to the card. (on the original one there are 4 bigger and 4 smaller screws only for the GPU)
2. the thermal grease and the surface of the heatsink for the GPU was ok, the thermal pads for the RAM chips also was fine, but the one long thermal pad for the 8 chips of the GPU's power supply was too short, so it was stretched to reach the 8th chip, therefore it became too thin between the 7th and 8th. I've cut some strips from the unnecessary parts of the long side of the thermal pad, and put it to the 8th chip. Unfortunately, this didn't helped. I couldn't disassemble the card since then, but I'll do it on 27th when I get home.

Zoltan, you have already tried a lot, and as you have tried the GPU on different rigs and with different operating systems, my guess is that the card is a dud, most likely GDDR/some capacitor. Alternatively the GPUGrid app/s simply don't work with it due to some obscure oddity in the app/bespoke card (slim chance).
Suggest you check the GPU physically (loose anything, burnt smell, dodgy soldering...), sunk PCIE power connector pins (on the card), does the GPU seat fully?

Yesterday I've checked both sides of the card (I did check the backside before). There is a lot of components on the front side of the board, so it's impossible to check every soldering (not to mention the BGA packaging of the GPU and the RAM chips, which covers the actual soldering). But I didn't notice any sloppy soldering, loose or missing components (however, there are unused component spaces on the board, but it's normal).
The PCIe power connectors (2x8 pin) are a little different on this card than usual: the latch(?) (I don't know how we call it even in my native language) in which the clamp of the PCIe connector clicks in is much shallower than usual (about 1/5th of the normal), so it's much easier to remove the PCIe power connectors from the card.

The most interesting part is that the card is running PrimeGrid tasks just fine. (I know that they're not comparable to GPUGrid tasks)

I could make the card consume more power with FurMark than when crunching GPUGrid tasks, and it was running for about an hour without errors.
I let Heaven Benchmark run for a whole night, and it didn't produce any artifacts.

Some loose/desperate suggestions (possibly already covered):
Try short tasks, if you haven't already.

I didn't try short runs before you asked, but the short run is also failed.

What was installed with the NVidia drivers; all the 3D and sound crap or just the drivers?

On the WinXPx64 host only the graphics drivers installed, on the same hardware I've installed everything under Win7, and on the other MB with Win8.1 I've also installed everything.

Have you tried lowering the power target (MSI Afterburner...)?

Yep. Lowering, increasing.

What about System Power settings?

Never go to sleep, never turn off monitor (never give up, never surrender :))

NVidia control panel settings? Prefer max performance,

That's why I've installed everything under Win7. For a moment I thought it helped, but after 10 secs the WU crashed.

PhysX pointing where - specific GPU or CPU (not sure that even matters though)?

It points to the GPU, but the WinXPx64 doesn't have PhysX, so that's irrelevant I guess.

Have you tried to drop the GPU memory to 3000MHz?

I've tried 3400MHz and 3300MHz.
I just did something new: flashed the BIOS of the Graphic card in a RealVNC session :). It's now down to 3000MHz.

Motherboard Bios upgrade (long shot).

Both MB have the latest BIOS installed (before it all began), GA-Z87X-OC: F6, DH87RL: 0323

CPU drivers from Intel (might be messing with the bus)?

Do such drivers exist? Could you please give me a link?

Chipset update?

The latest chipset drivers are installed (9.4.0.1027)

Different versions of Boinc (perhaps even completely uninstalling Boinc and then reinstalling.

I don't like the series 7 of the BOINC manager, but after the first couple of failures I've upgraded to 7.2.33 from 6.10.60. The only difference I've noticed, that now I can see such status messages as "Trying to restart unstable simulation" instead of "waiting for GPU memory" (the latter made me to upgrade to the latest BOINC manager). But it didn't help.

Tried Linux?

I'm kind of a Windows guy. :) I even hate power shell (namely the concept that there is a lot of things you can't do through GUI). I don't know Linux. I think it's not a good idea trying something unknown to fix a tricky error. Besides, I don't believe that the source is the OS, because another GTX 780 Ti is working fine on my system.

Have you tried to drop the GPU memory to 3000MHz?

Wow!
A short run is finished at 3000MHz memory clock. There was two restarts, so I'm lowering the RAM frequency to 2900MHz, and trying a long run.

I see you have had 2 more successful runs and one failure,

188x-SANTI_MAR422cap310-13-32-RND9223_1 5020610 24 Dec 2013 | 18:39:10 UTC 25 Dec 2013 | 2:47:24 UTC Completed and validated 21,301.04 21,173.92 115,650.00 Long runs (8-12 hours on fastest card) v8.14 (cuda55)

I161-SANTI_bax2-9-32-RND0580_0 5021047 24 Dec 2013 | 20:13:13 UTC 25 Dec 2013 | 6:49:52 UTC Completed and validated 21,901.13 21,750.64 154,050.00 Long runs (8-12 hours on fastest card) v8.14 (cuda55)

76x-SANTI_MAR422cap310-13-32-RND3988_0 5022310 25 Dec 2013 | 6:40:37 UTC 25 Dec 2013 | 10:29:10 UTC Error while computing 5,981.45 5,943.67 --- Long runs (8-12 hours on fastest card) v8.14 (cuda55)

Despite the memory drop that's still around 28% faster than my GTX770.

For each task, the logs show the GPU temps going up to 64C and then the card sometimes stops working for a while. This suggests to me that there is something not right with the cooling. While the GPU is fine, I suspect the GDDR5 is not (or something related to the GDDR). Perhaps a bad module that might not get used by other types of work.


* I would say "plastic clip" on the PCIE power connector.

---

FAQ's

HOW TO:
- Opt out of Beta Tests
- Ask for Help

I see you have had 2 more successful runs and one failure,

Yep, the error is still there, despite that now the RAM runs at 2800MHz. Sometimes lowering the frequency makes things worse, and I think that lowering even more the RAM frequency won't solve completely the problem of my card.

Despite the memory drop that's still around 28% faster than my GTX770.

This card runs under Win8.1, but I'll put it in my WinXPx64 host (if I can fix it), and it will be even more faster :).

For each task, the logs show the GPU temps going up to 64C and then the card sometimes stops working for a while. This suggests to me that there is something not right with the cooling. While the GPU is fine, I suspect the GDDR5 is not (or something related to the GDDR). Perhaps a bad module that might not get used by other types of work.

I'll dismount the cooler assembly once again when I get home, and check the thermal pads again, but I think this is either a memory power line failure or a RAM chip failure. I guess that some of the capacitors have insufficient capacity, or sloppily soldered (or missing). First I'll check it with my naked eye (through my reading glasses), but if I don't find something suspicious I'll take a couple of macro photos from different parts of the card, and check the photos. Now that I know what part of the card malfunctions, it's not a mysterious error anymore, and I have ideas about finding and fixing the card. If I can't fix it, I still can RMA the card, as now I'm confident that this card is bad, also I can prove it to the RMA guys.

I see you have had 2 more successful runs and one failure,

Yep, the error is still there, despite that now the RAM runs at 2800MHz. Sometimes lowering the frequency makes things worse, and I think that lowering even more the RAM frequency won't solve completely the problem of my card.

Well, fortunately I wasn't right about that: I've put this card to my WinXPx64 host's PCIe 2.0 x4 slot, and it had a couple of errors, so I've lowered the RAM frequency to 2700MHz, and now it's running smoothly. There is a 2000 sec loss compared to the (standard and oc-ed) card in the PCIe 3.0 x16 slot. I'll try the OC card in the PCIe 3.0 x8 slot (which will make the standard card to run at x8 also) to see how much loss is caused by the lowered RAM frequency.

I just got my new 780ti OC windforce 3x and have the same problems. My other card is fine but this card fails almost instantly with computation error. I tried on both Ubuntu (319.76 and 331.20 drivers) and Windows with no joy so far.. I haven't tried lowering the clocks yet I am still troubleshooting it :(

Did you ever make any progress on this?

Yes, this card is working fine since I've lowered its memory clock to 2700MHz.
For example:Task 7614006, Task 7612456, Task 7611911
You've answered my unasked question: is this problem by design, or just my card is faulty?
It seems to be some problem with the design of the card.

Yes, this card is working fine since I've lowered its memory clock to 2700MHz.
For example:Task 7614006, Task 7612456, Task 7611911
You've answered my unasked question: is this problem by design, or just my card is faulty?
It seems to be some problem with the design of the card.

Yes it would be great if someone with another brand has the OC version and see how that goes. I had the plan to buy a EVGA 780Ti OC, but as my "normal" 780Ti heavily under performs yours with Win7. I decided to wait for the Maxwell.
But will try Linux first in the coming days.

---

Greetings from TJ

You've answered my unasked question: is this problem by design, or just my card is faulty?
It seems to be some problem with the design of the card.

From your perspective it is the design of the card and I agree with your perspective. If you ask the manufacturer and present all the evidence you have uncovered, their response might be like "It's a problem with the application, that card is for gaming applications where a few errors won't be noticed. It's not for data crunching that requires high precision and reliability."

Do you plan to RMA it? IIUC, you have had to downclock the memory below the frequency used on the standard model (not OC), yes?

---

BOINC <<--- credit whores, pedants, alien hunters