I'm getting this again...It seems to happen after 2-3 days

Going to try vista.

Maybe the developers can take a look if there are some memory leaked.

Today I got 9 WUs crashed on XP Pro 64 with my GTX260²:

Cuda error in file 'deviceQuery.cu' in line 59 : out of memory.

Now I get no new WUs for today, also not after reboot, detach and re-attach.

Any hint to avoid this situation?

---

Today I got 9 WUs crashed on XP Pro 64 with my GTX260²:

Cuda error in file 'deviceQuery.cu' in line 59 : out of memory.

Now I get no new WUs for today, also not after reboot, detach and re-attach.

Any hint to avoid this situation?

There is no known root cause, or corrective action at this time. You will have to wait 24 hours for your daily quota to be reset, in order to get more tasks. Bottom line: When this problem happens, it takes a machine down for at least a day.

---

Reno, NV
Team: SETI.USA

I'm wondering: is the GPU memory actually blocked due to some memory leak?

Maybe test it like that: run 3DMark 2006 (should need some mem) after boot, note the score and rerun after you got this error. If the scores are identical (+/- 1%) I'd say the GPU memory is still available and something else is going on. There are also tools which can show you the utilization of GPU mem, I just don't know if they're any good.

MrS

---

Scanning for our furry friends since Jan 2002

... run 3DMark 2006 (should need some mem) after boot, note the score and rerun after you got this error. If the scores are identical ...

3DMark06 is not running under XP 64 bit, need incompatible DirectX Version.

---

try upgrading DirectX9 to the latest August 2008 build:
http://www.microsoft.com/downloads/details.aspx?familyid=2DA43D38-DB71-4C1B-BC6A-9B6652CD92A3&displaylang=en

I've used Rivatuner instead of 3DBenbch06 to check the used memory of the graphic card under XP 64 bit. There is a plugin named Vidmem.dll in the installation of the Rivatuner you can use. After a fresh reboot the card use 70.13 MB of Video RAM. After 12 hours runtime the card use 138.26 MB. I had quit the BOINC manager and restart him, the card used then 206.32 MB. I checked again and made a second restart of the BOINC manager and the card used 274.82 MB. You can provoke the memory fault with repeated restart of the BOINC manager, then you surely get the message:

Cuda error in file 'deviceQuery.cu' in line 59 : out of memory

after some restarts or latest after 3 or 4 days of running time.

My tip for XP 64 user: Restart your XP 64 often while using CUDA or change to Vista 64 or Linux, they don't have this problem.

---

I've used Rivatuner instead of 3DBenbch06 to check the used memory of the graphic card under XP 64 bit. There is a plugin named Vidmem.dll in the installation of the Rivatuner you can use. After a fresh reboot the card use 70.13 MB of Video RAM. After 12 hours runtime the card use 138.26 MB. I had quit the BOINC manager and restart him, the card used then 206.32 MB. I checked again and made a second restart of the BOINC manager and the card used 274.82 MB. You can provoke the memory fault with repeated restart of the BOINC manager, then you surely get the message:

Cuda error in file 'deviceQuery.cu' in line 59 : out of memory

after some restarts or latest after 3 or 4 days of running time.

My tip for XP 64 user: Restart your XP 64 often while using CUDA or change to Vista 64 or Linux, they don't have this problem.

My hunch of a memory leak seems true. Does Berkeley knows about this bug?

Anyways, changed my system to vista and apparently no issues with not enough memory but it took 600-700 secs longer and from 24ms/step to 25ms/step

I thought vista is 2000secs faster?

Cool, finally we could grab the problem by the horns! :D

> I thought vista is 2000secs faster?

That was my guess with the first reasonable performance data in. But quite a bit has changed since then, client and driver-wise.. and I could have been wrong from the beginning. Though the comparison would probably only be valid between XP32 and Vista anyway, because XP64 uses a different driver than XP32.

MrS

---

Scanning for our furry friends since Jan 2002

I have been rebooting my machine daily, to try to avoid this problem. It seemed to help for a while. It didn't happen for a couple of weeks anyway. Then last night, it happened again. It's with 6.3.19. Now I have to wait 24 hours until I can get more tasks to crunch.

---

Reno, NV
Team: SETI.USA

I have been rebooting my machine daily, to try to avoid this problem. It seemed to help for a while. It didn't happen for a couple of weeks anyway. Then last night, it happened again. It's with 6.3.19. Now I have to wait 24 hours until I can get more tasks to crunch.

Do you leave tasks in memory or are the removed when suspened/switch to waiting ?

I don't know but if removed, that may help.

Another note, this may be related ?

There are some recent improvments goigg into the next client(s) to help detect memory leaks.

I'm not exactly sure all what they are doing on this.

See changeset
16357
client: include precompiled header in rr_sim.cpp so memory leak detection will work.

Well, i got hit hard again by these errors. About a week ago I reported them in another thread and was directed here. After reading suggestions here, I rebooted the offending machine and it started to play nicely, until today. During that period I also upgraded BOINC client to 6.3.21, which is finnaly assigning correct number of tasks to CPU and GPU cores. Next I'll try upgrading graphics drivers to see if that sorts this problem.

Greetings,

---

I got hit by memory errors again on 2 wu's.

6.3.21
XP 64
177.84 driver

<core_client_version>6.3.21</core_client_version>
<![CDATA[
<message>
Incorrect function. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
# Using CUDA device 0
Cuda error in file 'deviceQuery.cu' in line 59 : out of memory.

I performed a re-boot. Hope that fixes it for a while..

EDIT: After the re-boot, I recieved 2 new tasks, and 1 is running fine now...

---

Consciousness: That annoying time between naps......

Experience is a wonderful thing: it enables you to recognize a mistake every time you repeat it.

Sorry, now I see already a thread about this.

I have a GTX280/9800GT (no SLI/not overclocked) and I am using 6.3.21 client with XP Pro 32bit and had this second times in a few days. Before I was using 6.3.19 and never had this problem!
My system is not running 24/7, only crunching when I am at home. Normally it takes me 2-3 days to finish.
It seems there is no solution for now?


<core_client_version>6.3.21</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
# Using CUDA device 1
# Device 0: "GeForce GTX 280"
# Clock rate: 1404000 kilohertz
# Device 1: "GeForce 9800 GT"
# Clock rate: 1620000 kilohertz
MDIO ERROR: cannot open file "restart.coor"
# Using CUDA device 0
# Device 0: "GeForce GTX 280"
# Clock rate: 1404000 kilohertz
# Device 1: "GeForce 9800 GT"
# Clock rate: 1620000 kilohertz
Cuda error: Kernel [reduce4_kernel] failed in file 'reduction.cu' in line 143 : unspecified launch failure.


<core_client_version>6.3.21</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
# Using CUDA device 0
# Device 0: "GeForce GTX 280"
# Clock rate: 1404000 kilohertz
# Device 1: "GeForce 9800 GT"
# Clock rate: 1620000 kilohertz
MDIO ERROR: cannot open file "restart.coor"
Cuda error: Kernel [angle_kernel] failed in file 'bonded.cu' in line 547 : unspecified launch failure.

---

Your error does not show the "out of memory" line, which this thread is about ;)

MrS

---

Scanning for our furry friends since Jan 2002

Sorry! But is there anyone who had this error or knows about it??

---

Hi,

I have a systems running with GTX 9800+ 512MB. Running with Vista64 and BOINC 6.3.21...

I have this one since Oct 28th:

<core_client_version>6.3.21</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
# Using CUDA device 0
Cuda error in file 'deviceQuery.cu' in line 59 : initialization error.

</stderr_txt>
]]>

I have updated to the latest nvidia driver (178.08) BOINC to the latest Version and yesterday a bigger PSU (550W).

The error is still the same. No probs before with BOINC 6.3.19 and appl. 6.42.

What I have tested is restarting BOINC before getting one WU (settings to get only 0,05 days work, don't end in the 4WU per day limit...) and in most of the cases the WU is done.

This could only be a temporary solution!

Has somebody an idea about this?

---

Do you mean you're getting it with every unit (if you don't baby-sit BOINC) or do you get the error occasionally?

MrS

---

Scanning for our furry friends since Jan 2002

You can have a look an it :)

http://www.ps3grid.net/results.php?hostid=9664

The successful results are by baby-sitting GPUGrid.

This is really not all that much different than what I posted in the 6.3.21 thread and never got a response on....that is my Linux 8.04 Q9550 9800GT will not d/l work under 6.3.21. I have to revert back to 6.3.19 get work and then re-install 6.3.21 to run. This is not a permanent solution. I also get major screen flicker in the message tab.I get the normal out of work message in 6.3.21...only cell proccesor work available.