Message boards : Number crunching : Error invoked kernel
| Author | Message |
|---|---|
|
# Engine failed: Error invoking kernel: CUDA_ERROR_LAUNCH_FAILED (719) | |
ID: 54377 | Rating: 0 | rate:
 /   Reply
Quote
| |
 Retvari Zoltan Send message Joined: 20 Jan 09 Posts: 2343 Credit: 16,206,655,749 RAC: 261,147 Level  Scientific publications   | |
# Engine failed: Error invoking kernel: CUDA_ERROR_LAUNCH_FAILED (719) You have also # Engine failed: Particle coordinate is nan which is usually the result of to much overclocking, or your card has a failing memory chip.NaN on Wikipedia | |
| ID: 54378 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Thank you, I will lookup und let run the cards still in standard frequency. | |
| ID: 54383 | Rating: 0 | rate:
/ Reply
Quote
| |
|
I would try a app_config.xml | |
| ID: 54411 | Rating: 0 | rate:
/ Reply
Quote
| |
|
admin pleasedelete the multiple messages | |
| ID: 54422 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa  Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level  Scientific publications  | |
|
Retvari Zoltan said: # Engine failed: Error invoking kernel: CUDA_ERROR_LAUNCH_FAILED (719) I don't think limiting your GPU usage will solve your errors. it's not a matter of percentage, but a matter of frequency that is causing tasks to fail when the wrapper starts the GPU. Some appear to be easier to crash than others. GPUGRID WUs are the most sensitive tasks I've seen to processor overclocking errors and I had to slow my GTX 1060 down when I came here even though it ran games and other BOINC projects OK. My errors were hit and miss like yours only not as many. they usually occurred at ~30 sec. Your base clock speed is 1000MHz per [url]https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-titan-x/specifications [/url] | |
| ID: 54427 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Aurum  Send message Joined: 12 Jul 17 Posts: 401 Credit: 16,755,010,632 RAC: 220,113 Level Scientific publications  | |
I would try a app_config.xmlYours looks exactly like mine except I only run one GG WU per GPU. BOINC always says that when you don't have a acemd3 WU downloaded. Wow! Eight duplicates. My record was three. No worries, it happens to us all and I don't know why. | |
| ID: 54431 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,828,866,430 RAC: 19,722,064 Level  Scientific publications   | |
|
Just got a "# Engine failed: Particle coordinate is nan" error on WU 19441088 - as have all my wingmates. | |
| ID: 54432 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Keith Myers  Send message Joined: 13 Dec 17 Posts: 1340 Credit: 7,653,273,724 RAC: 13,371,735 Level Scientific publications | |
Just got a "# Engine failed: Particle coordinate is nan" error on WU 19441088 - as have all my wingmates. I concur. Not all NaN errors are the result of a misbehaving card. Sometimes the task is just badly formatted. | |
| ID: 54433 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Retvari Zoltan Send message Joined: 20 Jan 09 Posts: 2343 Credit: 16,206,655,749 RAC: 261,147 Level Scientific publications | |
Just got a "# Engine failed: Particle coordinate is nan" error on WU 19441088 - as have all my wingmates. This task is a 2ph7A01_348_3-TONI_MDADpr4sp-7-10-RND7696 So it's the 7th of 10 workunits. Perhaps the previous host made an error, which resulted in a permanent NaN error on all hosts. | |
| ID: 54434 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
Just got a "# Engine failed: Particle coordinate is nan" error on WU 19441088 - as have all my wingmates. Hi Richard Haselgrove, Your 1660-S is not overclocked; Correct? It looks like we'll have to wait until that WU reaches the Apr 29 deadline on iBat's machine (after viewing it's task status), before seeing if it crashes again. I've been getting more tasks lately which have crashed on 1 or 2 other hosts before being sent to mine. I noticed several error prone machines were Science United and a few were grcpool hosts. Fascinating. | |
| ID: 54441 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
This task is a 2ph7A01_348_3-TONI_MDADpr4sp-7-10-RND7696 Zoltan, I think you meant to write 8th of 10, as the first one is always named 0-10. Or am I confused? 🤔 | |
| ID: 54442 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Retvari Zoltan Send message Joined: 20 Jan 09 Posts: 2343 Credit: 16,206,655,749 RAC: 261,147 Level Scientific publications | |
This task is a 2ph7A01_348_3-TONI_MDADpr4sp-7-10-RND7696 You're right, it's the 8th. Probably the host doing the 7th piece made an error. (that's what I should post to correctly include the number 7 in my post) | |
| ID: 54444 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Richard Haselgrove Send message Joined: 11 Jul 09 Posts: 1620 Credit: 8,828,866,430 RAC: 19,722,064 Level Scientific publications | |
Hi Richard Haselgrove, Your 1660-S is not overclocked; Correct? Correct. I gave that machine a complete motherboard/CPU/RAM transplant at the end of January, and fitted two brand-new, identical, 1600-S GPUs. It's in a high airflow case with a Corsair modular power supply. I can do basic hardware work on computers, but I'm not a hardware specialist, so I bought the motherboard bundle pre-assembled and tested from a local trade supplier, with CPU cooler ready attached. It ran on SETI until that project stopped sending out new work (bad timing on my part!), and started working here at the beginning of April. Application details Tasks I think 4 errors, with 1167 completed tasks, indicates the machine is basically healthy. Two of the other errors reached the full 8 failures on all machines that attempted them, and one seems to have been a ghost that I never received. | |
| ID: 54447 | Rating: 0 | rate:
/ Reply
Quote
| |
|
robertmiles Send message Joined: 16 Apr 09 Posts: 503 Credit: 755,370,933 RAC: 212,472 Level  Scientific publications  | |
admin pleasedelete the multiple messages If you try soon enough, you should be able to do part of the work by editing all but one of them down to just one character. Making most of them the same single character is likely to trigger an automatic process for hiding duplicate messages. | |
| ID: 54455 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
|
Richard, you have less errors than I do, I think. | |
| ID: 54491 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
...And sure enough, it lasted no longer than 16 seconds before it choked on everybody's hosts. Heinz is getting errors at later stages of the tasks than we are experiencing them when running bad WUs. https://www.gpugrid.net/results.php?hostid=159065 I have had errors before that were caused by running short of memory, although I see that is not a problem in Heinz's case. I had 7 Rosetta threads and two GPUGRID wrappers running in 8 GB of ram with 8182MB swapfile. Every time a Rosetta COVID task would suddenly hog memory, one of the wrappers would give a message that an output file could not be found (can't remember which) and throw an error. I've since increased to 12GB of ram and solved that issue. I had a PSU failure on my fast host today (a recycled 600W cheapo from the days of molex connectors) and it makes me wonder if Heinz might have power issues with his 3 GTX Titans in one host. Just a thought, but if they're clocked higher than factory specs IMHO that is the first thing to suspect. 🤔 | |
| ID: 54501 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Keith Myers Send message Joined: 13 Dec 17 Posts: 1340 Credit: 7,653,273,724 RAC: 13,371,735 Level Scientific publications | |
It ran on SETI until that project stopped sending out new work (bad timing on my part!) Ha ha LOL. I did the same thing. Rebuilt completely/upgraded the 3900X host for Seti and put it back online a few days before Seti pulled the plug. Now it just sits there, idle, looking pretty. | |
| ID: 54502 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
I have had errors before that were caused by running short of memory, although I see that is not a problem in Heinz's case. I had 7 Rosetta threads and two GPUGRID wrappers running in 8 GB of ram with 8182MB swapfile. Every time a Rosetta COVID task would suddenly hog memory, one of the wrappers would give a message that an output file could not be found (can't remember which) and throw an error... ...Which made me curious what that particular host is running on the CPU. I see that _heinz has recently switched to running World Community Grid- https://boinc.netsoft-online.com/e107_plugins/boinc/get_user.php?cpid=5e024335320e436c4d050e073963e326 Does anyone here know how much memory those tasks use? I found that LHC@home tasks were too memory hungry to run at 2GB of ram per thread. That might be an issue here. | |
| ID: 54503 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Keith Myers Send message Joined: 13 Dec 17 Posts: 1340 Credit: 7,653,273,724 RAC: 13,371,735 Level Scientific publications | |
|
I recently discovered a website with the ability to dive deep into the data for all the BOINC projects. This page has the RAM requirements for all the projects cpu apps. | |
| ID: 54504 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Retvari Zoltan Send message Joined: 20 Jan 09 Posts: 2343 Credit: 16,206,655,749 RAC: 261,147 Level Scientific publications | |
I recently discovered a website with the ability to dive deep into the data for all the BOINC projects. This page has the RAM requirements for all the projects cpu apps.I wondered how did they get their data. I realized the answer when I browsed to the root of this site: http://wuprop.boinc-af.org/ This is actually a BOINC project collecting data about the apps of other BOINC projects as you run it with your other projects. Nice! It's 10 years old, and I can't recall I heard about it. I'm shocked. | |
| ID: 54505 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Aurum Send message Joined: 12 Jul 17 Posts: 401 Credit: 16,755,010,632 RAC: 220,113 Level Scientific publications | |
I recently discovered a website with the ability to dive deep into the data for all the BOINC projects. This page has the RAM requirements for all the projects cpu apps.Nice page. Pretty close to my values. I always leave some head room, e.g. LHC ATLAS needs 2 GB. Rosetta is a problem as they stuff every project they have into one queue. Some need much more RAM than others so this is clearly an average that would benefit from knowing its standard deviation. Most of the time 0.8 GB is enough but a couple of projects use a good bit more so reserve a minimum of 1 GB and it'll come out in the wash. | |
| ID: 54508 | Rating: 0 | rate:
/ Reply
Quote
| |
ServicEnginIC Send message Joined: 24 Sep 10 Posts: 581 Credit: 9,770,362,024 RAC: 21,500,013 Level Scientific publications | |
I recently discovered a website with the ability to dive deep into the data for all the BOINC projects. This page has the RAM requirements for all the projects cpu apps. Thank you for sharing this! Not only did I like it, but I also joined WUProp@Home and I'm running my first task... Its seems to be very low resource demanding: Application: Data collect version 4 4.25 (nci) | |
| ID: 54510 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
|
Unfortunately, it appears that _heinz has given up on GPUGRID, looking at his task list. Regrettable as he is a veteran cruncher. You have also | |
| ID: 54568 | Rating: 0 | rate:
/ Reply
Quote
| |
|
Pop Piasa Send message Joined: 8 Aug 19 Posts: 252 Credit: 458,054,251 RAC: 0 Level Scientific publications | |
|
ServicEnginIC wrote Thank you for sharing this! That goes for me too! 👍👍 The more of us that contribute, the more accurate the statistics. "(non-CPU-intensive)" = extra BOINC credit without stopping anything else to do it... Genius! And, we get more insignias apestosas, hombres. 🥇😊 | |
| ID: 54570 | Rating: 0 | rate:
/ Reply
Quote
| |
|
[AF] fansyl Send message Joined: 26 Sep 13 Posts: 20 Credit: 1,714,356,441 RAC: 0 Level  Scientific publications | |
I recently discovered a website with the ability to dive deep into the data for all the BOINC projects. This page has the RAM requirements for all the projects cpu apps.I wondered how did they get their data. I realized the answer when I browsed to the root of this site: WUProp is a tool developped by an Alliance Francophone's developper. Feel free to come on our forum, and especially on the dedicated thread : https://forum.boinc-af.org/index.php?topic=3438.new;topicseen#new ____________ | |
| ID: 54572 | Rating: 0 | rate:
/ Reply
Quote
| |
Message boards : Number crunching : Error invoked kernel
