Average error % rate as of early 5/7/24 using last 3 data scrapes -

AVG – last 3: 15.0 all cards

8GB – last 3: 22.58 (2x 3070)
10GB – last 3: 13.74 (2x 3080)
12GB – last 3: 2.30 (1x 4070S)

Steve replied:

yes this make sense unfortunately. In the previous round of "inputs_v3**" it was calculating things incorrectly for any molecule containing Iodine. This is heaviest element in our dataset. The computational cost of this QM method scales with the size of the elements (it depends on the number of electrons). We are resending the incorrect calculations for Iodine containing molecules in this round of "v4" work units. Therefore the v4 set is a subset of the previous v3 WUs containing heavier elements, hence there are more OOM errors.

Any change in this situation?

I got my 12GB card back and my haphazard data collection seems to have it under a 9% error rate and with the very last grab showing 5.85%.

yes this make sense unfortunately. In the previous round of "inputs_v3**" it was calculating things incorrectly for any molecule containing Iodine. This is heaviest element in our dataset. The computational cost of this QM method scales with the size of the elements (it depends on the number of electrons). We are resending the incorrect calculations for Iodine containing molecules in this round of "v4" work units. Therefore the v4 set is a subset of the previous v3 WUs containing heavier elements, hence there are more OOM errors.

Download error causing the zip file to be corrupted because it is missing the end of file signature.

I was getting that on a Google Drive zip archive a couple of days ago. Switching browsers let me download the archive correctly so it would unpack.

to be expected with 8-10GB cards.

might get better context if you split the graphs up by card type. so you can see the relative error rate vs different VRAM sizes. I'm guessing most errors come from the 8GB cards.

between your systems and mine, looking at the error rates;

~23% of tasks need more than 8GB
~17% of tasks need more than 10GB
~4% of tasks need more than 12GB
<1% of tasks need more than 16GB

me personally, i wouldn't run these (as they are now) with less than 12GB VRAM.

Not sure why but...

Error rates seemed to start dropping after 5pm (23:00 Zulu) today. Overall error average since 2/11 across my 5 Nvid cards was 26.7% with it slowly creeping down over time. Early on a little bit of this was the result of lowering clocks to eliminate the occasional segfault (0x8b).

The average of the last two captures today across the 5 cards was 20.5%

For the last 6 hour period I just checked, my 10GB card average error rate dropped to 17.3% (15.92 & 18.7) and the 8GB card error rate was at 21.3%.

Skip

between your systems and mine, looking at the error rates;

~23% of tasks need more than 8GB
~17% of tasks need more than 10GB
~4% of tasks need more than 12GB
<1% of tasks need more than 16GB

me personally, i wouldn't run these (as they are now) with less than 12GB VRAM.

Guess I need to start a go-fund-me for some $600 12GB 4070 Super cards that I've been eyeing up ;-)

Skip

a $600 12GB Titan V is like 4x faster though.

other projects are a consideration of course.

Can you point me to someplace I can educate myself a bit on using Titan V cards for BOINC. I see some for $600 used on ebay. As u know there is no used market for used 'Super' cards yet. Did u mean 4x faster than a 4070 Super or than the 3070 I would replace with it?

Thanx, Skip

Guess I need to start a go-fund-me for some $600 12GB 4070 Super cards that I've been eyeing up ;-)

Skip

a $600 12GB Titan V is like 4x faster though.

other projects are a consideration of course.

between your systems and mine, looking at the error rates;

~23% of tasks need more than 8GB
~17% of tasks need more than 10GB
~4% of tasks need more than 12GB
<1% of tasks need more than 16GB

me personally, i wouldn't run these (as they are now) with less than 12GB VRAM.

I'm seeing a bunch of checksum errors during unzip, anyone else have this problem?

https://www.gpugrid.net/results.php?hostid=617834&offset=0&show_names=0&state=5&appid=

Stderr output
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
11:26:18 (177385): wrapper (7.7.26016): starting
lib/libcufft.so.10.9.0.58  bad CRC e458474a  (should be 0a867ac2)
boinc_unzip() error: 2

</stderr_txt>
]]>

The workunits seem to all run fine on a subsequent host.

I didn't find any of these in the 10GB 3080 errors that occurred so far today. Will check the 3070 cards shortly.

Skip

CUDA Error of GINTint2e_jk_kernel: out of memory

I'm seeing a bunch of checksum errors during unzip, anyone else have this problem?

https://www.gpugrid.net/results.php?hostid=617834&offset=0&show_names=0&state=5&appid=

Stderr output
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
11:26:18 (177385): wrapper (7.7.26016): starting
lib/libcufft.so.10.9.0.58  bad CRC e458474a  (should be 0a867ac2)
boinc_unzip() error: 2

</stderr_txt>
]]>

The workunits seem to all run fine on a subsequent host.

I didn't find any of these in the 10GB 3080 errors that occurred so far today. Will check the 3070 cards shortly.

Skip

CUDA Error of GINTint2e_jk_kernel: out of memory

its still an out of memory error. a little further up in the error log shows this:
"CUDA Error of GINTint2e_jk_kernel: out of memory"

so it's probably just running out of memory at a different stage of the task, producing a slightly different error, but still an issue with not enough memory.