finally I was able to finish up my newest GPUGRID system. It's one of my old SETI systems, but I needed to convert it from USB risers to ribbon risers (and motherboard swap) for the increased PCIe bandwidth requirements here.

CPU: Intel Xeon E5-2630Lv2 (6c/12t,2.6GHz)
MB: ASUS P9X79 E-WS
RAM: 32GB (4x8) DDR3L-1600MHz ECC UDIMM
GPUs: [7] EVGA RTX 2070
PSUs: 1200w PCP&C + 1200W HP server PSU






went with a 2U supermicro active CPU cooler so I had enough room for the ribbon risers on the 2 GPUs above it. replaced the 60mm fan on it with a Noctua one since even at 20% speed the stock fan was very noisy. the Noctua fan doesnt cool as well as the stock server fan that came with it, but it's enough for this 60W chip (temps in the 50's @65% load) and it's a lot quieter.

Nice!

Impressive. Thanks for the photos and description.

Great Googly-Moogly! You rule, Retvari!
As Ray Wiley Hubbard says: "Some things here under Heaven are just cooler 'n Hell".

https://www.youtube.com/watch?v=o6C579hWdsI

Maybe name your creation something like Chico Grosso?

he was quoting my post but fixed the hyperlinks for the images. I forgot that this site breaks urls that already include http in the link in BBcode.

---

On April 20th 2020 | 19:48:27 UTC Ian&Steve C. wrote:

finally I was able to finish up my newest GPUGRID system...

On April 20th 2020 | 23:32:26 UTC, Retvari Zoltan kindly "revealed" the images for this system, previously not able to be seen in original post. (Thank you!)

I'm not letting pass away two comments about it:

- I can't imagine a cleaner way to build a system like this. It's not only a "processing bomb", but also it is elegantly resolved.

- In 24 hours processing, since its first valid result on April 20th 2020 at 19:11 UTC to today's same hour: it had returned 270 valid WUs, and 0 (zero) errored WUs: 100% success.
Well done! It has qualified its first working day with maximum score.

😳 Oops... sorry guys. I was so busy drooling over the rig that I forgot to read the header.
Anyway...
That has to be the best design yet for a crunching machine. You've changed my thinking about what my next opus should look like. Thank you for sharing your expertise with us.

😳 Oops... sorry guys. I was so busy drooling over the rig that I forgot to read the header.
Anyway...
That has to be the best design yet for a crunching machine. You've changed my thinking about what my next opus should look like. Thank you for sharing your expertise with us.

I took cues from my experience with cryptocurrency mining. this is a pretty common type of mining setup, and the frame was cheap ($35 on Amazon), though most people doing that will use USB risers instead of these ribbon risers, both for cost and power delivery reasons. I actually had most of this hardware already, I converted it from a USB riser setup to a ribbon riser setup for the transition to GPUGRID.

Some things to keep in mind if you want to do something like this:

1. Be mindful of the specs of your system, in particular the link width and generation of your PCIe slots. Some slots might be x16 size and fit a GPU, but only have electrical connections for a x8 or x4 link. some older boards might have a mix of PCIe 3.0 slots and pcie 2.0 (half speed) slots. Some motherboards may disable certain slots when others are in use. Pay attention to where the lanes are coming from and where the bottlenecks are. One common thing I see people overlook are the lanes coming from the chipset. It may be able to supply many lanes, but the chipset itself then only has a PCIe 3.0 x4 link back to the CPU. Read your motherboard manual thoroughly and lookup the specs of your components to understand how resources are allocated for your board.

2. GPUGRID requires a lot of PCIe bandwidth, and that likely scales with GPU speed. I've measured up to 50% of a PCIe use on a PCIe 3.0 x8 link, or up to 25% of a PCIe 3.0 x16 link with my RTX 2070 and 2080 cards. If you have a fast GPU, I would not put it on anything slower than PCIe 3.0 x4 (not common anyway) or PCIe 2.0 x8. slower GPUs might get by on slower links.

3. Be mindful of how much power you are pulling from the motherboard. When using USB risers you do not have to worry about this since power is supplied from external connections. But a setup like mine is pulling some of the GPU power from the motherboard slots. My motherboard has a 6-pin VGA power connection to supply extra power to the motherboard PCIe slots. PCIe spec for a x16 slot is up to 75W each! but most GPUs won't pull that much (except 75W GPUs that do not have external power!). If you plug GPUs directly to the motherboard, or use ribbon risers like I have, I wouldn't recommend using more than 3, maybe 4 GPUs (pushing it) unless you are supplying extra power to the board somehow.

4. if on a PCIe 3.0 link, you'll want to get higher quality shielded risers. PCIe 3.0 is a lot more susceptible to interference and crosstalk in the data lines than PCIe 2.0 or 1.0. The shielded risers are a lot more expensive though. I bought what I consider to be "good enough" knockoffs and they work perfectly fine, but were still $25 each, and that's kind of the low end of the pricing for 20cm long risers. out of 14 of these brand risers that I've purchased, 2 were defective (bad PCIe signal quality causing low GPU utilization and GPU dropouts) and needed to be replaced, so test them!

I took all of these things into account to end up with what you see here :)

I have power limited all GPUs on this system to 165W (175W stock), and at full tilt the system pulls 1360W from the wall.

---

Wow thanks a million! Info I will definitely use.

Im Running V8-XEON built in May 2013 by myself.
Board Intel Skulltrail D5400XS. (2PCI, 4 PCI-E x16, 4 FB-DIMM, Audio, Gigabit LAN)
RAM 16GB FB-DIMM, Quadchannel, Kingston
Processors 2 x E5405 Xeon. LGA771
Grafik 3x EVGA Geforce GTX Titan, (before 2 GTX 470 + 1 GTX 570)
PSU LEPA 1600W continuous Power, Gold certfied

If runs empty, V8-Xeon pulls 285 Watt out of the wall.
If it is crunching on all GPUs it pulls 860 - 890 Watt
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I shreddert 2 Super Flower PSU 1000W After one and a half year.
The machine is absolut stable. The Xeons run over years with 100% CPU usage.
Old but fine..

---

Im Running V8-XEON built in May 2013 by myself.

Thank you very much for sharing your setup.
Casually, my oldest self-made system currently in production is this one, built on March 12th 2013, and from then, it has experienced successive upgrades.
It has cathed my attention that your system was built the same year, and for that time it was a quite advanced configuration, based on a bi-Xeon E5405 processor.
One particular trick: When I'm interested on any Intel processor specifications, I enter on Google web search "ark E5405" (for example), and the first match leads to something like this...
Best regards,

One of my best headless computers was giving me a lot of trouble. Intermittently it would just stop crunching but still be powered up. Sometimes rebooting would get it going again, for a while anyway. So I put it on my desk with a monitor and started swapping power cables and RAM. Then it failed while I was watching and the GPU lights came back on even though I had --assign GPULogoBrightness=0 active in the NVIDIA X Server Settings startup program. At the same time the fans went to max even though it wasn't hot. So I pulled the GPU card to try another and the locking clip on the back of the PCIe socket popped off. I got a flashlight and was trying to figure out how to reinstall the clip when I noticed dirt inside the slot on the contacts. EVGA cards are notorious for having this clear fluid ooze out and sometimes drip down on the motherboard. I assume it's a thermal compound but don't know. It seems to be nonconductive and I've had it on the card contacts before without stopping it from working. I always wipe the card clean when I take them out. But this time I looked in the female PCIe slot with my magnifying glass and saw the contacts were coated with a dusty grime that was mixed in this mystery fluid. I took a toothbrush and cleaner the slot out and then blew it out. Put the same card back and so far so good. One more thing to add to the troubleshooting list.

Another computer would randomly turn off. Sometimes rebooting got it going for a while. When taking it apart the 8-pin CPU power connector to the motherboard had one corner pin disintegrate when I pulled the plug out. It had been running trouble free for years but the plastic of the connector got brittle. Cleaned the connector on the MB out with an X-ACTO knife, turned it upside down and blew it out. Installed a new CPU power cable and it's good as new again.

The "mystery" fluid that oozes out of graphics cards is the silicon oil separating from the thermal pads on the VRM and memory chips or the from the thermal paste.

I noticed dirt inside the slot on the contacts. EVGA cards are notorious for having this clear fluid ooze out and sometimes drip down on the motherboard. ...
It seems to be nonconductive and I've had it on the card contacts before without stopping it from working. I always wipe the card clean when I take them out. But this time I looked in the female PCIe slot with my magnifying glass and saw the contacts were coated with a dusty grime that was mixed in this mystery fluid.

1. Cards made by any manufacturer leak this silicon oil if they are used long and hot enough. The oil's viscosity is much less on higher temperatures, so the thermal pads / grease leaks noticeable quantity of it over time.
2. Conductivity is a tricky property. It varies greatly depending on the frequency of the electromagnetic wave. Think of vacuum, which is the best insulator, light and radio waves still can travel through vacuum, as their frequency is high enough. The state of the art computers operate at the microwave frequency (GigaHertz) range, so the grime which is non-conductive on DC acts as a dielectric of a capacitor, which "turns" into a conductor at high frequencies. As grime builds up over time, it's capacitance increases, thus it's conductivity at high frequencies increases, and when it's enough to push the PCIe bus out of specifications, the GPU won't work anymore (or it will run at PCIe2.0 instead of 3.0).

On May 1st 2020 | 1:14:34 UTC Aurum wrote:

One more thing to add to the troubleshooting list.

Thank to all of you for helping to complete with this topic this somehow never-ending list.

Moreover, "non-conductive" fluids have usually a very high "efficiency" in retaining dust particles, that sometimes are conductive themselves, or when dampening with environment humidity... And then (misterious) problem(s) may arise.

1. Silicone oil, that makes perfect sense. I was tempted to turn the MB upside down and spray the slots with either isopropyl alcohol or brake cleaner (methanol, toluene, acetone & CO2). Thought better of it since the solvents might dissolve the phenolic or epoxy resin and my board would disintegrate in my hands :-)

The brake cleaner can has a warning: Electrical Shock Hazard: this spray will conduct electricity, keep away from all electrical sources. Imagine doing your brakes one evening with the drop lamp hanging in the wheel well and it's the last thing you ever see.

But seriously, my toothbrush was not a very good way to clean goo out of a little slot. Suggestions welcome.

2. I'm running full steam on Rosetta CPU WUs waiting for OpenPandemics to kick off. Rosetta needs about 1 GB RAM per WU. I've been frustrated with MBs that won't run multiple sticks of RAM. I've been trying to get 64 GB on my 40t & 44t CPUs but only one MB lets me run 4 x 16 GB even when they're the same. I've tried every combination I could. E.g. MSI X99A Gaming 9 ACK, MSI X99A Raider & MSI X99A SLI will only acknowledge 3 x 16 GB. But an MSI X99 SLI Plus will run 4 x 16 GB.
At first I thought it was a bad slot but then I could move the third stick to other array and it would work: DIMM slots: 1, 5 & 3 or 1, 5 & 7.
I bought these cheap on fleaBay so maybe gamers overclocked and overtaxed them and I'm dealing with cripples.

3. The best MB I've got is the cheapest: Huananzhi X99-8M Gaming.

4. Some MBs just won't run the full range of CPUs their specs claim. E.g., my MSI X99A SLI Plus has chronic intermittent stoppages with a Xeon E5-2673 v4 SR2KE but runs flawlessly when replaced with an i7-5930k SR20R. Maybe it's too old and just can't lift the weight any more, like me :-)

I've always found that not reading all the sticks of installed memory on Intel LGA socket motherboards is due the cpu not being inserted correctly in the socket.

Or bent out of place LGA pins in the socket or are overheated. The cpu needs to be correctly located in the socket and also the locking clamp and cooler need to be installed to the correct torque specifications.

The pins in the LGA socket undergo both lateral and vertical position displacement when a cpu is installed. The LGA socket in location is actually very tight. The pins and pads on the cpu need to maintain 40 micron absolute positional location to be within spec.

The notches in the cpu substrate that locate the cpu in the socket allow for a lot of slop. When I don't read all the channels in the installed memory. I always undo the socket clamp and wiggle the cpu in the socket to allow the LGA socket pins to orbit around and hopefully mate with the corresponding pad on the substrate. Then reclamp and test for all the memory to be picked up again.

If you look at the more recent Intel cpu pin mappings, you will find the outside perimeter of pins often contain the memory channel assignments. And those pins undergo some of the greatest positional translation when under compression.

I'm seeing X99 motherboards (e.g. i7-6950X or Xeon E5-2673 v4) that use DDR3 RAM or both slots for DDR3 & DDR4 to be used in an either/or way. My first reaction is what a nice way to get some more mileage from my old DDR3.

Is there a technical reason that combining DDR3 memory with the X99 generation of CPUs will slow them down or disable some of their functionality???

This company has both: http://www.huananzhi.com/html/1/184/185/index.html

Is there a technical reason that combining DDR3 memory with the X99 generation of CPUs will slow them down or disable some of their functionality???

I've found the motherboard you probably are referring to: http://www.huananzhi.com/html/1//184/185/362.html

My experience with combo mainboards:
I asked myself the same question several years ago... but when I was trying to squeeze a bit more some DDR2 memory modules.
I bought this MSI G41M-P33 COMBO
It is still working at my system #540272
But I've found several drawbaks that had made me to think not to repeat this policy.
Now this motherboard is running with 8GB of DDR3 1333 MHz, for better performance than DDR2 800 MHz on CPU tasks.
But I've had to set DDR3 1333 MHz to run at 1066 MHz for system stability reasons. (1333 MHZ is specified as overclock for this G41+ICH7 chipset)
And recently I've found new Nvidia Turing based graphics cards not being compatible with this motherboard. System doesn't even start.
I'm running a GTX 950 on it for this reason...

On the other hand, your suggested motherboard has attractive specifications, and it has made me to enter in doubt about my previous determination 🤔️

Some other opinions or experiences would be welcome...

you need to use specific CPUs that support both DDR3 and DDR4, and none of them have official intel ark pages. there appear to only be a handful of them:

E5-2678v3
E5-2696v3
E5-2629v3
E5-2649v3
E5-2669v3
E5-2672v3
E5-2673v3

if you use an "offical" chip like a retail i7 chip or other retail Xeon Chip, you will probably only be able to use the DDR4 slots, since those chips don't have DDR3 controllers. or maybe they wont work at all in this board, I'm not sure.

---

Ok that explains the ad I saw with that list of CPUs but no explanation:
https://www.amazon.com/gp/product/B07XLH1WSF/ref=ox_sc_saved_title_3?smid=A2M1V9OGLU9XW&psc=1
This idea sounds too risky. I'm sticking with DDR4 MBs. Thx folks.