It's time to do another home lab build! This time I want to break the RAM barrier for Homelab white box. In the past when I use to build a white boxes at the end I got always stuck not by CPU performance, but RAM limit. With 24Gb, 32Gb or 64Gb RAM respectively. No matter how hard you seek for a board when the RAM demands grow by running more and more VMs, the only way to go is to purchase a server motherboard, even if now with Haswell-EP you can go up to 64 Gb of RAM on desktop boards, it's still not enough for me and for majority of virtualization folks out there. I wanted to get something “future prove”, something that would still be relevant from 3 to 5 years from now on.
I also wanted to build a very Power efficient ESXi Home lab server which will not only be evolutive from the RAM perspective, but also from the storage or network perspective. If you're going after hyper converged lab architecture than this build might be a first part of the puzzle where you can add 2-3 other hosts later.
Imagine to pick up an Intel Ethernet Converged Network Adapters XL710 10/40 GbE and 10 or 40 Gb switch and you get a very efficient and performant VSAN cluster homelab. You might also want to build a All-In-One box and run everything nested, but you'll still need as much ram as possible. The Supermicro board can go up to 512 Gb of RAM.
Sky WAF is the limit …
The new Xeon E5 V3 platform is very interesting as the C612 Intel chipset can handle 10 SATA drives. By picking the Supermicro X10SRH-CLN4F board the onboard LSI 3008 SW based controller can handle 8x SAS3 (12Gbps) in RAID 0, 1, 10. If you don't want to go the “SAS” way you can use SATA on the same controller. Coupled with an energy efficient Xeon E5-2630L V3 which has 55W TDP only, it's possible to build a very efficient home server. The Xeon E5 2630L V3 is a 1.8GHz 8 core CPU and has all specs like the higher frequency brothers including 20Mb cache, Intel VT-X …
The Supermicro X10SRH-CLN4F board is single CPU board based on 2011-3 socket supporting latest Intel Xeon E5-26xx V3 or E5-16xx v3 series. The board has also built-in LSI SAS 3008 controller and Quad port Intel Intel® i350 Gigabit LAN ports. No 10GbE, but 4 gigabit ports allows to get started with. (Note: The LSI 3008 is
not on VMware VSAN hcl, but and it has a queue depth 600 which is fine for VSAN).
When few years back I was building one of my first Whiteboxes, the built-in LAN ports are sometimes not recognized, like the Realtek 8169 Gb LAN NIC. But then again you must use some dual or quad add-on NICs to have at least 3 NICs in each box…
There are 8 RAM slots which allow boosting the single server up to a total of 512 GB of RAM when using DDR4 LRDIMMs. Or when used with lower capacity DDR4 1.2V-2133 RAM ECC sticks (8 or 16Gb RAM sticks) you can go up to 64Gb or 128Gb of RAM respectively.
The board has an IPMI for remote management and Supermicro is one of the manufacturers which do not require an additional license to use this feature (Fujitsu, HP, Dell, do you hear me?). This is one of the reasons that Supermicro boards are so popular and that's why I decided to take the plunge and start to build a ESXi platform based on this board.
The X10SRH-CLN4F Specs:
- Single socket R3 (LGA 2011) supports Intel® Xeon® processor E5-2600 v3 and E5-1600 v3 family
- Intel C612 chipset to manage SATA storage – 10x SATA3 (6Gbps)
- Up to 512GB ECC DDR4 2133MHz LRDIMM – 8x DIMM slots (512 Gb RAM with 8x 64Gb mem sticks)
- 1 PCI-E 3.0 x4 (in x8), 1 PCI-E 3.0 x8 (in x16), 2 PCI-E 3.0 x8, 1 PCI-E 2.0 x2 (in x4), 1 PCI-E 2.0 x4 (in x8)
- Intel i350-AM4 Quad port GbE LAN
- 8x SAS3 (12Gbps) via LSI 3008 SW controller; RAID 0, 1, 10
- 1x VGA, 2x COM, 1x TPM
- 4x USB 3.0 ports, 8x USB 2.0 ports
- 2x SuperDOM with built-in power
- Intelligent Platform Management Interface v.2.0 (IPMI 2.0) with virtual media over LAN and KVM-over-LAN support
The Format and compatibility
There is also X10SRG-F which is listed as compatible with ESXi 5.5U2 on the Supermicro website, has the same set of components, but does uses “only” the C612 chipset for SATA drives. So there is no SAS with an LSI 3008 which provides SAS data transfer rates of 12, 6, and 3Gb/s per lane, and SATA at 6 and 3Gb/s data transfer rates per lane. But the format is a proprietary format (not ATX).
I wanted ATX format so I could get my own choce for a case. In fact I wanted to get Fractal Design Define R5, Black Pearl, ATX Mid Tower, Case with easy access to disks and easy opening, when testing.
The board has a standard 8x 4pin PWM fan connectors on the front and back sides. 1U, 2U, 3U or 4U/Tower chassis configurations are possible. If you want to build the system by using the Supermicro cases you can, but in this case you'll also have to be ready to pick a power supply at the same time. I picked 92+ platinum powersuply (92% efficiency) from FSP Group.
Expansion Slots and I/O of the X10SRH-CLN4F
Very large number of PCI Express lanes giving options for expansion.
- PCI-E lanes – x4 and three x8 v.3.0, x4 and x8 v.2.0
- USB 3.0 ports – 2 rear, 2 internal header.
- SuperDOM support
- SATA Disk-on-Module (DOM) – The ports are yellow color on the pic. above…
- Serial port (for UPS for example)
- IPMI for remote management
- 4 Intel i350-AM4 gigabit ports
- 1 VGA
Make sure that you get a narrow ILM socket compatible cooler as the square Socket 2011 coolers will not fit! Check the list of compatible hardware on the main page – Building Energy Efficient ESXi Home Lab.
The Haswell based platform is third generation E5 that has been announced as low power consumption optimized with large number of PCIe lanes and USB 3.0 ports. Coupling that with DDR4 which runs at 1.2V compared to 1.35 V for DDR3 are faster and more power efficient.
The Xeon E5-2600 V3 family does support up to 14 cores per socket and 28 threads coupled with fast 35 MB LLC (Last Level Cache) and having both the Intel Hyper Threading and Turbo Boost technologies.
Update: The Low power E5-2630L v3 (which I chosed) and E5-2650L v3 run at 1.8 GHz. They have 8 and 12 cores, and they have 55 Watt and 65 Watt TDP. However the latter one is inaccessible due to the high price.
System build around this architecture will cost you less on a long run when used at home 7/7 as a storage or virtualization server. The platform shall still be relevant not in 1-2 years from now on but perhaps 4-5 years from now on as well.
Update: The built-in LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express which is on VMware HCL. The LSI 3008 is 12 GbpS SAS where the H310 I used previously is 6Gbps only… You can check yours too – how to check queue depth of your storage adapter
Update: New esx cli commands shows also which controllers are used by vSAN cluster and what's the queue depth. It has been added in vSAN 6.6 (ESXi 6.5).
Just connect to any ESXi part of the vSAN cluster, via putty, and enter:
esxcli vsan debug controller list
You'll get this view…
Energy Efficient ESXi Home Lab:
- Efficient Home Server – Start with an Efficient Power Supply
- ESXi Home lab – Get a quality and comfortable case
- Supermicro Single CPU Board for ESXi Homelab – X10SRH-CLN4F – this post
- Supermicro Single CPU Board for ESXi Home lab – X10SRH-CLN4F – Part 2
- Supermicro Single CPU Board for ESXi Home lab – Upgrading LSI 3008 HBA on the X10SRH-CLN4F
- Building Energy Efficient ESXi Home Lab – parts list [this post]
- Homelab – Airflow Solar System