The main problem with an ESXi home lab running 7/7 is usually one factor – Power consumption as a primary ongoing cost. Especially when you run a lab with a several hosts. Additional factors like cooling or noise can be usually solved by moving the server(s) to the separate room with natural airflow, but the power consumption is something that you have to plan ahead and you'll be dealing with during the lifetime of the lab. So in this post we'll look at some possibilities to build an Energy Efficient ESXi Home Lab Host. The post is part of an article series starting here.
With this in mind I started to look around for parts that would fit as foundation bricks for the homelab. I checked the latest power efficient CPUs, but ones which would be able to deliver enough power for an ESXi lab host, but the same time I wanted to break the 32/64Gb RAM barrier. Also IPMI was one of the features I wanted to get in my lab evein if Supermicro's IPMI is a bit tricky…. (Java….) . Supermicro with IPMI based boards provides Integrated IPMI 2.0 and KVM with dedicated LAN.
I wanted to get something which is:
- Easy to install ESXi (no hacking of drivers or VIBs)
- Break the 64 Gb RAM barrier so It's possible to go and have some nested ESXi and stay on single host.
- Well designed, robust case
- Power efficient PSU
- Power efficient platform (DDR4, Xeon E5-2630L)
So that's why I opted for socket 2011 R3 and low power Intel Xeon E5-2630L v3 CPU which has only 55W TDP. You should read my previous post which details the choice of the motherboard – Supermicro Single CPU Board for ESXi Homelab – X10SRH-CLN4F.
The CPU's details are at Intel ARK's website. The only difference between the E5-2630L V3 and the E5-2630 V3 is that the latter one is on higher frequency (85W TDP). You can see the differences when you compare both CPUs. The yellow color differentiate which characteristics are different between those two.
And here where you can see that the E5-2630L V3 runs 12 Degrees cooler, plus other characteristics like Intel VT-x, VT-d…
Future prove costs money…
We can talk about home labs for hours, days… and I think that with online offers from Ravello where you can run a couple of ESXi hosts on Amazon AWS for a fraction of a cost, we shall rather talk about home datacenters .. -:) But on the long run and for the look and feel (I still like to touch my hardware, to do my “geek” … ) is best to own the hardware. Yes the cost is a factor, but we spend money on cars, apartments, houses, so why not home datacenters?
DDR4 is still about 30% more expensive than equal capacity of DDR3, but planning ahead and shopping for strict minimum RAM (adding more RAM later) shall be a good budget strategy for now and on the long run. With price drops expected this year, additional RAM can get purchased later. The DDR4 pricing shall fall in second half of 2015 to become cheaper than DDR3 in 2016 where DDR3 will start fading out. However PC and server hardware is never future prove…
The RAM is really the first resource which gets exhausted and if you don't have enough slots or you have populated them with lower capacity memory sticks you're stuck. Currently 8, 16, 32 or 64 Gb DDR4 RAM sticks are available with lately some manufacturers putting in production sticks with capacity 128GB.
Supermicro has a range of motherboards based on this new 2011-R3 socket architecture where the one I picked up (X10SRH-CLN4F-O) was the one having 8 RAM slots and supporting up to 512 GB of LRDIMM modules.
The board also has Four 1Gb Intel NICs and integrated LSI 3008 SAS disk controller card (up to 8 SAS drives in RAID 0, 1, 5, 10). I reference other boards from Supermicro, which are based on the same socket, but are cheaper due the fact that they have less onboard NICs or the integrated controller isn't the one from LSI, but the Intel C612 which can still drive up to 10 SATA devices in RAID 0, 1, 5, 10. However for performance reasons some might just prefer LSI which is understandable.
The C612 is also present ont the X10SRH-CLN4F-O which has the built-in LSI 3008 for SAS (or SATA) attachments, so basically you can attach up to 10SATA HDDs to the C612. Those boards can also be using Xeon E5-1600 v3 family CPUs which carries less cache and less CPU cores (4cpu cores). The 4 core CPUs are obviously priced cheaper allowing to build system which is perhaps 2-3 times cheaper than a “fully loaded” system.
The internal LSI 3008 is recognized by ESXi 5.5 or ESXi 6.0 out of the box, and recently the card has been added to the VMware VSAN HCL – check the VSAN ready nodes PDF. It's a 12 Gbps SAS3/HBA/2 internal mini ports. You can update it with an IT or IR firmware. I did an update with an IT fimware, to be able to pass the disk directly to the ESXi.
It's 12GBs SAS storage controller card so take a complete advantage of such a card, you'll have have some SAS drives (SSDs???)… Not my case for the moment. On this I'll report later.
Colin has had a question about fitting a RAM in the slots close to the heatsing. Check my answer, and here I also shooted a photo showing how I mounted the heatsing and that the fan can move up and down in order to fit the RAM sticks which need to fit into the slots close to the CPU…
Here is the reference of what's possible to assembly with latest Supermicro 20011-R3 based boards. You can build a very powerful system loaded with lots of ram (8 available RAM slots). The choice of RAM is just question of money for the moment – if you take 16Gb or 32 Gb sticks. You can also start with 8Gb RAM sticks, but this would give you “only” 64 Gb of RAM.
The CPU list is the lower cost selection with 6 or 8 cores. I don't imagine to buy a CPU for $1500… just to build a lab system. That's why I did not reference the higher-end models. The RAM selection goes up to 32 Gb modules. Obviously you'll need perhaps some storage and USB stick to install ESXi.
|Item||Power Efficient ESXi Lab Host 2015 based on Haswell-EP architecture||Notes||Shop|
|Supermicro ATX DDR4 LGA 2011 X10SRH-CLN4F-O (with LSI 3008)||SAS, SATA, 4NICs||Amazon US|
|Supermicro ATX DDR4 LGA 2011 X10SRH-CF-O (with LSI 3008)||SAS, SATA, 2NICs||Amazon US|
|Supermicro ATX DDR4 LGA 2011 X10SRI-F-O (No LSI 3008 card on this board…)||SATA, 2NICs||Amazon US|
|Intel Xeon E5-2630L v3 – 8 cores, 1.8 GHZ (55W TDP)||20 Mb Cache||Amazon US|
|Intel Xeon E5-2630 v3 – 8 cores, 2.4GHz ( 85W TDP)||20 Mb Cache||Amazon US|
|Xeon E5-2620 v3 – 6 Cores, 2.40 GHz (85W TDP)||15 Mb Cache||Amazon US|
|Xeon E5-2609 V3 – 6 cores, 1.90GHz (85 TDP)||15 Mb Cache||Amazon US|
|Intel Xeon E5-2603 V3 – 6 cores, 1.6GHz (85 W TDP)||15 Mb Cache||Amazon US|
|Xeon E5-1650 v3 – 6 cores, 3.50 GHz (140W TDP)||15 Mb Cache||Amazon US|
|Supermicro Certified MEM-DR432L-SL01-LR21 Samsung 32GB DDR4-2133 4Rx4 LP ECC LRDIMM Memory||32 Gb||Amazon US|
|Samsung DDR4-2133 16GB/2Gx72 ECC/REG CL15 Server Memory (M393A2G40DB0-CPB0)||16 Gb||Amazon US|
|Supermicro Certified MEM-DR416L-CL01-ER21 Micron Memory – 16GB DDR4-2133 2Rx4 ECC REG RoHS||16 Gb||Amazon US|
|Hynix DDR4-2133 16GB/2Gx72 ECC/REG CL13 Hynix Chip Server Memory||16 Gb||Amazon US|
|Samsung DDR4-2133 8GB/1Gx72 ECC/REG CL15 Server Memory (M393A1G40DB0-CPB)||8 Gb||Amazon US|
|Supermicro Certified MEM-DR480L-CL01-ER21 Micron Memory – 8GB DDR4-2133 1Rx4 ECC REG RoHS||8 Gb||Amazon US|
|Supermicro certified MEM-DR480L-HL01-ER21 Hynix Memory – 8GB DDR4-2133 1Rx4 ECC REG RoHS||8 Gb||Amazon US|
|FSP Group 450W Modular Cable Power Supply 80 PLUS Platinum AURUM 92+ Series (PT-450M)||450W||Amazon US|
|FSP Group 550W Modular Cable Power Supply 80 PLUS Platinum AURUM 92+ Series (PT-550M)||550W||Amazon US|
|FSP Group 650W Modular Cable Power Supply 80 PLUS Platinum AURUM 92+ Series (PT-650M)||650W||Amazon US|
|Noctua i4 CPU Cooler for Intel Xeon CPU_ LGA2011, 1356 and 1366 Platforms NH-U12DXi4 (support the Narrow ILM socket)||Amazon US|
|Noctua i4 CPU Cooler for Intel Xeon CPU_ LGA2011, 1356 and 1366 Platforms NH-U9DXi4 (support Narrow ILM and Square ILM)||Amazon US|
|Fractal Design Define R4, Black Pearl||Amazon US|
|Fractal Design Define R5, Black Pearl, ATX Mid Tower, Case (Black)||Amazon US|
Energy Efficient ESXi Home Lab
- Building Power Efficient ESXi Server – Start with an Efficient Power Supply
- ESXi Home Lab – Get a quality and comfortable case
- Supermicro Single CPU Board for ESXi Home lab – X10SRH-CLN4F
- Supermicro Single CPU Board for ESXi Homelab – X10SRH-CLN4F – part 2
- Supermicro Single CPU Board for ESXi Home lab – Upgrading LSI 3008 HBA on the X10SRH-CLN4F
- Building Energy Efficient ESXi Home Lab – parts list [this post]
- Homelab – Airflow Solar System