We continue to explore VMware technologies and teach some of the new terminologies that are relatively new for some, but quite old for others. When you think that 3 years ago only, VMware was rolling out vSAN 5.5 and now, they're just announced the 6th iteration of their popular hyper-converged software solution – VMware vSAN 6.6. Oh dear, time just goes so fast in the IT industry and so the terminology. Everything is evolving really fast.
VMware vSAN has introduced this concept of vSAN disk group. Compared to a storage device such as SAN or NAS, you have usually a certain number of spinning flash disks inside, you create your raid group and this is it.
The shared storage architecture with SAN or NAS devices isn't dead and it's deployed, but when you see the simplicity of managing storage vSAN brings in, you can't consider learning vSAN.
What is VMware vSAN Disk Group?
Each hosts participating in a vSAN cluster with local storage has the local disks configured in disk group(s). It's kind of a container, where the SSD for cache and capacity devices (SSD or HDDs) are in relation. The VMs are placed on the capacity tier but accelerated through the SSD cache tier. The SSD or PCIe flash device that is used for that I/O acceleration is the one which is in the same disk group as the capacity devices on which the VM is placed.
On each ESXi host, disks are organized into disk groups. A disk group is a main unit of storage on a host. Each disk group includes one SSD and one or multiple HDDs (magnetic disks). Up to 6 seven magnetic disks.
All-Flash or Hybrid…
VMware vSAN uses an aggregation of disk groups to create a single datastore when you enable vSAN.
In the disk group, the SSD primarily serves as a read cache write buffer, while the HDDs are used for permanent storage. Typically, a higher SSD to HDD ratio, both in size and quantity, improves performance. Depending on the mode you select when enabling Virtual SAN on a cluster, you can use different ways to organize disks into groups.
What's the maximum number of Disk Groups per host?
It is possible to have five disk groups per host. Each of these disk groups can contain a maximum of one SSD and seven capacity devices.
- 5 Disk groups
- One disk group = 1 SSD (cache) + 7 capacity devices (SSD or HDD)
With multiple disk groups per host you'll get overall more performance and more disk space, but perhaps you might think that when you think about failures, having multiple disk groups per host does not play to your advantage.
The opposite is true. VMware vSAN is pretty clever, as the system make sure that components of the same object part of the RAID-1 tree will not reside on the same host. So you don't have to be worried about what's gonna happen when you lose a full host.
On the other hand, losing one host will probably trigger some big rebalancing and recomposing activity if the host does not come back before 60 min. After 60 min, the vSAN starts to recreate missing components.
Disk groups are the base building blocks of VMware vSAN. Perhaps a good way to start is to consider a single disk group per host? Knowing that one disk group can have up to seven capacity devices, this gives you an opportunity to size accordingly. With Flash-based devices which prices are falling rapidly, it's possible to design high capacity vSAN solution.
It's definitely cheaper too, but from the licensing perspective (vSAN is licensed per CPU socket), to maximize the ROI, it's preferable to have multiple disk groups per host.
How does it look like?
A quick screenshot from the lab showing small 2-node VMware vSAN cluster with a single disk group in each host. It's an All-Flash vSAN configuration.
Can I add more than one caching device to a disk group?
The answer is no. Only one flash device can be configured per disk group.
Can I add more capacity devices to a disk group?
Yes, with the hard limit of 7 capacity devices per disk group. But you can start as low as one capacity device if you like. Also, you don't have to have the same number of capacity devices per disk group, but it's preferable to homogenize the configuration.
What Happens when a capacity device fails?
Well, in this case, after 60 min of wait time (correct me if I'm wrong) the system will rebalance the vSAN cluster and recreate the missing components elsewhere and rebalance the existing components accordingly.
What happens if cache device fails?
In this case, the whole disk group is declared as failed and again, after 60 min the system will start recreating and rebuilding the missing components.
Wrap up:
VMware and vSAN keep growing. Latest numbers announced by VMware were over 7000 vSAN customers with now the majority going for All-Flash version. You now can deploy and use All Flash (AF) version of vSAN not only on Enterprise and Advanced but also on Standard. VMware Licensing has changed starting with vSAN 6.5.
More posts from ESX Virtualization:
- Dedicated vSphere 6.5 Page
- VMware VSAN 6.5 – What's New?
- VMware vSAN 6.5 Licensing PDF
- What Is Erasure Coding?
- 2 Nodes 10GbE Switch-Less Configs from Starwind
Stay tuned through RSS, and social media channels (Twitter, FB, YouTube)
Just a quick note about component availability from failures or maintenance…
When a device isn’t present, either through a network partition, host reboot, or other, this is an ABSENT situation. When a device physically fails (S.M.A.R.T. errors, or other), that is a DEGRADED situation.
Absent events wait for the 60 minute clock timer (unless changed, which isn’t recommended) to initiate a repair process.
Degraded events initiate the repair process immediately.
Check out https://storagehub.vmware.com/ and http://virtualblocks.com/ for more info on vSAN Disk Groups and more.
Thanks for your comment Jase, and for pointing out the difference between those two. Degraded triggers the repair without the 60 min wait.
“Each disk group includes one SSD and one or multiple HDDs (magnetic disks). Up to 6 magnetic disks.”
Incorrect; 7 is max.
https://www.vmware.com/pdf/vsphere6/r65/vsphere-65-configuration-maximums.pdf
TheBobkin,
your referenced documents states the following below on P25
Magnetic disks per disk group 7
SSD disks per disk group 1
@VLADAN, what options are available to me when I have unclaimed disks(HDD), five each across my hosts(7) because I do not have a flash/SSD as a cache to create another disk group within the hosts.
You mean if you can add an additional disk to your disk group as capacity? Yes.