There is a new VMware PDF document showing many tips when evaluating and deploying VSAN. The PDF is a very good read. It walks you through to make sure that you know the difference on VSAN behavior when running on 3 or 4 nodes.
I actually got some good information about not only that when VSAN runs on 3 node cluster not all data are migrated of host when putting in maintenance. But also that in fact the rebuilding process when VSAN runs on 3 node cluster never happens. VSAN is not able to rebuild a component on another host in order to tolerate another failure. So 3 host VSAN cluster is good, but 4 is better (safer).
Another good tip I had a good read was concerning the one about multiple disk groups per host, to reduce the risk of failure. When only single disk group per host and one component fails, not only a very large amount of data will have to be rebuild on another host, but also as there is only single disk group the resiliency is not optimal.
Some tips for the network as the network performance and availability is crucial:
Test multicast operations on the network:
tcpdump-uw udp port 23451 –v will show if the master is correctly sending multicast packets (port and IP info) and if all other hosts in
the cluster receive them. So on the master, this command will show the packets being sent out to the multicast. On all other hosts, the
same exact packets should be seen (from the master to the multicast). If they are not seen, multicast is not working correctly.
tcpdump-uw igmp will tell if host is sending regular IGMP membership reports, aka “join”
Test throughput, looking for dropped packets, etc.
iperf – a third party tool called iperf (which will need to run in virtual machines) is useful for examining the network bandwidth, etc.
There is tons of other tips and every single of them is very useful. It's simply a must have document.
VMware VSAN is very simple to deploy, but It's worth to take time and know the advantages and dis-advantages of the solution. The document which you can download from this blog post is a very good help. I can only highly recommend that.