I jumped into the middle of a Twitter conversation between Scott Lowe, Chris Wahl and Jason Nash. They were talking about their disappointment that VSAN traffic isn’t routable over layer 3. For the 1.0 release of the product, this makes perfect sense to me. But some of my subsequent Twitter conversations didn’t make sense about the why it isn’t supported. I received a briefing from VMware’s VSAN team last week on VSAN. I wanted to understand the networking required to support VSAN. All nodes in the cluster (those with and without storage) must have at least one physical NIC in the same L2 collision domain and Multicasting is enabled on the network equipment. There’s no requirement to actually configure an IP address on the logical VMKernel port.
This doesn’t mean that only layer 2 protocols are used for VSAN traffic. The shared layer 2 network is just a requirement. The rules of TCP/IP networking still need to be followed unless VMware has developed new layer 2 point-to-point protocols that handles error correction and is robust enough to support storage communication. I highly doubt VMware is using a non-standard storage protocol as VMware solutions are for the most part standards conforming as it relates to networking. I at some point they’d want to support VSAN clusters spread across multiple layer 3 networks.
When I asked the VSAN product manager about the details of how management traffic is handled between nodes, he explained to me how IP6v dynamic addressing is used to establish connectivity between the VMKernels participating in the storage network. My assumption is that a combination of Multicast and IPv6 is used for VSAN traffic. I guess as the product matures, VMware will peel back the veil and allow users to route VSAN traffic. The fact that IPv6 is being used currently indicates to me this isn’t something that’s on the immediate roadmap. Rob Nelson pointed out to me that VSAN doesn’t currently support IPv6.
Again, just some random thoughts.
– Update 8/7/14 3:45 PM CST
I stand corrected. VMware did go propriety and doesn’t use IPv6 at all.
@virtualizedgeek FYI, VSAN does not use IPv6 to form clusters. We use L2 multicast and a proprietary protocol that is TCP based.
— Wade Holmes (@wholmes) August 7, 2014
Some clarification by Duncan Epping. Evidendently, the Product Manager I was talking with threw some EVO:Rail into the conversation which does use the IPv6 discovery discussed above. It’s standalone VSAN that doesn’t use IPv6 for configuring the VSAN cluster.
@virtualizedgeek btw, I think there was a confusion between EVO:RAIL and VSAN. EVO:RAIL leverages IPv6 for auto-discovery and form clusters
— Duncan Epping (@DuncanYB) September 4, 2014