Load Based Teaming in vSphere 4.1

http://www.blueshiftblog.com/?p=148

Many commonly assume that VMware will balance network traffic across all the physical NICs that are in the same team, but this is not necessarily the case.  Let’s take a closer look at this, and also how a new feature in vSphere 4.1 can help to prevent network I/O bottlenecks.

Let’s say you have a vSwitch comprised of 4 physical NIC interfaces which is used by 5 different VM’s –each with 2 vNICs.  How does network traffic get distributed across these 4 interfaces?

As of vSphere 4.0 there were 3 teaming policies available.  Let’s look at the default first:

Route based on the originating virtual port ID (default)

This is the default setting and it means each virtual switch port will alternate affinity to each physical NIC.

The numbered blue boxes above represent ports inside the virtual switch.  Each port is assigned affinity to a physical NIC in round-robin fashion.  The first vNIC that joins the switch will be assigned pNIC1, the 2^nd gets pNIC2 and so on.

There is absolutely no consideration to traffic load here.  Furthermore, the port assignment is random – it is based on the order in which the vNICs came online.  Another key point here is that any one vNIC never has access to more than one physical NIC at a time.

All this could result potentially result in an allocation of network resources that may be less than optimal – especially if you are using 1 Gig Ethernet rather than 10G.

Now the other two options in vSphere 4.0 were Route by IP Hash and Route by MAC hash.  These options have the ability to segregate the traffic by conversation.  Each IP/MAC to IP/MAC conversation could traverse a different physical NIC in the team.  While this can result in a more balanced traffic distribution, it will likely take some collaboration with your network team in order to implement successfully.

VMWare sums up the problem with this statement:

These three policies provide static mapping from vSwitch port to pNIC adapter. It is possible that two heavy network load virtual machines are mapped to same physical adapter that is congested while the other adapters still have free bandwidth.

Now in vSphere 4.1 there is an additional option which is displayed as “Route based on physical NIC load” for distributed vSwitches, which the marketing folks refer to as “Load Based Teaming”:

With Load Based Teaming in vSphere 4.1, after initial port based assignment, the load balancing algorithm regularly checks the load of all the teaming NICs. If one gets overloaded while another one has bandwidth available, it reassigns the port-NIC mapping to reach a balanced status. During the period until the next check is performed, the mapping is stable.

Note: Bandwidth is still limited to the maximum bandwidth a single pNIC provides.

This feature is a significant improvement in the sense that it can guard against a pNIC being overloaded.  This does not balance ALL traffic, but it will balance the connections (based on load) across the pNICs and optimize them.  A vNIC can still only have affinity to a single pNIC, but the Load Based Teaming feature in vSphere 4.1 can prevent against network resources being imbalanced and overloaded within a team.

Load Based Teaming in vSphere 4.1 requires a distributed virtual switch (vDS) which requires vSphere Enterprise Plus.

For more details, review the following KB article (which was the source of the above quotes):

http://kb.vmware.com/kb/1022590 Load Based Teaming in vSphere 4.1