(6) Switching Performance – Chaining OVS bridges

The Openvswitch installed on a Linux system provides the feature to configure multiple bridges. Multiple OVS bridges behave like independent local switches. This is the way OVS is providing switch virtualization.

It is possible to chain multiple OVS bridges on one system. Chaining multiple bridges is used by Openstack neutron, when using the default networking setup.

If Neutron is configured to use VXLAN or GRE tunnels, the integration bridge br-int is connected to the tunnel bridge br-tun. Neutron is using two Openvswitch patch ports to connect br-int and br-tun. If Neutron is configured to use Flat or Vlan networking, the integration bridge br-int is connected to the bridge br-eth1 (br-eth1 is used here as an example), with the physical NIC attached to it, using a Linux veth pair. This setup is shown in the following drawing.

Openstack-Node

One question is now: What is the performance loss, if multiple OVS bridges are chained using OVS patch ports or Linux veth pairs?

Testing method

In a previous article (Switching Performance – Connecting Linux Network Namespaces) I showed the performance of different methods to interconnect linux namespaces. The same procedure will be used to check the performance of chained Openvswitch bridges.

Two Linux network namespaces with attached ip interfaces will be used as the source and destination of traffic. The test tool used is iperf, using a different number of threads. TSO and other offloading features of the virtual nic will be switched on and off. A setup with three OVS bridges is shown in the drawing below.

The number of bridges is 1 to 9.

The OVS patch ports between the bridges are created using the following commands:

function f_create_ovs_chain_patch {
    local BRIDGEPREFIX=$1
    local NUMOFOVS=$2
    #
    # create the switches
    for I in $(seq 1 $NUMOFOVS)
    do
      BNAMEI="$BRIDGEPREFIX-$I"
      ovs-vsctl add-br $BNAMEI
      if [ $I -gt 1 ]
      then
        let K=I-1
        BNAMEK="$BRIDGEPREFIX-$K"
        PNAMEI="patch-CONNECT-$I$K"
        PNAMEK="patch-CONNECT-$K$I"
        # need to connect this bridge to the previous bridge
        ovs-vsctl add-port $BNAMEI $PNAMEI -- set Interface $PNAMEI type=patch options:peer=$PNAMEK
        ovs-vsctl add-port $BNAMEK $PNAMEK -- set Interface $PNAMEK type=patch options:peer=$PNAMEI
      fi
    done
}

function f_create_ovs_chain_patch {

local BRIDGEPREFIX=$1

local NUMOFOVS=$2

# create the switches

for I in $(seq 1 $NUMOFOVS)

BNAMEI="$BRIDGEPREFIX-$I"

ovs-vsctl add-br $BNAMEI

if [ $I -gt 1 ]

then

let K=I-1

BNAMEK="$BRIDGEPREFIX-$K"

PNAMEI="patch-CONNECT-$I$K"

PNAMEK="patch-CONNECT-$K$I"

# need to connect this bridge to the previous bridge

ovs-vsctl add-port $BNAMEI $PNAMEI -- set Interface $PNAMEI type=patch options:peer=$PNAMEK

ovs-vsctl add-port $BNAMEK $PNAMEK -- set Interface $PNAMEK type=patch options:peer=$PNAMEI

done

}

The Linux veth pairs between the bridges are created using the following commands:

function f_create_ovs_chain_veth {
    local BRIDGEPREFIX=$1
    local NUMOFOVS=$2
    local VETHPREFIX="ovschain"
    echo "*** Creating interfaces"
#
# create the switches
    local I
    for I in $(seq 1 $NUMOFOVS)
    do
      BNAMEI="$BRIDGEPREFIX-$I"
      ovs-vsctl add-br $BNAMEI
      if [ $I -gt 1 ]
      then
        let K=I-1
        BNAMEK="$BRIDGEPREFIX-$K"
        PNAMEI="$VETHPREFIX-$I$K"
        PNAMEK="$VETHPREFIX-$K$I"
        # need to connect this bridge to the previous bridge
        ip link add $PNAMEI type veth peer name $PNAMEK
        ovs-vsctl add-port $BNAMEI $PNAMEI
        ovs-vsctl add-port $BNAMEK $PNAMEK
        ip link set dev $PNAMEI up
        ip link set dev $PNAMEK up
      fi
    done
}

function f_create_ovs_chain_veth {

local BRIDGEPREFIX=$1

local NUMOFOVS=$2

local VETHPREFIX="ovschain"

echo "*** Creating interfaces"

# create the switches

local I

for I in $(seq 1 $NUMOFOVS)

BNAMEI="$BRIDGEPREFIX-$I"

ovs-vsctl add-br $BNAMEI

if [ $I -gt 1 ]

then

let K=I-1

BNAMEK="$BRIDGEPREFIX-$K"

PNAMEI="$VETHPREFIX-$I$K"

PNAMEK="$VETHPREFIX-$K$I"

# need to connect this bridge to the previous bridge

ip link add $PNAMEI type veth peer name $PNAMEK

ovs-vsctl add-port $BNAMEI $PNAMEI

ovs-vsctl add-port $BNAMEK $PNAMEK

ip link set dev $PNAMEI up

ip link set dev $PNAMEK up

done

}

It must be ensured, that there are no iptables rule active when running tests using veth pairs.

Results

The chart below shows the test results using my Linux desktop (i5-2550 CPU@3.30GHz, 32 GB RAM, Kernel 3.13, OVS 2.01, Ubuntu 14.04). The values shown for one OVS bridge are used as the baseline to compare to.

The chart shows, that the OVS using OVS patch ports behaves well. When one iperf thread is running in each namespace, this consumes 1.8 CPU cores, 1.0 for the sender side and 0.8 for the receiver side. The four cpu cores are running at 100% usage, when using four or more threads. The throughput is independent (within the measurement precision) from the number of chained OVS bridges, at least up to the tested 8.

Connecting OVS bridges using veth pairs behaves quite different. The numbers for two bridges are showing a performance loss of 10% when running with one or two iperf threads. When using more threads, the veths behave very bad. This might be a result of not having enough CPU cores, but compared to the OVS patch port, these numbers are very bad.

I’m looking for servers providing more CPU cores and sockets to run the tests on single and dual socket Xeons (16 or 20 cores including hyperthreading per socket). But I have no access to such systems.

Conclusion

My conclusion from the test is:

use OVS patch ports (in Openstack Juno the connection between br-int and the bridges for vlans is not longer using veths — very good!)
do not use Linux veth pairs

Openstack Neutron Juno is using the veths method for the Hybrid driver.