My story
As an IT engineer, part of my job is to design new networks, where I often find that the main bottleneck is the network switches. About a year ago I searched for a more robust switch implementation to use in a new large network I designed, and I came across OVS. After reading more about it, and doing a lot of testing, I started using OVS as my go-to production switch.
The current OVS version (which I’ve been using) uses DPDK 18.11. When the new DPDK version was released, I wondered whether I should update my switches. Mainly, I had two questions:
- How easy is it to upgrade OVS to the new DPDK version?
- Is the new version of DPDK better?
Compiling Open vSwitch with DPDK 19.11
OVS has a very detailed guide about compiling DPDK 18.11. They haven’t updated it to version 19.11 (yet, hopefully) so I needed to figure out how to do so by myself.
I started out with trying to compile everything exactly as the guide says. It wasn’t very surprising that a few errors appeared:
LIB/NETDEV-DPDK.C:86:38: ERROR: ‘ETHER_HDR_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:86:54: ERROR: ‘ETHER_CRC_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:618:20: ERROR: ‘ETHER_MTU’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:88:46: ERROR: ‘ETHER_HDR_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:88:62: ERROR: ‘ETHER_CRC_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:91:40: ERROR: ‘ETHER_HDR_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:91:56: ERROR: ‘ETHER_CRC_LEN’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:933:20: ERROR: ‘ETHER_MTU’ UNDECLARED (FIRST USE IN THIS FUNCTION)
LIB/NETDEV-DPDK.C:1045:23: ERROR: STORAGE SIZE OF ‘ETH_ADDR’ ISN’T KNOWN
This may look intimidating at first, but in fact all the errors are very similar — they’re all about undeclared symbols (with names starting with “ETHER”). Considering OVS compiled successfully with the previous DPDK version, I assumed the problem originated from some DPDK API change.
I searched DPDK 18.11 for files containing these undeclared variables and found /lib/librte_net/rte_ether.h:
#DEFINE ETHER_ADDR_LEN 6 /**< LENGTH OF ETHERNET ADDRESS. */ #DEFINE ETHER_TYPE_LEN 2 /**< LENGTH OF ETHERNET TYPE FIELD. */ #DEFINE ETHER_CRC_LEN 4 /**< LENGTH OF ETHERNET CRC. */ #DEFINE ETHER_HDR_LEN (ETHER_ADDR_LEN * 2 + ETHER_TYPE_LEN) /**< LENGTH OF ETHERNET HEADER. */
#DEFINE ETHER_MIN_LEN 64 /**< MINIMUM FRAME LEN, INCLUDING CRC. */
#DEFINE ETHER_MAX_LEN 1518 /**< MAXIMUM FRAME LEN, INCLUDING CRC. */
#DEFINE ETHER_MTU (ETHER_MAX_LEN - ETHER_HDR_LEN - ETHER_CRC_LEN) /**< ETHERNET MTU. */
#DEFINE ETHER_MAX_VLAN_FRAME_LEN (ETHER_MAX_LEN + 4) /**< MAXIMUM VLAN FRAME LENGTH, INCLUDING CRC. */
#DEFINE ETHER_MAX_JUMBO_FRAME_LEN 0X3F00 /**< MAXIMUM JUMBO FRAME LENGTH, INCLUDING CRC. */
#DEFINE ETHER_MAX_VLAN_ID 4095 /**< MAXIMUM VLAN ID. */ #DEFINE ETHER_MIN_MTU 68 /**< MINIMUM MTU FOR IPV4 PACKETS, SEE RFC 791. */
I opened the same file in version 19.11 — and found the changes:
#DEFINE RTE_ETHER_ADDR_LEN 6 /**< LENGTH OF ETHERNET ADDRESS. */ #DEFINE RTE_ETHER_TYPE_LEN 2 /**< LENGTH OF ETHERNET TYPE FIELD. */ #DEFINE RTE_ETHER_CRC_LEN 4 /**< LENGTH OF ETHERNET CRC. */
#DEFINE RTE_ETHER_HDR_LEN (RTE_ETHER_ADDR_LEN * 2 + RTE_ETHER_TYPE_LEN) /**< LENGTH OF ETHERNET HEADER. */ #DEFINE RTE_ETHER_MIN_LEN 64 /**< MINIMUM FRAME LEN, INCLUDING CRC. */
#DEFINE RTE_ETHER_MAX_LEN 1518 /**< MAXIMUM FRAME LEN, INCLUDING CRC. */
#DEFINE RTE_ETHER_MTU (RTE_ETHER_MAX_LEN - RTE_ETHER_HDR_LEN - RTE_ETHER_CRC_LEN) /**< ETHERNET MTU. */ #DEFINE RTE_ETHER_MAX_VLAN_FRAME_LEN (RTE_ETHER_MAX_LEN + 4) /**< MAXIMUM VLAN FRAME LENGTH, INCLUDING CRC. */ #DEFINE RTE_ETHER_MAX_JUMBO_FRAME_LEN 0X3F00 /**< MAXIMUM JUMBO FRAME LENGTH, INCLUDING CRC. */
#DEFINE RTE_ETHER_MAX_VLAN_ID 4095 /**< MAXIMUM VLAN ID. */ #DEFINE RTE_ETHER_MIN_MTU 68 /**< MINIMUM MTU FOR IPV4 PACKETS, SEE RFC 791. */
We can clearly see that the symbols stayed pretty much the same — with “RTE_” prefix added to their names. This change is documented in version 19.08’s API changes:
“The network structures, definitions and functions have been prefixed by rte_ to resolve conflicts with libc headers”.
To sort this problem out, I wrote a small shell script that changes the netdev-dpdk.c file in OVS’s source code to use the new interface:
#!/bin/bash
sed -i "s/ ETHER_/ RTE_ETHER_/" lib/netdev-dpdk.c
sed -i "s/(ETHER_/(RTE_ETHER_/" lib/netdev-dpdk.c
sed -i "s/ ETHER_/ RTE_ETHER_/" lib/netdev-dpdk.c
sed -i "s/ e_RTE_METER_/ RTE_COLOR_/" lib/netdev-dpdk.c
sed -i "s/struct ether_addr/struct rte_ether_addr/" lib/netdev-dpdk.c
sed -i "s/struct ether_hdr/struct rte_ether_hdr/" lib/netdev-dpdk.c
After doing so, the code compiled successfully and my DPDK 19.11 Open vSwitch was ready to go!
$ ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.12.0
DPDK 19.11.0
$ ovs-vsctl get Open_vSwitch . dpdk_version
"DPDK 19.11.0"
Testing the new switch
After all the compilation we are left with the most important question — is the new version better?
I chose to test my switch’s capabilities in two aspects — traffic rate and latency. More precisely, I conducted two tests:
- What’s the best traffic rate I can get through the switch?
- What’s the average latency in the switch? How does the traffic rate affect the latency?
Both tests took place in a new network I’m setting up. This is a company’s telephony network, which consists of two large telephony switches, a few dozen VoIP phones (which made the tests more realistic), a couple of administration computers and my OVS as the main switch of the network.
I used two of the computers to test the switch’s performance. Also, to gain a broader perspective I compiled my switch with DPDK versions 19.05 and 18.05 (which was easily done, same as 18.11).
Test 1: traffic rate
This test was very straight forward — I used JPerf to find the maximum traffic rate I can generate between the two computers through my switch. The software opens a couple of UDP connections and tries to transfers as much random data as possible. These are the results: