网络性能优化GSO/GIO研究

性能检测工具安装

# curl -O http://downloads.es.net/pub/iperf/iperf-3.0.6.tar.gz
# tar axf iperf-3.0.6.tar.gz
# cd iperf-3.0.6
# ./bootstrap.sh
# ./configure
# make -j $(nproc)
# make install

基本命令

### 网卡参数查询
# ethtool -k <dev>
### 网卡参数配置
# ethtool -K <dev> <tso/gso/gro/ufo> <on/off>
### 启动监听服务
# iperf3 -s
### 发送测试数据
# iperf3 -c <srv> -l <size>
### 网卡抓包
# tcpdump -i <dev> <udp/tcp> -v

测试

测试环境网络拓扑

br-ex用于ssh连接，br-mgmt上网卡用于测试

不启用GSO/GIO

网卡配置

Host A

选项	状态
tcp-segmentation-offload	off
generic-segmentation-offload	off
generic-receive-offload	off
udp-fragmentation-offload	off

Host B

选项	状态
tcp-segmentation-offload	off
generic-segmentation-offload	off
generic-receive-offload	off
udp-fragmentation-offload	off

测试步骤

### Host B
# tcpdump -i em1 tcp -v
# iperf3 -s

### Host A
# tcpdump -i eno1 tcp -v
# iperf3 -c 92.0.0.240 -l 2000

### 测试部分结果如下Host A & Host B相同

14:11:12.099562 IP (tos 0x0, ttl 64, id 35338, offset 0, flags [DF], proto TCP (6), length 1500)
    host-92-0-0-240.as43234.net.47050 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], cksum 0xbfaf (incorrect -> 0x686c), seq 33370646:33372094, ack 1, win 229, options [nop,nop,TS val 13486141 ecr 13531163], length 1448
14:11:12.099563 IP (tos 0x0, ttl 64, id 35339, offset 0, flags [DF], proto TCP (6), length 1500)
    host-92-0-0-240.as43234.net.47050 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], cksum 0xbfaf (incorrect -> 0x2b13), seq 33372094:33373542, ack 1, win 229, options [nop,nop,TS val 13486141 ecr 13531163], length 1448
14:11:12.099804 IP (tos 0x0, ttl 64, id 63470, offset 0, flags [DF], proto TCP (6), length 52)
    host-92-0-0-241.as43234.net.targus-getdata1 > host-92-0-0-240.as43234.net.47050: Flags [.], cksum 0xadf6 (correct), ack 33359062, win 1720, options [nop,nop,TS val 13531164 ecr 13486140], length 0

### 通过测试结果可以看出，2000字节的数据包被拆分成两个包

启用TSO/GSO/GIO

网卡配置

Host A

选项	状态
tcp-segmentation-offload	on
generic-segmentation-offload	on
generic-receive-offload	on
udp-fragmentation-offload	off

Host B

选项	状态
tcp-segmentation-offload	on
generic-segmentation-offload	on
generic-receive-offload	on
udp-fragmentation-offload	off

测试步骤

### Host B
# tcpdump -i em1 tcp -v
# iperf3 -c 92.0.0.240 -l 5000

### Host A
# tcpdump -i eno1 tcp -v
# iperf3 -s

### 测试部分结果如下

### Host B（发送端seq 41190502:41213670）
    host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0x1488 (incorrect -> 0x9555), seq 41190502:41213670, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 23168
15:12:09.859303 IP (tos 0x0, ttl 64, id 62667, offset 0, flags [DF], proto TCP (6), length 52)
    host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [.], cksum 0x3bc8 (correct), ack 40724246, win 16852, options [nop,nop,TS val 15459784 ecr 15504810], length 0
15:12:09.859553 IP (tos 0x0, ttl 64, id 62668, offset 0, flags [DF], proto TCP (6), length 52)
    host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [.], cksum 0x3078 (correct), ack 40727142, win 16852, options [nop,nop,TS val 15459784 ecr 15504810], length 0

### Host A
14:44:05.781627 IP (tos 0x0, ttl 64, id 28209, offset 0, flags [DF], proto TCP (6), length 1500)
    host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0xc904 (correct), seq 41190502:41191950, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 1448
14:44:05.781631 IP (tos 0x0, ttl 64, id 29535, offset 0, flags [DF], proto TCP (6), length 40)
    host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [R], cksum 0x80af (correct), seq 1314350815, win 0, length 0
14:44:05.781872 IP (tos 0x0, ttl 64, id 28210, offset 0, flags [DF], proto TCP (6), length 2948)
    host-92-0-0-241.as43234.net.43976 > host-92-0-0-240.as43234.net.targus-getdata1: Flags [.], cksum 0xc557 (incorrect -> 0x9be3), seq 41191950:41194846, ack 1, win 229, options [nop,nop,TS val 15504851 ecr 15459784], length 2896
14:44:05.781876 IP (tos 0x0, ttl 64, id 29536, offset 0, flags [DF], proto TCP (6), length 40)
    host-92-0-0-240.as43234.net.targus-getdata1 > host-92-0-0-241.as43234.net.43976: Flags [R], cksum 0x80af (correct), seq 1314350815, win 0, length 0

### 可以看出接收端和发送端的length都超过了mtu 1500，说明GSO和GRO都生效了

启用GSO/GIO

网卡配置

Host A

选项	状态
tcp-segmentation-offload	off
generic-segmentation-offload	on
generic-receive-offload	on
udp-fragmentation-offload	off

Host B

选项	状态
tcp-segmentation-offload	on
generic-segmentation-offload	on
generic-receive-offload	on
udp-fragmentation-offload	off

测试步骤

### Host B
# tcpdump -i em1 tcp
# iperf3 -s

### Host A
# tcpdump -i eno1 tcp
# iperf3 -c 92.0.0.240 -l 5000

### 测试部分结果如下

15:32:41.857417 IP host-92-0-0-240.as43234.net.51572 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], seq 59386070:59387518, ack 1, win 229, options [nop,nop,TS val 16691752 ecr 16736846], length 1448
15:32:41.857540 IP host-92-0-0-240.as43234.net.51572 > host-92-0-0-241.as43234.net.targus-getdata1: Flags [.], seq 59387518:59388966, ack 1, win 229, options [nop,nop,TS val 16691752 ecr 16736846], length 1448
15:32:41.857542 IP host-92-0-0-241.as43234.net.targus-getdata1 > host-92-0-0-240.as43234.net.51572: Flags [.], ack 59388966, win 1786, options [nop,nop,TS val 16736849 ecr 16691752], length 0

### 但是iperf显示的cwnd（拥塞窗口）比全部关闭是要大，但是小于全部开启

### Host B
# tcpdump -i em1 udp
# iperf3 -s

### Host A
# tcpdump -i eno1 udp
# iperf3 -c 92.0.0.240 -u -l 5000