在cmm02node01上将ROS个数增加到16个，每个ROS处理一个socket，tcp传输的包长为2KB。发送和接收的套接字缓冲区设置为[8MB,8MB]. 得到的平均总带宽为31Gb/s. 下面分别是系统的各项指标：
1. tcp发送端的send-Q和recv-Q大小：
Recv-Q
The count of bytes not copied by the user program connected to this socket.
Send-Q
The count of bytes not acknowledged by the remote host.
send-Q: 每个socket的发送队列长度都不为0， 下面数值的单位为MB, 发送队列的长度接近于发送端缓冲区长度8MB.
[root@cmm02node06 ~]# netstat -en|grep 192.168.250.141|grep -v ":22"|awk '{print $4,$3/1024/1024}' 192.168.250.146:3347 7.68687 192.168.250.146:3346 6.15338 192.168.250.146:3344 5.56078 192.168.250.146:3341 6.76311 192.168.250.146:3333 7.51598 192.168.250.146:3348 5.20684 192.168.250.146:3339 5.7579 192.168.250.146:3335 7.00312 192.168.250.146:3342 7.45697 192.168.250.146:3337 6.22381 192.168.250.146:3340 7.11814 192.168.250.146:3334 6.00944 192.168.250.146:3338 7.45279 192.168.250.146:3343 7.27193 192.168.250.146:3345 7.5564 192.168.250.146:3336 6.56213
recv-Q:大部分时间大部分socket的接收队列为0，有瞬时接近8MB的长度出现。
[zengtx@cmm02node01 simulation_FADC]$netstat -en|grep 192.168.250.146|grep -v ":22"|awk '{print $5,$2/1024/1024}' 192.168.250.146:3341 0 192.168.250.146:3339 0.0190735 192.168.250.146:3335 0 192.168.250.146:3334 7.4211 192.168.250.146:3338 0 192.168.250.146:3343 0 192.168.250.146:3347 0 192.168.250.146:3333 0 192.168.250.146:3340 0.606239 192.168.250.146:3344 0 192.168.250.146:3337 0.0764236 192.168.250.146:3345 5.72362 192.168.250.146:3342 0 192.168.250.146:3336 0.0157013 192.168.250.146:3348 0.377197 192.168.250.146:3346 0
发送方已经发出去，还没收到ack的数据长度为5~7MB, 而接收端还没被ROS读走的数据长度基本为0，在瞬时状态，出现接近5~7MB，这说明接收端ROS从接收缓冲区拿数据的速度还挺快的啊。。要不然Recv-Q应该是连续非零的状态；发送方没收到ack的数据长度决定于下面三种情况：
1， 接收方收到了，还没发出ack
2， 发送方发出的数据，接收端还没收到
3，接收方收到了，发出了ack，还没到达发送方
由于Recv-Q常为0，第1部分应该很少，所以send-Q的长度应该大部分来自于正在传输的链路上的数据。
2. tcp 的窗口变化情况: 接收端的通告窗口有时会出现win 1的情况，也就是说接收端缓冲区有时会满。
[root@cmm02node01 part_dk_ef]# tcpdump -nn -i eth4 dst cmm02node06 -c 100
17:53:24.200582 IP 192.168.250.141.40182 > 192.168.250.146.3333: Flags [.], ack 886177, win 378, options [nop,nop,TS val 2976514857 ecr 2976304391], length 0 17:53:24.200641 IP 192.168.250.141.49010 > 192.168.250.146.3334: Flags [.], ack 1107721, win 383, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200728 IP 192.168.250.141.50816 > 192.168.250.146.3343: Flags [.], ack 1225009, win 383, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200745 IP 192.168.250.141.46021 > 192.168.250.146.3338: Flags [.], ack 4241527581, win 1, options [nop,nop,TS val 2976514857 ecr 2976304391], length 0 17:53:24.200772 IP 192.168.250.141.49010 > 192.168.250.146.3334: Flags [.], ack 1185913, win 382, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200781 IP 192.168.250.141.32838 > 192.168.250.146.3345: Flags [.], ack 3388321, win 382, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200786 IP 192.168.250.141.40415 > 192.168.250.146.3341: Flags [.], ack 977401, win 383, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200790 IP 192.168.250.141.58558 > 192.168.250.146.3339: Flags [.], ack 65161, win 383, options [nop,nop,TS val 2976514857 ecr 2976304391], length 0 17:53:24.200791 IP 192.168.250.141.40415 > 192.168.250.146.3341: Flags [.], ack 1013601, win 382, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200807 IP 192.168.250.141.47775 > 192.168.250.146.3348: Flags [.], ack 1676785, win 383, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200807 IP 192.168.250.141.32838 > 192.168.250.146.3345: Flags [.], ack 3414385, win 382, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200822 IP 192.168.250.141.40415 > 192.168.250.146.3341: Flags [.], ack 1016497, win 383, options [nop,nop,TS val 2976514857 ecr 2976304392], length 0 17:53:24.200824 IP 192.168.250.141.58558 > 192.168.250.146.3339: Flags [.], ack 91225, win 383, options [nop,nop,TS val 2976514857 ecr 2976304391], length 0 17:53:24.200837 IP 192.168.250.141.58558 > 192.168.250.146.3339: Flags [.], ack 101361, win 383, options [nop,nop,TS val 2976514857 ecr 2976304391], length 0
3. CPU占用率
发送端的CPU占用率：
Tasks: 687 total, 2 running, 520 sleeping, 165 stopped, 0 zombie Cpu(s): 1.4%us, 18.0%sy, 0.0%ni, 74.9%id, 0.0%wa, 0.0%hi, 5.8%si, 0.0%st Mem: 32979804k total, 2236852k used, 30742952k free, 192056k buffers Swap: 33554428k total, 0k used, 33554428k free, 886620k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28677 zengtx 20 0 91936 1392 1188 S 56.9 0.0 23:58.37 main 28682 zengtx 20 0 91936 1392 1188 S 56.9 0.0 27:17.57 main 28681 zengtx 20 0 91936 1392 1188 S 44.3 0.0 23:17.63 main 28670 zengtx 20 0 91936 1392 1188 S 40.7 0.0 17:35.56 main 28671 zengtx 20 0 91936 1396 1188 S 40.7 0.0 22:43.92 main 28675 zengtx 20 0 91936 1392 1188 S 40.0 0.0 19:13.46 main 28678 zengtx 20 0 91936 1388 1188 S 39.7 0.0 20:53.11 main 28680 zengtx 20 0 91936 1392 1188 S 34.4 0.0 15:31.37 main 28672 zengtx 20 0 91936 1388 1188 S 34.1 0.0 16:54.74 main 28673 zengtx 20 0 91936 1388 1188 S 33.7 0.0 17:31.83 main 28674 zengtx 20 0 91936 1392 1188 S 32.1 0.0 15:25.26 main 28685 zengtx 20 0 91936 1392 1188 S 29.1 0.0 15:43.46 main 28683 zengtx 20 0 91936 1392 1188 S 24.2 0.0 10:19.80 main 28684 zengtx 20 0 91936 1392 1188 S 23.8 0.0 10:47.21 main 28679 zengtx 20 0 91936 1392 1188 S 23.5 0.0 10:57.48 main 28676 zengtx 20 0 91936 1392 1188 S 23.2 0.0 10:29.21 main
接收端的cpu占用率：
Tasks: 680 total, 1 running, 679 sleeping, 0 stopped, 0 zombie Cpu(s): 1.9%us, 17.7%sy, 0.0%ni, 66.6%id, 0.0%wa, 0.0%hi, 13.8%si, 0.0%st Mem: 32979808k total, 14069232k used, 18910576k free, 179696k buffers Swap: 33554428k total, 140404k used, 33414024k free, 11910912k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25220 lhaaso 20 0 1008m 29m 20m S 90.1 0.1 40:42.02 ReadoutApplicat 25242 lhaaso 20 0 1008m 27m 20m S 82.2 0.1 33:03.06 ReadoutApplicat 25253 lhaaso 20 0 1008m 29m 20m S 65.7 0.1 37:04.55 ReadoutApplicat 25144 lhaaso 20 0 1008m 29m 20m S 62.7 0.1 33:54.31 ReadoutApplicat 25262 lhaaso 20 0 1008m 28m 20m S 62.7 0.1 21:53.74 ReadoutApplicat 25161 lhaaso 20 0 1008m 29m 20m S 56.8 0.1 22:50.42 ReadoutApplicat 25177 lhaaso 20 0 1008m 27m 20m S 51.2 0.1 21:55.97 ReadoutApplicat 25135 lhaaso 20 0 1008m 29m 20m S 49.5 0.1 24:16.46 ReadoutApplicat 25206 lhaaso 20 0 1008m 30m 20m S 48.9 0.1 23:39.08 ReadoutApplicat 25172 lhaaso 20 0 1008m 29m 20m S 40.0 0.1 17:57.41 ReadoutApplicat 25234 lhaaso 20 0 1008m 27m 20m S 39.0 0.1 24:50.91 ReadoutApplicat 25119 lhaaso 20 0 1008m 28m 20m S 35.3 0.1 12:42.17 ReadoutApplicat 25147 lhaaso 20 0 1008m 30m 20m S 35.0 0.1 13:37.37 ReadoutApplicat 25155 lhaaso 20 0 1008m 29m 20m S 33.0 0.1 19:14.23 ReadoutApplicat 25192 lhaaso 20 0 1008m 29m 20m S 24.8 0.1 10:47.17 ReadoutApplicat 25126 lhaaso 20 0 1008m 29m 20m S 24.1 0.1 12:57.40 ReadoutApplicat
4. 发送端 tcp 重传率 ： 0.3-0.5% 发生重传是因为接收端处理不过来丢包了？还是网络环境不好中途丢包了？
[root@cmm02node06 ~]# sar -n TCP,ETCP 10 5 Linux 2.6.32-504.el6.x86_64 (cmm02node06) 04/05/2017 _x86_64_ (24 CPU) 06:02:22 PM active/s passive/s iseg/s oseg/s 06:02:32 PM 0.00 0.00 174057.86 145585.28 06:02:22 PM atmptf/s estres/s retrans/s isegerr/s orsts/s 06:02:32 PM 0.00 0.00 670.01 0.00 0.00 06:02:32 PM active/s passive/s iseg/s oseg/s 06:02:42 PM 0.00 0.00 176412.57 143916.91 06:02:32 PM atmptf/s estres/s retrans/s isegerr/s orsts/s 06:02:42 PM 0.00 0.00 472.11 0.00 0.00 ^C [root@cmm02node06 ~]# . diubao2.sh 0.39 0.33 0.36
5. 接收端 eth4 网卡的丢包率，10s内ifconfig dropped packets / rx packets =0.01% << 发送端的重传率。
[lhaaso@cmm02node01 ~]$ . drop.sh .00011 .00010 .00007 .00007 .00007 .00007 .00009 .00009 .00008 .00005
6. 每秒中断次数和每秒上下文切换次数。
接收端：
[zengtx@cmm02node01 simulation_FADC]$vmstat 10 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 8 0 140404 18897704 179700 11920592 0 0 0 0 0 0 0 2 98 0 0 12 0 140404 18892516 179700 11920596 0 0 0 47 153692 123239 2 31 67 0 0 11 0 140404 18867876 179700 11920804 0 0 0 58 154928 126105 2 30 68 0 0 5 0 140404 18907464 179700 11920800 0 0 0 28 150326 120655 2 31 67 0 0 7 0 140404 18891920 179700 11920912 0 0 0 35 154780 130286 2 30 68 0 0
发送端：
[root@cmm02node06 ~]# vmstat 10 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 5 0 0 30739460 192056 886640 0 0 0 0 0 0 0 0 99 0 0 1 0 0 30735084 192056 886640 0 0 0 0 241738 4395 1 24 75 0 0 4 0 0 30741316 192056 886640 0 0 0 0 242603 4348 1 24 75 0 0 8 0 0 30745592 192056 886640 0 0 0 0 241574 3938 1 24 75 0 0 7 0 0 30747076 192056 886640 0 0 0 0 238078 3774 1 24 74 0 0