1.关于 SO_REUSEADDR 的讨论 MSDN关于 SO_REUSEADDR 的解释: The state of the SO_REUSEADDR socket option determines whether the local transport address to which a socket will be bound is always shared with other sockets. This socket option applies only to listening sockets, datagram sockets, and connection-oriented sockets. 情况一: 当服务器A在开启 SO_REUSEADDR 绑定到一个指定的端口X，此时另一个也开启了 SO_REUSEADDR 的服务器B也绑定同一个端口X，此时第二个服务器的 bind 与 listen 仍会成功 情况一发生的条件是当且仅当服务器A与B同时开启了 SO_REUSEADDR 情况一的两个服务器的状态:在服务器A与服务器B均开启的情况下，此时客户端在指定端口X上请求连接，服务器A将进行响应。当服务器A关闭后，客户端继续在指定端口X上请求连接，此时服务器B将进行响应 根据MSDN中说明，以上情况是未定义状态 情况二: 当客户端A指定了本地一个端口X进行连接某个服务器，此时又有一个服务器B也在那个端口X上执行绑定，当且仅当客户端A与服务器B都开启了 SO_REUSEADDR 时，服务器B的绑定才会成功，且服务器B能正常工作 考虑一个问题:当服务器监听了一个端口X，此时客户端与服务器建立连接，服务器将返回一个用于与客户端通信的套接字，此套接字对应的端口仍是监听端口X，然而服务器能够区分在端口X上的数据，到底是监听套接字 的，还是与客户端通信的套接字的。 TCP无法仅仅通过查看目的端口号来分离外来的分节到不同的端点。他必须查看套接字对的所有四个元素才能确定由哪个端点接收某个到达的分节 Using SO_REUSEADDR The SO_REUSEADDR socket option allows a socket to forcibly bind to a port in use by another socket. The second socket calls setsockopt with the optname parameter set to SO_REUSEADDR and the optval parameter set to a boolean value of TRUE before calling bind on the same port as the original socket. Once the second socket has successfully bound, the behavior for all sockets bound to that port is indeterminate. For example, if all of the sockets on the same port provide TCP service, any incoming TCP connection requests over the port cannot be guaranteed to be handled by the correct socket — the behavior is non-deterministic. A malicious program can use SO_REUSEADDR to forcibly bind sockets already in use for standard network protocol services in order to deny access to those service. No special privileges are required to use this option. If a client application binds to a port before a server application is able to bind to the same port, then problems may result. If the server application forcibly binds using the SO_REUSEADDR socket option to the same port, then the behavior for all sockets bound to that port is indeterminate. The exception to this non-deterministic behavior is multicast sockets. If two sockets are bound to the same interface and port and are members of the same multicast group, data will be delivered to both sockets, rather than an arbitrarily chosen one. Using SO_EXCLUSIVEADDRUSE Before the SO_EXCLUSIVEADDRUSE socket option was introduced, there was very little a network application developer could do to prevent a malicious program from binding to the port on which the network application had its own sockets bound. In order to address this security issue, Windows Sockets introduced the SO_EXCLUSIVEADDRUSE socket option, which became available on Windows NT 4.0 with Service Pack 4 (SP4) and later. The SO_EXCLUSIVEADDRUSE socket option can only be used by members of the Administrators security group on Windows XP and earlier. The reasons why this requirement was changed on Windows Server 2003 and later are discussed later in the article. The SO_EXCLUSIVEADDRUSE option is set by calling the setsockopt function with the optname parameter set to SO_EXCLUSIVEADDRUSE and the optval parameter set to a boolean value of TRUE before the socket is bound. After the option is set, the behavior of subsequent bind calls differs depending on the network address specified in each bind call. The table below describes the behavior that occurs in Windows XP and earlier when a second socket attempts to bind to an address previously bound to by a first socket using specific socket options. 参考资料: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/so-reuseaddr https://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx 2.关于 SO_EXCLUSIVEADDRUSE 的讨论 SO_EXCLUSIVEADDRUSE:Enables a socket to be bound for exclusive access. Does not require administrative privilege. 双网卡A和B，启动三个TCP服务如下，测试结果如下: Srv0 Ip A Port X Srv1 Ip B Port X Srv2 Ip 0.0.0.0 Port X (A):则三个Srv均能成功，连接Ip A 的客户端会与Srv0进行通信，连接Ip B的客户端会与Srv1进行通信，此过程不需要使用 SO_REUSEADDR, 当Srv0和Srv1断开后，Srv2能处理来自IP A 和IP B 的连接，使用netstat -a 发现会有三个套接字在执行 LISTENING 服务 (B):当Srv0、Srv1使用 SO_EXCLUSIVEADDRUSE 时， Srv2仍能 bind 成功，当Srv2使用了 SO_EXCLUSIVEADDRUSE ，Srv0、Srv1的bind会失败，即使使用了 SO_REUSEADDR 也一样 (C):先正常开启Srv2，然后Srv0、Srv1指定 SO_EXCLUSIVEADDRUSE 进行 bind ，情况如(A)中一样 In the case where the first bind sets no options or SO_REUSEADDR, and the second bind performs a SO_REUSEADDR, the second socket has overtaken the port and behavior regarding which socket will receive packets is undetermined. SO_EXCLUSIVEADDRUSE was introduced to address this situation.A socket with SO_EXCLUSIVEADDRUSE set cannot always be reused immediately after socket closure. For example, if a listening socket with the exclusive flag set accepts a connection after which the listening socket is closed, another socket cannot bind to the same port as the first listening socket with the exclusive flag until the accepted connection is no longer active. This situation can be quite complicated; even though the socket has been closed, the underlying transport may not terminate its connection. Even after the socket is closed, the system must send all of the buffered data, transmit a graceful disconnect to the peer, and wait for a graceful disconnect from the peer. It is therefore possible that the underlying transport may never release the connection, such as when the peer advertises a zero-size window, or other such attacks. In the previous example, the listening socket was closed after a client connection was accepted. Now even if the client connection is closed, the port still may not be reused if the client connection remains in an active state because of unacknowledged data, and so forth. To avoid this situation, applications should ensure a graceful shutdown: call shutdown with the SD_SEND flag, then wait in a recv loop until zero bytes are returned. Doing so avoids the problem associated with port reuse, guarantees all data was received by the peer, and assures the peer that all its data was successfully received. The SO_LINGER option may be set on a socket to prevent the port going into one of the active wait states; however, doing so is discouraged because it can lead to undesired effects, because it can cause the connection to be reset. For example, if data has been received but not yet acknowledged by the peer, and the local computer closes the socket with SO_LINGER set, the connection is reset and the peer discards the unacknowledged data. Also, picking a suitable time to linger is difficult; a value too small results in many aborted connections, while a large timeout can leave the system vulnerable to denial of service attacks by establishing many connections, and thereby stalling numerous application threads. Note A socket that is using the SO_EXCLUSIVEADDRUSE option must be shut down properly prior to closing it. Failure to do so can cause a denial of service attack if the associated service needs to restart. //参考资料 https://msdn.microsoft.com/zh-cn/subscriptions/downloads/cc150667.aspx 3..关于TCP客户端与服务端执行主动关闭的讨论 TCP客户端在主动断开连接后，马上用相同本地地址进行重连，此时 bind 会失败，但是指定了 SO_REUSEADDR 后， bind 会成功。马上进行 connect 由于客户端执行了主动的断线，会处于 TIME_WAIT 状态， 所以 connect 会失败。若TCP客户端在执行主动断线前，使用 SO_LINGER 并指定 linger l_onoff 为1，l_linger为0，则不会产生 TIME_WAIT 状态，此时的 connect 会成功 struct linger { u_short l_onoff; /* option on/off */ u_short l_linger; /* linger time */ }; 服务器在指定端口X进行监听，此时客户端与服务器建立连接，然后服务器立刻主动断开与客户端的连接，此时服务端用于通信的 socket 会处于 TIME_WAIT 状态，但是此时继续关闭监听的套接字然后立刻重启服务器， 服务器仍然可以在端口X建立连接。对于监听套接字，对其执行关闭操作，并不会产生 TIME_WAIT 状态，而只是通知内核不在该端口进行监听。 对于服务器来说，若服务器总是对已建立连接的套接字执行关闭操作，那么服务器端的 socket 将大量处于 TIME_WAIT 状态，将会影响服务器申请新的 socket 资源，一般HTTP服务器需要考虑这一点 4.对于未设置 SO_KEEPALIVE 的TCP连接 经测试，当建立连接后拔掉网线，发送端总是能知道网络已经断开，因为TCP发送数据会进行重传，在重传一定次数后， send 就会失败，然而接收端却不能发现网络已经断开 主机A无限给主机B使用 TCP send ，主机B一直不 recv ，当 send 被阻塞后， 不管拔掉哪端的网线， 使用 send 的一端总是能检测到网线被断开， 而 recv 的那端并不能检测到 TCP/IP详解 卷一:协议 中p501页指出: 通告窗口指示了接收端可接受的数据量。当窗口值变为0时，可以有效的阻止发送端继续发送，直到窗口大小恢复为非零值。当接收端重新获得可用空间时，会给发送端传输一个 窗口更新 告知其可继续发送数据。 这样的窗口更新通常都不包含数据(为纯 ACK )。如果一个包含窗口更新的ACK丢失，通信双方将一直处于等待状态:发送方等待收到窗口更新告知其可继续发送，接收方等待接收数据(已将窗口设为非0值)。 为了防止上述死锁的发生，发送端会采用一个 持续计时器 间歇的查询接收端，看其窗口是否已增长。持续计时器会触发 窗口探测 的传输，强制要求接收端返回 ACK(其中包含了窗口大小字段)。 5. 已建立的Tcp连接，禁用网络 主动禁用端 recv 会立刻返回 -1, WSAGetLastError() 返回 WSAECONNRESET 被动禁用端 send 可能会成功，在一次测得返回了8193字节(每次发3字节)，还有一次返回65536(每次发665536)，然后返回 -1, WSAGetLastError() 返回 WSAECONNRESET 被动禁用端 recv 不会得到通知 // MessageId: WSAECONNRESET // // MessageText: // // An existing connection was forcibly closed by the remote host. // #define WSAECONNRESET 10054L 6. TCP接收端一般做法: 使用非阻塞模式，将接受缓冲区设为 50 MB左右(这个值只要比较大就可)，使用 recv 判断返回值，若是 WSAEWOULDBLOCK 则 sleep(1) ,否则就断线 使用以上方法，可以使得网络使用率很高的同时，cpu使用率很低 不要滥用 select ，此函数开销其实挺高的