在WinSock上使用IOCP

在WinSock上使用IOCP
本文章假设你已经理解WindowsNT的I/O模型以及I/O完成端口(IOCP)，并且比较熟悉将要用到的API，如果你打算学习IOCP，请参考Jeffery Richter的Advanced Windows(第三版)，第15章I/O设备，里面有极好的关于完成端口的讨论以及对即将使用API的说明。
IOCP提供了一个用于开发高效率和易扩展程序的模型。Winsock2提供了对IOCP的支持，并在WindowsNT平台得到了完整的实现。然而IOCP是所有WindowsNT I/O模型中最难理解和实现的，为了帮助你使用IOCP设计一个更好的Socket服务，本文提供了一些诀窍。
Tip 1:使用Winsock2 IOCP函数例如WSASend和WSARecv，如同Win32文件I/O函数，例如WriteFile和ReadFile。
微软提供的Socket句柄是一个可安装文件系统(IFS)句柄，因此你可以使用Win32的文件I/O函数调用这个句柄，然而，将Socket句柄和文件系统联系起来，你不得不陷入很多的Kernal/User模式转换的问题中，例如线程的上下文转换，花费的代价还包括参数的重新排列导致的性能降低。
因此你应该使用只被Winsock2中IOCP允许的函数来使用IOCP。在ReadFile和WriteFile中会发生的额外的参数重整以及模式转换只会发生在一种情况下，那就是如果句柄的提供者并没有将自己的WSAPROTOCOL_INFO结构中的DwServiceFlags1设置为XP1_IFS_HANDLES。
注解：即使使用WSASend和WSARecv，这些提供者仍然具有不可避免的额外的模式转换，当然ReadFile和WriteFile需要更多的转换。
TIP 2: 确定并发工作线程数量和产生的工作线程总量。
并发工作线程的数量和工作线程的数量并不是同一概念。你可以决定IOCP使用最多2个的并发线程以及包括10个工作线程的线程池。工作线程池拥有的线程多于或者等于并发线程的数量时，工作线程处理队列中一个封包的时候可以调用win32的Wait函数，这样可以无延迟的处理队列中另外的封包。
如果队列中有正在等待被处理的封包，系统将会唤醒一个工作线程处理他，最后，第一个线程确认正在休眠并且可以被再次调用，此时，可调用线程数量会多于IOCP允许的并发线程数量(例如,NumberOFConcurrentThreads)。然而，当下一个线程调用GetQueueCompletionStatus并且进入等待状态，系统不会唤醒他。一般来说，系统会试图保持你设定的并发工作线程数量。
一般来讲，每拥有一个CPU，在IOCP中你可以使用一个并发工作线程，要做到这点，当你第一次初始化IOCP的时候，可以在调用CreateIOCompletionPort的时候将NumberOfConcurrentThreads设置为0。
TIP 3：将一个提交的I/O操作和完成封包的出列联系起来。
当对一个封包进行出列，可以调用GetQueuedCompletionStatus返回一个完成Key和一个复合的结构体给I/O。你可以分别的使用这两个结构体来返回一个句柄和一个I/O操作信息，当你将IOCP提供的句柄信息注册给Socket，那么你可以将注册的Socket句柄当做一个完成Key来使用。为每一个I/O的"extend"操作提供一个包含你的应用程序IO状态信息的复合结构体。当然，必须确定你为每个的I/O提供的是唯一的复合结构体。当I/O完成的时候，会返回一个指向结构体的指针。
TIP 4:I/O完成封包队列的行为
IOCP中完成封包队列的等待次序并不决定于Winsock2 I/O调用产生的顺序。如果一个Winsock2的I/O调用返回了SUCCESS或者IO_PENDING，那么他保证当I/O操作完成后，完成封包会进入IOCP的等待队列，而不管Socket句柄是否已经关闭。如果你关闭了socket句柄，那么将来调用WSASend,WSASendTo,WSARecv和WSARecvFrom会失败并返回一个不同于SUCCES或者IO_PENDING的代码，这时将不会产生一个完成封包。而在这种情况下，前一次使用GetQueuedCompletionStatus提交的I/O操作所得到的完成封包，会显示一个失败的信息。
如果你删除了IOCP本身，那么不会有任何I/O请求发送给IOCP，因为IOCP的句柄已经不可用，尽管系统底层的IOCP核心结构并不会在所有已提交I/O请求完成之前被移除。
TIP5:IOCP的清除
很重要的一件事是使用复合I/O时候的IOCP清除：如果一个I/O操作尚未完成，那么千万不要释放该操作创建的复合结构体。HasOverlappedIoCompleted函数可以帮助你检查一个I/O操作是否已经完成。
关闭服务一般有两种情况，第一种你并不关心尚未结束的I/O操作的完成状态，你只希望尽可能快的关闭他。第二种，你打算关闭服务，但是你需要获知未结束I/O操作的完成状态。
第一种情况你可以调用PostQueueCompletionStatus(N次，N等于你的工作线程数量)来提交一个特殊的完成封包，他通知所有的工作线程立即退出，关闭所有socket句柄和他们关联的复合结构体，然后关闭完成端口(IOCP)。在关闭复合结构体之前使用HasOverlappedIOCompleted检查他的完成状态。如果一个socket关闭了，所有基于他的未结束的I/O操作会很快的完成。
在第二种情况，你可以延迟工作线程的退出来保证所有的完成封包可以被适当的出列。你可以首先关闭所有的socket句柄和IOCP。可是，你需要维护一个未完成I/O的数字，以便你的线程可以知道可以安全退出的时间。尽管当队列中有很多完成封包在等待的时候，活动的工作线程不能立即退出，但是在IOCP服务中使用全局I/O计数器并且使用临界区保护他的代价并不会象你想象的那样昂贵。
INFO: Design Issues When Using IOCP in a Winsock Server
适用于
This article was previously published under Q192800
SUMMARY
This article assumes you already understand the I/O model of the Windows NT I/O Completion Port (IOCP) and are familiar with the related APIs. If you want to learn IOCP, please see Advanced Windows (3rd edition) by Jeffery Richter, chapter 15 Device I/O for an excellent discussion on IOCP implementation and the APIs you need to use it.
An IOCP provides a model for developing very high performance and very scalable server programs. Direct IOCP support was added to Winsock2 and is fully implemented on the Windows NT platform. However, IOCP is the hardest to understand and implement among all Windows NT I/O models. To help you design a better socket server using IOCP, a number of tips are provided in this article.
MORE INFORMATION
TIP 1: Use Winsock2 IOCP-capable functions, such as WSASend and WSARecv, over Win32 file I/O functions, such as WriteFile and ReadFile.
Socket handles from Microsoft-based protocol providers are IFS handles so you can use Win32 file I/O calls with the handle. However, the interactions between the provider and file system involve many kernel/user mode transition, thread context switches, and parameter marshals that result in a significant performance penalty. You should use only Winsock2 IOCP- capable functions with IOCP.
The additional parameter marshals and mode transitions in ReadFile and WriteFile only occur if the provider does not have XP1_IFS_HANDLES bit set in dwServiceFlags1 of its WSAPROTOCOL_INFO structure.
NOTE: These providers have an unavoidable additional mode transition, even in the case of WSASend and WSARecv, although ReadFile and WriteFile will have more of them.
TIP 2: Choose the number of the concurrent worker threads allowed and the total number of the worker threads to spawn.
The number of worker threads and the number of concurrent threads that the IOCP uses are not the same thing. You can decide to have a maximum of 2 concurrent threads used by the IOCP and a pool of 10 worker threads. You have a pool of worker threads greater than or equal to the number of concurrent threads used by the IOCP so that a worker thread handling a dequeued completion packet can call one of the Win32 "wait" functions without delaying the handling of other queued I/O packets.
If there are completion packets waiting to be dequeued, the system will wake up another worker thread. Eventually, the first thread satisfies it's Wait and it can be run again. When this happens, the number of the threads that can be run is higher than the concurrency allowed on the IOCP (for example, NumberOfConcurrentThreads). However, when next worker thread calls GetQueueCompletionStatus and enters wait status, the system does not wake it up. In other words, the system tries to keep your requested number of concurrent worker threads.
Typically, you only need one concurrent worker thread per CPU for IOCP. To do this, enter 0 for NumberOfConcurrentThreads in the CreateIoCompletionPort call when you first create the IOCP.
TIP 3: Associate a posted I/O operation with a dequeued completion packet.
GetQueuedCompletionStatus returns a completion key and an overlapped structure for the I/O when dequeuing a completion packet. You should use these two structures to return per handle and per I/O operation information, respectively. You can use your socket handle as the completion key when you register the socket with the IOCP to provide per handle information. To provide per I/O operation "extend" the overlapped structure to contain your application-specific I/O-state information. Also, make sure you provide a unique overlapped structure for each overlapped I/O. When an I/O completes, the same pointer to the overlapped I/O structure is returned.
TIP 4: I/O completion packet queuing behavior.
The order in which I/O completion packets are queued in the IOCP is not necessarily the same order the Winsock2 I/O calls were made. Additionally, if a Winsock2 I/O call returns SUCCESS or IO_PENDING, it is guaranteed that a completion packet will be queued to the IOCP when the I/O completes, regardless of whether the socket handle is closed. After you close a socket handle, future calls to WSASend, WSASendTo, WSARecv, or WSARecvFrom will fail with a return code other than SUCCESS or IO_PENDING, which will not generate a completion packet. The status of the completion packet retrieved by GetQueuedCompletionStatus for I/O previously posted could indicate a failure in this case.
If you delete the IOCP itself, no more I/O can be posted to the IOCP because the IOCP handle itself is invalid. However, the system's underlying IOCP kernel structures do not go away until all successfully posted I/Os are completed.
TIP 5: IOCP cleanup.
The most important thing to remember when performing ICOP cleanup is the same when using overlapped I/O: do not free an overlapped structure if the I/O for it has not yet completed. The HasOverlappedIoCompleted macro allows you to detect if an I/O has completed from its overlapped structure.
There are typically two scenarios for shutting down a server. In the first scenario, you do not care about the completion status of outstanding I/Os and you just want to shut down as fast as you can. In the second scenario, you want to shut down the server, but you do need to know the completion status of each outstanding I/O.
In the first scenario, you can call PostQueueCompletionStatus (N times, where N is the number of worker threads) to post a special completion packet that informs the worker thread to exit immediately, close all socket handles and their associated overlapped structures, and then close the completion port. Again, make sure you use HasOverlappedIoCompleted to check the completion status of an overlapped structure before you free it. If a socket is closed, all outstanding I/O on the socket eventually complete quickly.
In the second scenario, you can delay exiting worker threads so that all completion packets can be properly dequeued. You can start by closing all socket handles and the IOCP. However, you need to maintain a count of the number of outstanding I/Os so that your worker thread can know when it is safe to exit the thread. The performance penalty of having a global I/O counter protected with a critical section for an IOCP server is not as bad as might be expected because the active worker thread does not switch out if there are more completion packets waiting in the queue.