ThreadPool growth related important details
When CLR loaded into a process, it creates a ThreadPool. Each CLR has its own ThreadPool which shares between all AppDomains within the CLR. The ThreadPool provides two types of threads: Worker and I/O Completion Port(IOCP).
Worker threads are used for things like processing the Task.Run(...) or ThreadPool.QueueUserWorkItem(...) methods.
IOCP threads are used when asynchronous IO happens, such as when reading from the network.
The thread pool provides new worker or I/O completion threads on demand(without any throttling) until it reaches the "Minimum" setting for each type of thread. By Default, the minimum number of threads is set to the number of processors on a system.
Once the number of existing(busy) threads hits the "minimum" number of threads, the ThreadPool will throttle the rate at which it injects new threads to one thread per 500 ms.
Typically, if your system gets a burst of work needing an IOCP thread, it will process that work quickly. However, if the burst of work is more than the configured "Minimum" setting, there will be some delay in processing some of work as the ThreadPool watis for one of two things to happen:
1.An existing thread becomes free to process the work.
2.No existing thread becomes free for 500ms, so a new thread is created.
Basically, it means that when the number of Busy threads is greater than Min threads, you are likely paying 500ms delay before network traffic is processed by the application. Also, it is important to note that when an existing thread stays idle for longer than 15s(based on what i remember), it will be cleaned up and this cycle of growth and shrinkage can repeat.
Recommendation
Given this information, we strongly recommend that customers set the minimum configuration value for IOCP and WORKER threads to something larger than the default value. We can't give one-size-fits-all guidance. so each customer needs to fine-tune this setting to their specific needs. A good starting place is 200 or 300, then test and tweak as needed.
Minimum threads setting configuration
a. using the ThreadPool.SetMinThreads(...) method in application during starting. this is preferred mechanism.
b. using the minIoThreads or minWorkerThreads configuration setting under the <processModel> configuration element in Machine.config, usually located at %SytemRoot%Microsoft.NETFramework[versionNumber]CONFIG.