ConnectionPoolTimeoutException: Timeout waiting for connection from pool

背景

今天在通过监控系统发现一个错误，错误如下

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

动作

1. 起初怀疑是连接池设置过小，故查看代码对连接池大小的设置

2. 发现连接池设置已经足够大，我们的业务完全用不到这么多的连接数，故怀疑连接用完没有被释放。查看服务器上连接数

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

查看代码， 发现确实没有释放连接，如下

HttpResponse response = httpclient.execute(httpPost);

List<String> lines = IOUtils.readLines(response.getEntity().getContent(), "utf-8");
StringBuffer res = new StringBuffer();
for (String line : lines) {
   res.append(line);
}
       
ChatMessage chatMessage = getChatMessage(sendChatMessageDTO, replyMessageInfo, sendMessageesResultBO);
return chatMessage;

解决

即然发现了问题，我们把连接用完归还到连接池即可。

HttpPost httpPost = new HttpPost(connectUrl);
InputStream inputStream = null;
try {
            
    HttpResponse response = httpclient.execute(httpPost);
    inputStream = response.getEntity().getContent();
    List<String> lines = IOUtils.readLines(inputStream, "utf-8");
} catch (Exception e){
    logger.error("connect error", e);
    if(httpPost != null){
    	httpPost.abort();
     }
} finally {
    if(inputStream != null){
        try {
            inputStream.close();
         } catch (Exception e){
            logger.error("", e);
         }
    }
}

思考(基于版本4.5)

• httpclient 连接池原理是什么样的呢？

连接池中主要有三个属性，用于不同状态的连接

//存放正在使用的连接
private final Set<E> leased;
//存放可用的连接
private final LinkedList<E> available;
//存放 pending 的连接 
private final LinkedList<PoolEntryFuture<E>> pending;

• 默认连接池的大小是多少？

我们查看官方代码(版本4.5)， 发现其默认总大小为20，每个 route 的默认最大为 2

 public PoolingHttpClientConnectionManager(
        final HttpClientConnectionOperator httpClientConnectionOperator,
        final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
        final long timeToLive, final TimeUnit timeUnit) {
        super();
        this.configData = new ConfigData();
        this.pool = new CPool(new InternalConnectionFactory(
                this.configData, connFactory), 2, 20, timeToLive, timeUnit);
        this.pool.setValidateAfterInactivity(2000);
        this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator");
        this.isShutDown = new AtomicBoolean(false);
    }

• close 的原理是怎样呢？

有两种 close

1. CloseableHttpResponse.close(): 这种方式是将用完的连接放回连接池的可用集合，并不是将连接真正关闭。内部调用的是 release()
2. httpClient.close(): 这种是关闭连接池，将池中的所有资源释放，即真正关闭连接