那些年我们踩到过的坑（二）：3.1 版 MultiThreadedHttpConnectionManager 未releaseConnection导致应用服务器宕机

昨天短信服务又宕机了，jstack打出线程信息发现 所有线程池的线程都在wait，栈信息如下：

at java.lang.Object.wait(Native Method)
- waiting on [0x000000070754fb60] (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:518)
- locked [0x000000070754fb60] (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.dhf.sms.util.HttpUtil.getHtml(HttpUtil.java:75)
at com.dhf.sms.core.impl.MDSmsserviceImpl.send(MDSmsserviceImpl.java:165)
at com.dhf.sms.core.SmsTask.run(SmsTask.java:156)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at com.kuaidadi.framework.thread.bean.TaxiFutureTask.run(TaxiFutureTask.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

发现所有线程都卡在MultiThreadedHttpConnectionManager 的doGetConnection方法：

   hostPool.waitingThreads.addLast(waitingThread);
                        connectionPool.waitingThreads.addLast(waitingThread);
                        connectionPool.wait(timeToWait);//卡在这

查看代码和google，发现应该是connection连接未释放的问题的。如果不调用getMethod.getResponseBodyAsString()，就不会释放connection。并且我们没有指定maxConnectionsPerHost，默认为2，所以当以下代码在第16行抛出exception后连接不会被释放掉，一旦发生两次，那么仅有的2个connection都被占用，新进来的线程就卡住了。

 public static String getHtml(String url,String charSet,Integer channelId) throws IOException, SmsHttpStatusException {
        StaticLogInfo staticLogInfo = new StaticLogInfo();
        staticLogInfo.setCmd("Channel "+channelId+"("+getHost(url)+")");
        GetMethod getMethod = null;
        try {
            if (StringUtils.isBlank(charSet)) {
                charSet = SmsConstant.DEFAULT_CHARSET;
            }
            
            getMethod = new GetMethod(url);
            int statusCode = httpClient.executeMethod(getMethod);
            if (statusCode != HttpStatus.SC_OK 
                    && statusCode != HttpStatus.SC_MOVED_PERMANENTLY 
                    && statusCode != HttpStatus.SC_MOVED_TEMPORARILY) {    
                log.error("statusCode:" + statusCode + " url请求失败========>" + url);
                throw new SmsHttpStatusException("请求失败");
            }
            
            String responseBody = new String(getMethod.getResponseBodyAsString().getBytes(charSet),SmsConstant.DEFAULT_CHARSET);
            return responseBody;
        } catch (IOException e) {
            staticLogInfo.setResultCode("1");
            throw e;
        } catch (SmsHttpStatusException e) {
            staticLogInfo.setResultCode("1");
            throw e;
        } finally {
           
            reqLogger.info(staticLogInfo);
        }
    }
    

解决办法是在finally里释放connection：

finally {
            if(getMethod!=null){
                getMethod.releaseConnection();
            }
            reqLogger.info(staticLogInfo);
        }

并且增加最大连接数：

httpConnectionManagerParams.setMaxTotalConnections(100);
        httpConnectionManagerParams.setDefaultMaxConnectionsPerHost(20);

参考：http://blog.csdn.net/xiaojianpitt/article/details/7818971