消息服务dubbo接口性能压测性能优化案例

最近项目中的消息服务做了运营商的改动，导致这个服务做了重新开发

压测脚本如下：

 开启200线程压测：

tps只有200-300之间，平均耗时在700ms左右

开启500线程压测

500并发压测，发现平均耗时到了1100ms，感觉肯定存在问题，这时候tps才400多

通过打印线程的dump日志：截取部分

2019-01-17 18:05:18
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f971400d800 nid=0x24cc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"ZodiacServer-172.20.20.115:6004-thread-1670" daemon prio=10 tid=0x00007f94f8138800 nid=0x24b2 waiting on condition [0x00007f94ca0c6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007801d5c20> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at ooh.bravo.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:27)
at ooh.bravo.logging.Journal.info0(Journal.java:46)
at ooh.bravo.logging.LoggerProxy.logging(LoggerProxy.java:25)
at ooh.bravo.logging.LoggerProxy.info(LoggerProxy.java:57)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:135)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$6f7ebe16.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x000000079e4182e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ActiveMQ Session Task-99" prio=10 tid=0x00007f969007d000 nid=0x24b0 waiting on condition [0x00007f94ca18a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000790e42e98> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

"bravo-HikariCP connection adder" daemon prio=10 tid=0x00007f9610075800 nid=0x248e runnable [0x00007f94ca1cb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000780150d20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

"ZodiacServer-172.20.20.115:6004-thread-1669" daemon prio=10 tid=0x00007f94d40ab000 nid=0x248d waiting on condition [0x00007f94ca20b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007801d5c20> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at ooh.bravo.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:27)
at ooh.bravo.logging.Journal.info0(Journal.java:46)
at ooh.bravo.logging.LoggerProxy.logging(LoggerProxy.java:25)
at ooh.bravo.logging.LoggerProxy.info(LoggerProxy.java:57)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:69)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x000000079d2cf080> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1668" daemon prio=10 tid=0x00007f94d40a9000 nid=0x248c runnable [0x00007f94ca24c000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$6f7ebe16.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x000000079d2d0798> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1667" daemon prio=10 tid=0x00007f94d40a7800 nid=0x248b waiting on condition [0x00007f94ca28e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000780386568> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- None

"ZodiacServer-172.20.20.115:6004-thread-1666" daemon prio=10 tid=0x00007f94d40a6000 nid=0x248a waiting on condition [0x00007f94ca30f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007801d5c20> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at ooh.bravo.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:27)
at ooh.bravo.logging.Journal.info0(Journal.java:46)
at ooh.bravo.logging.LoggerProxy.logging(LoggerProxy.java:25)
at ooh.bravo.logging.LoggerProxy.info(LoggerProxy.java:57)
at com.neo.xnol.message.zxsms.engine.impl.DefaultChannelRouteSelector.select(DefaultChannelRouteSelector.java:57)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:41)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$6f7ebe16.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x000000079d2eb5c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1665" daemon prio=10 tid=0x00007f94d40a4800 nid=0x2489 waiting on condition [0x00007f94ca350000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007801d5c20> (a java.util.concurrent.locks.ReentrantLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at ooh.bravo.logging.slf4j.Slf4jLogger.info(Slf4jLogger.java:27)
at ooh.bravo.logging.Journal.info0(Journal.java:46)
at ooh.bravo.logging.LoggerProxy.logging(LoggerProxy.java:25)
at ooh.bravo.logging.LoggerProxy.info(LoggerProxy.java:57)
at com.neo.xnol.message.zxsms.engine.filter.PriorityFilter.doFilter(PriorityFilter.java:36)
at com.neo.xnol.message.zxsms.engine.filter.PriorityFilter.doFilter(PriorityFilter.java:20)
at com.neo.xnol.message.zxsms.engine.impl.DefaultChannelRouteSelector.select(DefaultChannelRouteSelector.java:50)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:41)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$6f7ebe16.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x000000079d2fd160> (a java.util.concurrent.ThreadPoolExecutor$Worker)

发现很多线程都在处理at ch.qos.logback.classic.Logger.info(Logger.java:579)日志相关的

通过检查项目中的logback.xml文件，发现日志的配置文件开启了控制台输出，导致线程阻塞在日志输出

注释这一部分，重新压测，发现500并发，tps已经提高到了560左右

但是平均耗时还是差不多880多ms，那么需要继续优化

日志问题解决后，重新dump线程日志，发现log相关的日志已经不存在了，说明日志瓶颈问题已经解决

Locked ownable synchronizers:
- <0x0000000799a406e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1441" daemon prio=10 tid=0x00007f4da005b800 nid=0xa08 waiting on condition [0x00007f4d6cbb1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007dab96528> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:136)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799a3fae0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1449" daemon prio=10 tid=0x00007f4d740ba000 nid=0xa07 waiting on condition [0x00007f4d6cbf2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007cd6538d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:136)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799a6f400> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1379" daemon prio=10 tid=0x00007f4d900a3000 nid=0xa06 waiting on condition [0x00007f4d6cc74000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d6ed1768> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:136)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799a15e80> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1443" daemon prio=10 tid=0x00007f4d9402b000 nid=0xa05 waiting on condition [0x00007f4d6ccb5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e6f9c7a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:136)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799a80d78> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1430" daemon prio=10 tid=0x0000000001ba1000 nid=0xa04 waiting on condition [0x00007f4d6cd37000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007fe038380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:136)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799ac7018> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ZodiacServer-172.20.20.115:6004-thread-1437" daemon prio=10 tid=0x00007f4d7c091800 nid=0xa03 runnable [0x00007f4d6cd78000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.neo.xnol.message.http.HttpUtil.postBody(HttpUtil.java:120)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.doSendSms(WxchinaMessageChannel.java:97)
at com.neo.xnol.message.zxsms.adapter.wxchina.WxchinaMessageChannel.sendSingle(WxchinaMessageChannel.java:70)
at com.neo.xnol.message.zxsms.adapter.DefaultMessageChannelAdapter.sendSingle(DefaultMessageChannelAdapter.java:33)
at com.neo.xnol.message.zxsms.engine.impl.SmsMessageProcessor.process(SmsMessageProcessor.java:60)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher.dispatch(DefaultMessageDispatcher.java:48)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$FastClassBySpringCGLIB$$fb6cd581.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.neo.xnol.message.zxsms.baffle.aspect.BaffleAspect.execute(BaffleAspect.java:42)
at sun.reflect.GeneratedMethodAccessor283.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.neo.xnol.message.zxsms.engine.impl.DefaultMessageDispatcher$$EnhancerBySpringCGLIB$$58597f1f.dispatch(<generated>)
at com.neo.xnol.message.zxsms.facade.impl.MessageGatewayFacadeImpl.sendSingle(MessageGatewayFacadeImpl.java:49)
at ooh.bravo.zodiac.common.bytecode.Wrapper37.invokeMethod(Wrapper37.java)
at ooh.bravo.zodiac.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:26)
at ooh.bravo.zodiac.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:54)
at ooh.bravo.zodiac.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:35)
at ooh.bravo.zodiac.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:35)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.monitor.support.MonitorFilter.invoke(MonitorFilter.java:40)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ContextFilter.invoke(ContextFilter.java:36)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DynamicLinkTrackingFilter.invoke(DynamicLinkTrackingFilter.java:57)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.DubboProviderFilter.invoke(DubboProviderFilter.java:34)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:18)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.EchoFilter.invoke(EchoFilter.java:19)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.filter.ProviderHandlingLoggingFilter.invoke(ProviderHandlingLoggingFilter.java:48)
at ooh.bravo.zodiac.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:78)
at ooh.bravo.zodiac.rpc.protocol.zodiac.ZodiacProtocol$1.reply(ZodiacProtocol.java:78)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:59)
at ooh.bravo.zodiac.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:148)
at ooh.bravo.zodiac.remoting.transport.DecodeHandler.received(DecodeHandler.java:30)
at ooh.bravo.zodiac.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x0000000799aa4eb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

 

发现线程的耗时，卡在java.io.InputStream:read()方法上

实际上不管是使用什么协议（HTTP，HTTPS，JDBC 等）请求远程服务，Java VM最终都将委托给Socket去实现。首先会通过connect方法与远程服务建立连接，连接建立成功后就可以 write/read 与远程服务实现数据通信。

Socket.connect()、Socket.write()、Socket.read() 方法调用都被称为阻塞IO调用。这些调用由于网络延时、服务器响应缓慢等问题往往会导致线程挂起。

由线程堆栈可以看到大量的线程阻塞到socket读的地方，推断可能是由于网络延迟或FCM服务连接后没有响应造成 java.net.SocketInputStream.socketRead0一直阻塞等待读取

通过耗时链路监控，发现java.io.InputStream的read调用耗时在600ms左右

完全不正常

由于项目配置了发送短信的运营商采用的是挡板程序，其实挡板程序很简单，就是一个post请求，输入参数后，直接返回一个状态码而已，没有任何业务逻辑处理

然后返回一个状态码

这就是这个挡板，不错任何业务处理

我们的消息服务内部是通过httpclient调用这个http://172.20.20.160:9500/api/msg/xuanwu/massSend接口发送短信给用户

在压测中，发现所有的耗时基本卡在这个httpclient处理上，而httpclient已经配置线程池的模型，并配置连接池为300，已经够用了

而这个httpclient实际上我们消息服务内部调用运营商的接口地址，然后发送短信，返回给我们

通过监控发现和开发确认分析，问题可能出现在挡板程序上，那么现在通过上面直接压测挡板

发现挡板在500并发，耗时也是在800多ms，tps也是500多

通过监控挡板程序，python的挡板程序的cpu只能使用单核，cpu已经100%了，基本可以确定消息服务压测的结果和挡板的效果一样，进一步怀疑是挡板程序造成的

为了解决这个问题，重新通过nginx做负载均衡，启动2个挡板程序，再次压测，发现只压测挡板程序已经tps达到了800多，性能提升很明显

然后直接开启三个挡板程序，挡板tps已经达到1200左右：

挡板服务器已经满负载了

然后重新压测消息服务接口，看看效果怎么样？

但是挡板已经满负载了

到此，问题排除完成