dubbo常见异常及原因分析（<i>UPDATING...</i>）

说来真巧，2018年下半年时使用Thrift作为局域网服务rpc通信框架，整理了一下Thrift常见问题，博客访问量较高，也有同学私信讨论Thrift遇到的问题。此后的工作中，用dubbo比较多，好脑袋不如烂笔头，经常排查dubbo问题，还是整理一下分享出来。

■ com.alibaba.dubbo.rpc.RpcException: No provider available for service

异常描述1（dubbo2.5.3）：

com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method *** in the service com..**Service. No provider available for the service test/com..***Service from registry 192.168.40.21:2181 on the consumer 192.168.40.51 using the dubbo version 2.5.3. Please check if the providers have been started and registered.

异常描述2（dubbo2.7.3）：

org.apache.dubbo.rpc.RpcException: No provider available from registry 192.168.40.84:2181 for service dev/com..**Service on consumer 192.168.48.96 use dubbo version 2.7.3, please check status of providers(disabled, not registered or in blacklist).

 异常堆栈：

com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method getAgent in the service com.emaxcard.agent.modules.agent.service.AgentService. 
No provider available for the service test/com.emaxcard.agent.modules.agent.service.AgentService from registry 192.168.40.21:2181 
on the consumer 192.168.40.51 using the dubbo version 2.5.3. Please check if the providers have been started and registered.
    at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:246)
    at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:55)
    at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
    at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
    at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
    at com.alibaba.dubbo.common.bytecode.proxy1.getAgent(proxy1.java)
    at com.yft.service.impl.TAgentServiceImpl.getAgentInfo(TAgentServiceImpl.java:86)

 产生原因：

No provider available--很明显，就是说指定的服务在注册中心不存在。

Please check if the providers have been started and registered.-请确保服务提供者已经启动，并且已经注册到注册中心。

1. dubbo服务没有启动

2. dubbo服务不可用 / 在dubboadmin上做了disable操作

 3. 如果使用了dubbo分组（group），可能是指定为这个group的dubbo服务没有启动

■ com.alibaba.dubbo.remoting.RemotingException: message can not send, because channel is closed

异常描述：

com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method *** in the service com..***Service. Tried 1 times of the providers [192.168.40.83:20899] (1/1) from the registry 192.168.40.21:2181 on the consumer 192.168.40.51 using the dubbo version 2.5.3.----Caused by: com.alibaba.dubbo.remoting.RemotingException: message can not send, because channel is closed . 

异常堆栈：

2021-11-04 13:55:36.596 [ERROR] [riskOder_synchronizePlatOrder_1636005060003S457] [com.yft.service.impl.TAgentServiceImpl:89] *****获得渠道商信息，dubbo调用异常：
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method getAgent in the service com.emaxcard.agent.modules.agent.service.AgentService. 
Tried 1 times of the providers [192.168.40.83:20899] (1/1) from the registry 192.168.40.21:2181 on the consumer 192.168.40.51 using the dubbo version 2.5.3. 
Last error is: Failed to invoke remote method: getAgent, 
provider: dubbo://192.168.40.83:20899/com.emaxcard.agent.modules.agent.service.AgentService?anyhost=true&application=youfu-merchant-consumer&bean.name=ServiceBean:com.emaxcard.agent.modules.agent.service.AgentService&check=false&default.check=false&default.group=test&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&interface=com.emaxcard.agent.modules.agent.service.AgentService&methods=saveAgent,selectSecondAndThirdAgent,getAgentByAgentCode,changeLoginPass,selectFirstAgent,listAgentByAgentName,initPassword,updateAgent,page,agentList,getAgentByName&pid=17492&register=true&release=2.7.3&retries=0&revision=1.0.1-SNAPSHOT&side=consumer&timeout=3000&timestamp=1635997957739, 
cause: message can not send, because channel is closed . 
url:dubbo://192.168.40.83:20899/com.emaxcard.agent.modules.agent.service.AgentService?anyhost=true&application=youfu-merchant-consumer&bean.name=ServiceBean:com.emaxcard.agent.modules.agent.service.AgentService&check=false&codec=dubbo&default.check=false&default.group=test&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&heartbeat=60000&interface=com.emaxcard.agent.modules.agent.service.AgentService&methods=saveAgent,selectSecondAndThirdAgent,getAgentByAgentCode,changeLoginPass,selectFirstAgent,listAgentByAgentName,initPassword,updateAgent,page,agentList,getAgentByName&pid=17492&register=true&release=2.7.3&retries=0&revision=1.0.1-SNAPSHOT&side=consumer&timeout=3000&timestamp=1635997957739
    at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
    at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:227)
    at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)
    at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
    at com.alibaba.dubbo.common.bytecode.proxy1.getAgent(proxy1.java)
    at com.yft.service.impl.TAgentServiceImpl.getAgentInfo(TAgentServiceImpl.java:86)
    at ...
Caused by: com.alibaba.dubbo.remoting.RemotingException: message can not send, because channel is closed . 
url:dubbo://192.168.40.83:20899/com.emaxcard.agent.modules.agent.service.AgentService?anyhost=true&application=youfu-merchant-consumer&bean.name=ServiceBean:com.emaxcard.agent.modules.agent.service.AgentService&check=false&codec=dubbo&default.check=false&default.group=test&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&heartbeat=60000&interface=com.emaxcard.agent.modules.agent.service.AgentService&methods=updateStateAgent,saveAgent,selectSecondAndThirdAgent,getAgentByAgentCode,changeLoginPass,listAgentByName,getAgent,selectFirstAgent,listAgentByAgentName,initPassword,belongingSubordinate,reset,updateAgent,getHomeInfo,page,agentList,getAgentByName&pid=17492&register=true&release=2.7.3&retries=0&revision=1.0.1-SNAPSHOT&side=consumer&timeout=3000&timestamp=1635997957739
    at com.alibaba.dubbo.remoting.transport.AbstractClient.send(AbstractClient.java:268)
    at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request(HeaderExchangeChannel.java:112)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeClient.request(HeaderExchangeClient.java:91)
    at com.alibaba.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:81)
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)
    at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)
    at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke$original$YXlgcHUl(MonitorFilter.java:75)
    at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke$original$YXlgcHUl$accessor$WxlrIuUu(MonitorFilter.java)
    at com.alibaba.dubbo.monitor.support.MonitorFilter$auxiliary$LX9nuPMk.call(Unknown Source)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
    at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
    at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
    at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
    at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)
    at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
    at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
    ... 69 more

产生原因：

dubbo服务没有启动/dubbo服务挂掉了。此时，consumer会尝试重新建立连接重新调用。重试次数取决于为属性dubbo.consumer.retries配置的值。retries的默认值为2，对于增删改操作，服务端要注意做幂等控制，避免出现重复处理的情况。当然，为了避免重复调用，有时我们会设置dubbo.consumer.retries=0，上面异常里Tried 1 times就是未重试。

■ org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer

 异常描述：

Failed to invoke the method *** in the service com..***Service. Tried 1 times of the providers [192.168.40.48:20881] (1/1) from the registry 192.168.40.20:2181 on the consumer 192.168.40.85 using the dubbo version 2.7.3. Last error is: Invoke remote method timeout.  cause: org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2021-09-26 10:09:04.842, end time: 2021-09-26 10:09:34.871, client elapsed: 1 ms, server elapsed: 30028 ms, timeout: 30000 ms.

异常堆栈：

2021-09-26 10:09:27.888 [][] [DubboServerHandler-192.168.40.85:20888-thread-199] INFO  com.emax.zhenghe.common.tracking.DubboTraceFilter:62 - found slow method:com.emax.zhenghe.portalapi.modules.enterprise.service.EnterpriseInfoApi:onlineAuthEnterpriseStatus,exec time:30030 ms
2021-09-26 10:09:34.873 [][] [EnterpriseInfoApiImpl_onlineAuthEnterpriseStatus1632622144838] ERROR c.e.z.rpcapi.provider.portal.EnterpriseInfoApiImpl:95 - 在线企业认证状态获取失败
org.apache.dubbo.rpc.RpcException: Failed to invoke the method queryOrganizationStatus in the service com.auth.api.service.ContractSignOrganizationService. 
Tried 1 times of the providers [192.168.40.48:20881] (1/1) from the registry 192.168.40.20:2181 on the consumer 192.168.40.85 using the dubbo version 2.7.3. 
Last error is: Invoke remote method timeout. method: queryOrganizationStatus, 
provider: dubbo://192.168.40.48:20881/com.auth.api.service.ContractSignOrganizationService?anyhost=true&application=zhenghe-provider&bean.name=ServiceBean:com.auth.api.service.ContractSignOrganizationService&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=test&interface=com.auth.api.service.ContractSignOrganizationService&lazy=false&methods=organizationStatusUpdate,getListByParam,getContractSignInfo,queryOrganizationStatus,organizationCreate,update,insert,list,updateOrganizationCreateStatus&pid=29399&qos.enable=false&register=true&register.ip=192.168.40.85&release=2.7.3&remote.application=auth-service-channel&retries=0&side=consumer&sticky=false&timeout=30000&timestamp=1632622128250, 
cause: org.apache.dubbo.remoting.TimeoutException: Waiting server-side response timeout by scan timer. start time: 2021-09-26 10:09:04.842, end time: 2021-09-26 10:09:34.871, client elapsed: 0 ms, server elapsed: 30028 ms, timeout: 30000 ms, request: Request [id=71603, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=queryOrganizationStatus, parameterTypes=[class java.lang.String], arguments=[1626769691952966], attachments={input=696, path=com.auth.api.service.ContractSignOrganizationService, sw8-x=0- , sw8=1-M2EyNWZlODhlZGUzNDAwNjlhYmFiMmYzZWJhZWMyYjkuNzkuMTYzMjYyMjE0NDgxNzAyMjc=-NTVmOTViYWI1M2UwNGI4ZWE1ZTMxZTA5NmY4ZDM1MDEuMzgwLjE2MzI2MjIxNDQ4Mzg1MDc4-6-emhlbmdoZS1ycGNhcGktcHJvdmlkZXI=-emhlbmdoZS1ycGNhcGktcHJvdmlkZXJAYm9nb24=-dGVzdC9jb20uZW1heC56aGVuZ2hlLnBvcnRhbGFwaS5tb2R1bGVzLmVudGVycHJpc2Uuc2VydmljZS5FbnRlcnByaXNlSW5mb0FwaS5vbmxpbmVBdXRoRW50ZXJwcmlzZVN0YXR1cyhMb25nKQ==-MTkyLjE2OC40MC40ODoyMDg4MQ==, sw8-correlation=, interface=com.auth.api.service.ContractSignOrganizationService, version=0.0.0, timeout=30000, group=test}]], channel: /192.168.40.85:55882 -> /192.168.40.48:20881
	at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113)
	at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:248)
	at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:78)
	at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:55)
	at org.apache.dubbo.common.bytecode.proxy10.queryOrganizationStatus(proxy10.java)
	at ...
Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.TimeoutException: 
Waiting server-side response timeout by scan timer. start time: 2021-09-26 10:09:04.842, end time: 2021-09-26 10:09:34.871, client elapsed: 0 ms, server elapsed: 30028 ms, timeout: 30000 ms, 
request: Request [id=71603, version=2.0.2, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=queryOrganizationStatus, parameterTypes=[class java.lang.String], arguments=[1626769691952966], attachments={input=696, path=com.auth.api.service.ContractSignOrganizationService, sw8-x=0- , sw8=1-M2EyNWZlODhlZGUzNDAwNjlhYmFiMmYzZWJhZWMyYjkuNzkuMTYzMjYyMjE0NDgxNzAyMjc=-NTVmOTViYWI1M2UwNGI4ZWE1ZTMxZTA5NmY4ZDM1MDEuMzgwLjE2MzI2MjIxNDQ4Mzg1MDc4-6-emhlbmdoZS1ycGNhcGktcHJvdmlkZXI=-emhlbmdoZS1ycGNhcGktcHJvdmlkZXJAYm9nb24=-dGVzdC9jb20uZW1heC56aGVuZ2hlLnBvcnRhbGFwaS5tb2R1bGVzLmVudGVycHJpc2Uuc2VydmljZS5FbnRlcnByaXNlSW5mb0FwaS5vbmxpbmVBdXRoRW50ZXJwcmlzZVN0YXR1cyhMb25nKQ==-MTkyLjE2OC40MC40ODoyMDg4MQ==, sw8-correlation=, interface=com.auth.api.service.ContractSignOrganizationService, version=0.0.0, timeout=30000, group=test}]], 
channel: /192.168.40.85:55882 -> /192.168.40.48:20881
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:56)
	at com.emax.zhenghe.common.tracking.DubboTraceFilter.invoke(DubboTraceFilter.java:32)
	at ... 78 common frames omitted

产生原因：

dubbo调用超时。服务端处理超时。dubbo超时时间通过dubbo.provider.timeout或dubbo.consumer.timeout来设置。dubbo默认超时时间是1000ms=1秒。客户端dubbo调用超时后会尝试重试，重试次数取决于上文的retries属性。see dubbo超时重试

在生产环境，要控制dubbo超时时间，尤其是consumer端。timeout过长容易引起雪崩。

■ com.alibaba.dubbo.rpc.RpcException: Forbid consumer *** access service *** from registry *** use dubbo version 2.5.3, Please check registry access list (whitelist/blacklist).

异常描述：

com.alibaba.dubbo.rpc.RpcException: Forbid consumer 10.0.2.224 access service com.emax.zhenghe.portalapi.modules.sms.service.EmaxSettlementSmsRecordService from registry 10.0.2.8:2181 use dubbo version 2.5.3, Please check registry access list (whitelist/blacklist).

异常堆栈：

2021-09-16 22:24:03.290 [ERROR] [a54f25179c9a4062abe4457cebb65a34] [paymentQuery_timer_1631802240392CA90D] [com.cn.yft.bank.p600001.own.PaymentTransQueryByOWN:298] 获取下发成功人员信息失败，发送通知短信异常：
com.alibaba.dubbo.rpc.RpcException: Forbid consumer 10.0.2.224 access service com.emax.zhenghe.portalapi.modules.sms.service.EmaxSettlementSmsRecordService from registry 10.0.2.8:2181 use dubbo version 2.5.3, Please check registry access list (whitelist/blacklist).
        at com.alibaba.dubbo.registry.integration.RegistryDirectory.doList(RegistryDirectory.java:579)
        at com.alibaba.dubbo.rpc.cluster.directory.AbstractDirectory.list(AbstractDirectory.java:73)
        at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.list(AbstractClusterInvoker.java:260)
        at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:219)
        at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)

产生原因：

当某个服务的provider有变化时就会被调用，例如zookeeper上某个服务的provider目录里的内容发生变化，则zk监听器会被触发，由于provider的数量会发生变化，例如有一个新的provider启动了，有一个provider下线了，所以必须刷新本地的对provider的连接。这时会涉及到服务提供者与消费者的分组（group）、版本（version）、接口名，如果不一致，就会报这个“forbid consumer access service”的错误。可参见csdn这篇帖子。