hbase 异常

1.org.apache.hadoop.hbase.ipc.CallTimeoutException 

　　a.出现情况描述：使用java API进行hbase数据的scan操作，发现有的数据可以scan到，有的数据scan不到（报超时异常）。

　　b.异常原因：执行scan的数据量太多，在设置的超时时间段里面，程序没有找到想要的结果。

　　c.异常具体样例

2017-04-21 17:53:23,253 INFO  [Thread-15] zookeeper.ZooKeeper: Initiating client connection, connectString=slave1:2181,slave2:2181,slave3:2181 sessionTimeout=90000 watcher=hconnection-0x222a0f2e0x0, quorum=slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase
2017-04-21 17:53:23,255 INFO  [Thread-15-SendThread(slave2:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave2/192.168.240.167:2181. Will not attempt to authenticate using SASL (unknown error)
2017-04-21 17:53:23,255 INFO  [Thread-15-SendThread(slave2:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.240.162:48001, server: slave2/192.168.240.167:2181
2017-04-21 17:53:23,256 INFO  [Thread-15-SendThread(slave2:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave2/192.168.240.167:2181, sessionid = 0x25b86981f993878, negotiated timeout = 40000
Exception in thread "Thread-15" java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Fri Apr 21 17:54:23 CST 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60304: row '' on table 'noc_caller' at region=noc_caller,,1492616514434.09a155eeeba545376fa7f2d2f8e95a5a., hostname=slave3,60020,1492161485334, seqNum=1422155

        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
        at com.xwtech.noc.database.HbaseConnection.scan(HbaseConnection.java:57)
        at com.xwtech.noc.database.HbaseService.getRecords(HbaseService.java:35)
        at com.xwtech.noc.database.RedisConnection.getSearchPage(RedisConnection.java:61)
        at com.xwtech.noc.httpServer.LinkClientThread.run(LinkClientThread.java:40)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Fri Apr 21 17:54:23 CST 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60304: row '' on table 'noc_caller' at region=noc_caller,,1492616514434.09a155eeeba545376fa7f2d2f8e95a5a., hostname=slave3,60020,1492161485334, seqNum=1422155

        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
        ... 5 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60304: row '' on table 'noc_caller' at region=noc_caller,,1492616514434.09a155eeeba545376fa7f2d2f8e95a5a., hostname=slave3,60020,1492161485334, seqNum=1422155
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: java.io.IOException: Call to slave3/192.168.240.161:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2, waitTime=60001, operationTimeout=60000 expired.
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:291)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1273)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:219)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
        ... 4 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2, waitTime=60001, operationTimeout=60000 expired.
        at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:73)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1247)
        ... 13 more

　　d.解决方案

　　　　1. 增加超时的时间设置（默认为60s）

　　　　　　配置文件或者程序中设置相关参数　　　

conf.setInt("hbase.client.operation.timeout", 60000);
conf.setInt("hbase.rpc.timeout", 60000);
conf.setInt("hbase.client.scanner.timeout.period", 60000);
conf.setInt("mapreduce.task.timeout", 60000);

　　　　　　配置文件中设置，只要将上面的参数以及值设置到hbase-site.xml就可以了，这里就不写了

　　　　2.设置scan的开始和结束范围（这个是重点，初学者最容易忽视的，我查了大量的异常描述，就是没有提到这一点）

　　　　　　在数据量非常大是，一定要限制每一次的查询scan范围　　

scan.setStartRow(Bytes.toBytes());
scan.setStopRow(Bytes.toBytes());

　　　　　　如果不设置查询范围，默认的查询会查询范围会冲整个hbase表的开始到结束。

　　　　3.如果查询范围已经无法进行缩小，只能增加时间，配置scan的参数等等，进行简单的优化了。