zoukankan      html  css  js  c++  java
  • Scala操作Hbase空指针异常java.lang.NullPointerException处理

    Hbase版本:Hortonworks Hbase 1.1.2

    问题描述:
    使用Scala操作Hbase时,发生空指针异常(java.lang.RuntimeException: java.lang.NullPointerException),异常如下:

    18/09/13 20:28:51 ERROR Executor: Exception in task 1.0 in stage 361.0 (TID 18925)
    java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
    at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
    at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1255)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1161)
    at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
    at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
    at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
    at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
    at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449)
    at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1052)
    at scala.OperateMysqlTable$$anonfun$main$1$$anonfun$apply$3$$anonfun$apply$4.apply(AccompanyAnalyze.scala:177)
    at scala.OperateMysqlTable$$anonfun$main$1$$anonfun$apply$3$$anonfun$apply$4.apply(AccompanyAnalyze.scala:161)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1954)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1954)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:99)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:399)
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:552)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1191)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1158)
    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    ... 22 more

    问题分析:

    该异常为客户端去zookeeper获取Hbase相关信息时,没有找到,抛出空指针异常

    问题原因:
    不同发行商的hbase的在zookeeper的znode设置不同
    Hortonworks的hbase在zookeeper中znode路径为/hbase-unsecure

    查看该路径下hbase的相关信息

    [zk: localhost:2181(CONNECTED) 1] ls /hbase-unsecure
    [replication, meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, region-in-transition, online-snapshot, acl, switch, master, running, recovering-regions, draining, namespace, hbaseid, table]


    解决办法:
    我使用的是Hortonworks的Hbase,所以要在Hbase的conf中添加如下配置信息

    conf.set("zookeeper.znode.parent", "/hbase-unsecure")

    如果是Cloudera的Hbase,添加配置信息如下:

    conf.set("zookeeper.znode.parent", "/hbase")

    添加后,重新执行程序,Hbase表创建成功,问题解决

  • 相关阅读:
    [转] linux虚拟带库操作笔记
    [转] 在Linux平台使用mhVTL虚拟化磁带库
    [转] mhvtl虚拟磁带库的安装与应用
    RHEL 集群(RHCS)配置小记 -- 文档记录
    Oracle Database常用补丁集Patch号及各版本PSU
    [原创] ORA-01536 超出表空间 'xxxx' 的空间限额
    div里面的图片超出div的宽度,自动调整大小。
    SDNU 1269.整数序列(水题)
    SDNU 1254.Problem B. SOS(水题)
    SDNU 1245.这题超难的,建议先做其他的,看都不要看(思维)
  • 原文地址:https://www.cnblogs.com/felixzh/p/10478381.html
Copyright © 2011-2022 走看看