zoukankan      html  css  js  c++  java
  • 【原创】大叔问题定位分享(25)ambari metrics collector内置standalone hbase启动失败

    ambari metrics collector内置hbase目录位于

    /usr/lib/ams-hbase

    配置位于

    /etc/ams-hbase/conf

    通过ruby启动

    /usr/lib/ams-hbase/bin/hirb.rb

    实际的启动命令为

    /usr/lib/ams-hbase/bin/hbase-daemon.sh --config /etc/ams-hbase/conf foreground_start master

    但是启动一段时间报错:

    java.lang.RuntimeException: Master not initialized after 200000ms
    at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
    at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197)
    at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3100)

    查看日志发现第一个错:

    2019-01-16 16:20:29,295 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: FileSystem doesn't support setStoragePolicy; HDFS-6584, HDFS-9345 not available. This is normal and expected on earlier Hadoop versions.
    java.lang.NoSuchMethodException: org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path, java.lang.String)
    at java.lang.Class.getDeclaredMethod(Class.java:2130)
    at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:533)
    at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:514)
    at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:482)
    at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:193)
    at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:258)
    at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5571)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1025)
    at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1022)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    这个错可以忽略,然后是第二个错:

    2019-01-17 15:49:08,730 ERROR [Thread-24] master.HMaster: Failed to become active master
    java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1082)
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423)
    at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:714)
    at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1398)
    at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:857)
    at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)
    at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)
    at java.lang.Thread.run(Thread.java:748)

    这个可以通过设置hbase.unsafe.stream.capability.enforce=false解决,详见 https://www.cnblogs.com/barneywill/p/10283076.html

    最后发现很多warn:

    2019-01-17 18:22:12,657 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
    2019-01-17 18:22:13,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
    2019-01-17 18:22:15,660 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
    2019-01-17 18:22:19,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
    2019-01-17 18:22:27,661 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
    2019-01-17 18:22:43,662 WARN [Thread-24] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1547720528009, server=test-server-01,16020,1547717775199}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.

    清空hbase.rootdir,hbase.tmp.dir,hbase.zookeeper.property.dataDir这3个目录,同时设置好用户权限,然后重启hbase即可;

  • 相关阅读:
    CEF3研究(三)
    [你必须知道的.NET]第二十一回:认识全面的null
    [你必须知道的.NET]第二十回:学习方法论
    [你必须知道的.NET]第十九回:对象创建始末(下)
    [你必须知道的.NET]第十八回:对象创建始末(上)
    [你必须知道的.NET]第十七回:貌合神离:覆写和重载
    纯js的N级联动列表框 —— 基于jQuery
    基于存储过程的MVC开源分页控件
    【程序7】统计出所输入字符中英文字母、空格、数字和其它字符的个数
    【程序6】 求最大公约数和最小公倍数
  • 原文地址:https://www.cnblogs.com/barneywill/p/10283849.html
Copyright © 2011-2022 走看看