zoukankan      html  css  js  c++  java
  • Hadoop 学习笔记之 Incompatible namespaceIDs问题

    Hadoop重新格式化后,仍然datanode启动失败,查看log:

    org.apache.hadoop.hdfs.server.datanode.DataNode: Java.io.IOException: Incompatible namespaceIDs in /home/admin/joe.wangh/hadoop/data/dfs.data.dir: namenode namespaceID = 898136669; datanode namespaceID = 2127444065 

    搜索了网上,尝试了一下,可行!

    Workaround 1: Start from scratch

    I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:

    1.     stop the cluster

    2.     delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data

    3.     reformat the namenode (NOTE: all HDFS data is lost during this process!)

    4.     restart the cluster

    When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.

    Workaround 2: Updating namespaceID of problematic datanodes

    Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:

    1.     stop the datanode

    2.     edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode

    3.     restart the datanode

    If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).

    If you wonder how the contents of VERSION look like, here's one of mine:

    #contents of <dfs.data.dir>/current/VERSION

    namespaceID=393514426

    storageID=DS-1706792599-10.10.10.1-50010-1204306713481

    cTime=1215607609074

    storageType=DATA_NODE

    layoutVersion=-13

    原因:每次namenode format会重新创建一个namenodeId,而dfs/data下包含了上次format下的namenode id,namenode format清空了namenode下的数据,但是没有清空datanode下的数据,导致启动时新创建的namenodeId与datanode 中显示的namenodeId不一致,clusterId 冲突而失败,所要做的就是每次fotmat前,清空dfs/data一下 的目录.

  • 相关阅读:
    hdu 3652 【数位dp】
    02 -body标签中相关标签
    01-html介绍和head标签
    python 核心编程第九章文件
    python核心编程 第七章 字典,集合 练习
    常用链接
    python核心编程 第七章 字典
    python核心编程 第六章 字符串,元组,列表 字符串元组只读不可变。列表可变。
    python核心编程 第五章 数字
    python读取文件中的路径内容,保存到另外的路径中
  • 原文地址:https://www.cnblogs.com/AK47Sonic/p/7198033.html
Copyright © 2011-2022 走看看