zoukankan      html  css  js  c++  java
  • HDFS-HA高可用 | Yarn-HA

     HDFS-HA

    HA(High Available),即高可用(7*24小时不中断服务)

    单点故障即有一台机器挂了导致全部都挂了;HA就是解决单点故障,就是针对NameNode;

    主Active:读写、从standby只读;所依赖的服务都必须是高可用的;

    两种解决共享空间的方案:NFS、QJM(主流的)

    奇数台机器;QJM跟zookeeper(数据全局一致;半数以上的机器存活就可以提供服务)高可用的方式一模一样,;

    QJM也是基于Paxos算法,系统容错不能超过n-1/2, 5台容错2台;

     

     这个架构只能手动决定哪个是active哪个是standby;必须只能有一个active!!如果出现两个NameNode,即两个都是active,它可能还不报错,那么可能会导致整个集群的数据都是错的,问题很严重!两个AA的情况叫脑裂(split brain缩写sb)。

    standby要想变成active,要确保active那个,需要安全可靠的zookeeper(文件系统+通知机制)第三方来联系两方  ---> 实现故障的自动转移;

     由Zkfc来联系zookeeper,并不是namenode直接联系,(zookeeper客户端);HA是hadoop2.0才有的,namenode在1.0时就有了;没有把zkfc写进namenode是为了保持NameNode的健壮性,没有zkfc之前就已经运行的很好了(鲁棒性);NameNode和Zkfc虽然是两个进程但它们是绑定到一起的

    两个zkfc怎么决定谁初始化就是active呢,谁快谁就是active; 是active状态它会在zookeeper中有一个临时节点,zkfc会尝试看看zookeeper中有没有这个临时节点,如果没有我就变成这个临时节点,成为active,慢的一看有了,就变成standby;

    NameNode发生假死,zkfc就会把zookeeper中的临时节点删除,去通知另外一个namenode的zkfc,让它去成为active,这个namenode就会去强行杀死假死的namenode,防止脑裂!如果杀不死就自定义一个脚本强制它关机,成功之后才会变成active。

    现在合并fsimage是由standby来完成的,没有secondaryNameNode;

     

    在module目录下创建一个ha文件夹
    mkdir ha
    
    将/opt/module/下的 hadoop-2.7.2拷贝到/opt/module/ha目录下
    cp -r hadoop-2.7.2/ /opt/module/ha/
    删除data logs等文件

     配置core-site.xml

    <configuration>
        <!-- 把两个NameNode)的地址组装成一个集群mycluster -->
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://mycluster</value>
        </property>
    
        <!-- 指定hadoop运行时产生文件的存储目录 -->
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/module/ha/hadoop-2.7.2/data/tmp</value>
        </property>
    
        <property>
            <name>ha.zookeeper.quorum</name>
            <value>hadoop101:2181,hadoop102:2181,hadoop103:2181</value>
        </property>
    
    </configuration>

    配置hdfs-site.xml

    <configuration>
        <!-- 完全分布式集群名称 -->
        <property>
            <name>dfs.nameservices</name>
            <value>mycluster</value>
        </property>
    
        <!-- 集群中NameNode节点都有哪些 -->
        <property>
            <name>dfs.ha.namenodes.mycluster</name>
            <value>nn1,nn2</value>
        </property>
    
        <!-- nn1的RPC通信地址 -->
        <property>
            <name>dfs.namenode.rpc-address.mycluster.nn1</name>
            <value>hadoop101:9000</value>
        </property>
    
        <!-- nn2的RPC通信地址 -->
        <property>
            <name>dfs.namenode.rpc-address.mycluster.nn2</name>
            <value>hadoop102:9000</value>
        </property>
    
        <!-- nn1的http通信地址 -->
        <property>
            <name>dfs.namenode.http-address.mycluster.nn1</name>
            <value>hadoop101:50070</value>
        </property>
    
        <!-- nn2的http通信地址 -->
        <property>
            <name>dfs.namenode.http-address.mycluster.nn2</name>
            <value>hadoop102:50070</value>
        </property>
    
        <!-- 指定NameNode元数据在JournalNode上的存放位置 -->
        <property>
            <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://hadoop101:8485;hadoop102:8485;hadoop103:8485/mycluster</value>
        </property>
    
        <!-- 配置隔离机制,即同一时刻只能有一台服务器对外响应 -->
        <property>
            <name>dfs.ha.fencing.methods</name>
            <value>sshfence</value>
        </property>
    
        <!-- 使用隔离机制时需要ssh无秘钥登录-->
        <property>
            <name>dfs.ha.fencing.ssh.private-key-files</name>
            <value>/home/kris/.ssh/id_rsa</value>
        </property>
    
        <!-- 声明journalnode服务器存储目录-->
        <property>
            <name>dfs.journalnode.edits.dir</name>
            <value>/opt/module/ha/hadoop-2.7.2/data/jn</value>
        </property>
    
        <!-- 关闭权限检查-->
        <property>
            <name>dfs.permissions.enable</name>
            <value>false</value>
        </property>
    
        <!-- 访问代理类:client,mycluster,active配置失败自动切换实现方式-->
        <property>
              <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
        </property>
    
        <property>
            <name>dfs.ha.automatic-failover.enabled</name>
            <value>true</value>
        </property>
    
    </configuration>
    发送到其他机器
    xsync ha

    启动HDFS-HA集群
    1. 在各个JournalNode节点上,输入以下命令启动journalnode服务
        sbin/hadoop-daemons.sh start journalnode //加个s就可以3台一块启动;都启动之后才能格式化namenode;只能格式化一次!
    2. 在[nn1]上,对其进行格式化,并启动
        bin/hdfs namenode -format 
    19/02/13 02:15:00 INFO util.GSet: Computing capacity for map NameNodeRetryCache
    19/02/13 02:15:00 INFO util.GSet: VM type = 64-bit
    19/02/13 02:15:00 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
    19/02/13 02:15:00 INFO util.GSet: capacity = 2^15 = 32768 entries
    19/02/13 02:15:01 INFO namenode.FSImage: Allocated new BlockPoolId: BP-26035536-192.168.1.101-1549995301800
    19/02/13 02:15:01 INFO common.Storage: Storage directory /opt/module/ha/hadoop-2.7.2/data/tmp/dfs/name has been successfully formatted.
    19/02/13 02:15:02 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
    19/02/13 02:15:02 INFO util.ExitUtil: Exiting with status 0
    19/02/13 02:15:02 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop101/192.168.1.101
    ************************************************************/
      启动namenode:  sbin/hadoop-daemon.sh start namenode
    
    3.    在[nn2]上,同步nn1的元数据信息
        bin/hdfs namenode -bootstrapStandby
    ......
    STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2017-05-22T10:49Z
    STARTUP_MSG: java = 1.8.0_144
    ************************************************************/
    19/02/21 17:56:25 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
    19/02/21 17:56:25 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
    =====================================================
    About to bootstrap Standby ID nn2 from:
    Nameservice ID: mycluster
    Other Namenode ID: nn1
    Other NN's HTTP address: http://hadoop101:50070
    Other NN's IPC address: hadoop101/192.168.1.101:9000
    Namespace ID: 411281390
    Block pool ID: BP-1462258257-192.168.1.101-1550740170734
    Cluster ID: CID-d20dda0d-49d1-48f4-b9e8-2c99b72a15c2
    Layout version: -63
    isUpgradeFinalized: true
    =====================================================
    19/02/13 02:16:51 INFO common.Storage: Storage directory /opt/module/ha/hadoop-2.7.2/data/tmp/dfs/name has been successfully formatted.
    19/02/13 02:16:51 INFO namenode.TransferFsImage: Opening connection to http://hadoop101:50070/imagetransfer?getimage=1&txid=0&storageInfo=-63:1640720426:0:CID-81cbaa0d-6a6f-4932-98ba-ff2a46d87514
    19/02/13 02:16:51 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
    19/02/13 02:16:52 INFO namenode.TransferFsImage: Transfer took 0.02s at 0.00 KB/s
    19/02/13 02:16:52 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes.
    19/02/13 02:16:52 INFO util.ExitUtil: Exiting with status 0
    19/02/13 02:16:52 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop102/192.168.1.102
    ************************************************************/
    出现这个信息是:(只需格式化一次;)
      Re-format filesystem in Storage Directory /opt/module/ha/hadoop-2.7.2/data/tmp/dfs/name ? (Y or N) N
    Format aborted in Storage Directory /opt/module/ha/hadoop-2.7.2/data/tmp/dfs/name
    19/02/21 19:06:50 INFO util.ExitUtil: Exiting with status 5 ##这个是退出状态!
    19/02/21 19:06:50 INFO namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at hadoop102/192.168.1.102
    ************************************************************/
    4.    启动[nn2]
        sbin/hadoop-daemon.sh start namenode
    6.    在[nn1]上,启动所有datanode; hadoop-daemons.sh是3台都启动datanode
        sbin/hadoop-daemons.sh start datanode
    ################手动切换namenode
    7. 将[nn1]切换为Active bin/hdfs haadmin -transitionToActive nn1 5. 查看是否Active bin/hdfs haadmin -getServiceState nn1 http://hadoop101:50070/dfshealth.html#tab-overview http://hadoop102:50070/dfshealth.html#tab-overview
    
    
    
     

    配置HDFS-HA自动故障转移(直接启动不需先按手动的启动方式

    配置好自动故障转移后手动模式就不可用了

    Automatic failover is enabled for NameNode at hadoop102/192.168.1.102:9000
    Refusing to manually manage HA state, since it may cause a split-brain scenario or other incorrect state.
    If you are very sure you know what you are doing, please 
    specify the --forcemanual flag.
    配置HDFS-HA自动故障转移
    1.    具体配置 ;在上个基础上添加如下:
        (1)在hdfs-site.xml中增加
    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
    </property>2)在core-site.xml文件中增加
    <property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop101:2181,hadoop102:2181,hadoop103:2181</value>
    </property>
    2.    启动
        (1)关闭所有HDFS服务:
    sbin/stop-dfs.sh2)启动Zookeeper集群:
    bin/zkServer.sh start
        (3)初始化HA在Zookeeper中状态:(只初始化一次)
    bin/hdfs zkfc -formatZK
    
    19/02/13 02:04:02 INFO zookeeper.ClientCnxn: Socket connection established to hadoop102/192.168.1.102:2181, initiating session
    19/02/13 02:04:02 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop102/192.168.1.102:2181, sessionid = 0x268e2dee6e40000, negotiated timeout = 5000
    19/02/13 02:04:02 INFO ha.ActiveStandbyElector: Session connected.
    19/02/13 02:04:02 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
    19/02/13 02:04:02 INFO zookeeper.ZooKeeper: Session: 0x268e2dee6e40000 closed
    19/02/13 02:04:02 INFO zookeeper.ClientCnxn: EventThread shut down
    
        (4)启动HDFS服务:
    sbin/start-dfs.sh
    
    hadoop101: starting namenode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-namenode-hadoop101.out
    hadoop102: starting namenode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-namenode-hadoop102.out
    hadoop102: starting datanode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-datanode-hadoop102.out
    hadoop101: starting datanode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-datanode-hadoop101.out
    hadoop103: starting datanode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-datanode-hadoop103.out
    Starting journal nodes [hadoop101 hadoop102 hadoop103]
    hadoop103: starting journalnode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-journalnode-hadoop103.out
    hadoop102: starting journalnode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-journalnode-hadoop102.out
    hadoop101: starting journalnode, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-journalnode-hadoop101.out
    Starting ZK Failover Controllers on NN hosts [hadoop101 hadoop102]
    hadoop101: starting zkfc, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-zkfc-hadoop101.out
    hadoop102: starting zkfc, logging to /opt/module/ha/hadoop-2.7.2/logs/hadoop-kris-zkfc-hadoop102.out
    
    3.    验证
        (1)将Active NameNode进程kill
        kill -9 namenode的进程id   ---> 另外一个namenode上位;( 杀掉之后,可单独启动: sbin/hadoop-daemon.sh start namenode,启动之后它不会变成之前的active;而是standby)
         (2) 将DFSZKFailoverController的进程kill
          kill -9 8819   --->尽管它的namenode没有挂,但另外一个namenode也会上位,它变成standby
          再启动它要先停止->sbin/stop-dfs.sh  sbin/start-dfs.sh3)将Active NameNode机器断开网络
        sudo service network stop  --->  把网络断开之后,配的隔离机制是sshfence尝试远程登录区杀敌hadoop101,直到把101网络连接上才杀死,102才成为active;

     

    断网上位很容易脑裂,那边网已恢复就炸了;如果真的断网,它这个是不会自动切的是为了防脑裂。虽然可以通过配置断网可以直接上,但很危险;断网上不去反而安全。

     

    [zk: localhost:2181(CONNECTED) 0] ls /
    [zookeeper, hadoop-ha]
    [zk: localhost:2181(CONNECTED) 1] ls /
    [zookeeper, hadoop-ha]
    [zk: localhost:2181(CONNECTED) 2] ls /hadoop-ha
    [mycluster]
    [zk: localhost:2181(CONNECTED) 3] ls /hadoop-ha/mycluster
    [ActiveBreadCrumb, ActiveStandbyElectorLock] 选举的关键节点,谁占领了这个节点谁就是active; 
    [zk: localhost:2181(CONNECTED) 4] get /hadoop-ha/mycluster/ActiveStandbyElectorLock
    
            myclusternn1 hadoop101 �F(�>
    cZxid = 0x10000000f
    ctime = Wed Feb 13 02:16:22 CST 2019
    mZxid = 0x10000000f
    mtime = Wed Feb 13 02:16:22 CST 2019
    pZxid = 0x10000000f
    cversion = 0
    dataVersion = 0
    aclVersion = 0
    ephemeralOwner = 0x268e2dee6e40002  临时节点,
    dataLength = 33
    numChildren = 0

    进程

    hadoop101                hadoop102                hadoop103
     NameNode                NameNode
     JournalNode            JournalNode                JournalNode
     DataNode                DataNode                DataNode
     DFSZKFailoverController DFSZKFailoverController ZooKeeperMain(bin/zkCli.sh ,启动zookeeperd客户端)
     ResourceManager         ResourceManager
     NodeManager             NodeManager            NodeManager    
    
    QuorumPeerMain            QuorumPeerMain            QuorumPeerMain (bin/zkServer.sh start 启动zookeeper服务器)
    
    DFSZKFailoverController是Hadoop-2.7.0中HDFS NameNode HA实现的中心组件,它负责整体的故障转移控制等。
    它是一个守护进程,通过main()方法启动,继承自ZKFailoverController。  zkfc
     使用JournalNode实现两个NameNode(Active和Standby)之间数据的共享
     

    YARN-HA配置

    YARN-HA工作机制

     Yarn原生根zookeeper兼容很好,配置比较简单;

     配置YARN-HA集群

      yarn-site.xml

        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>  //混合服务还是shuffle,用shuffle确定reducer获取数据的方式
        </property>
    
        <!--启用resourcemanager ha-->
        <property>
            <name>yarn.resourcemanager.ha.enabled</name>
            <value>true</value>
        </property>
     
        <!--声明两台resourcemanager的地址-->
        <property>
            <name>yarn.resourcemanager.cluster-id</name>
            <value>cluster-yarn1</value>
        </property>
    
        <property>
            <name>yarn.resourcemanager.ha.rm-ids</name>
            <value>rm1,rm2</value>
        </property>
    
        <property>
            <name>yarn.resourcemanager.hostname.rm1</name>
            <value>hadoop101</value>
        </property>
    
        <property>
            <name>yarn.resourcemanager.hostname.rm2</name>
            <value>hadoop102</value>
        </property>
     
        <!--指定zookeeper集群的地址--> 
        <property>
            <name>yarn.resourcemanager.zk-address</name>
            <value>hadoop101:2181,hadoop102:2181,hadoop103:2181</value>
        </property>
    
        <!--启用自动恢复--> 
        <property>
            <name>yarn.resourcemanager.recovery.enabled</name>
            <value>true</value>
        </property>
     
        <!--指定resourcemanager的状态信息存储在zookeeper集群--> 
        <property>
            <name>yarn.resourcemanager.store.class</name>     
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
        </property>

    yarn-site.xml 在hadoop101中配置好将其分发到其他机器:xsync etc/ 

    启动zookeeper;3台其他都启动;
      [kris@hadoop101 ~]$ /opt/module/zookeeper-3.4.10/bin/zkServer.sh start
    启动hdfs 
      sbin/start-dfs.sh
    
    启动YARN 
    (1)在hadoop101中执行:
      sbin/start-yarn.sh
    [kris@hadoop101 hadoop-2.7.2]$ sbin/start-yarn.sh 
    starting yarn daemons
    starting resourcemanager, logging to /opt/module/ha/hadoop-2.7.2/logs/yarn-kris-resourcemanager-hadoop101.out
    hadoop101: starting nodemanager, logging to /opt/module/ha/hadoop-2.7.2/logs/yarn-kris-nodemanager-hadoop101.out
    hadoop103: starting nodemanager, logging to /opt/module/ha/hadoop-2.7.2/logs/yarn-kris-nodemanager-hadoop103.out
    hadoop102: starting nodemanager, logging to /opt/module/ha/hadoop-2.7.2/logs/yarn-kris-nodemanager-hadoop102.out
    
    
    (2)在hadoop102中执行:
      sbin/yarn-daemon.sh start resourcemanager //脚本只在上面一台上启动了,hadoop102上的resourcemanager要手动启;
    [kris@hadoop102 hadoop-2.7.2]$ sbin/yarn-daemon.sh start resourcemanager
    starting resourcemanager, logging to /opt/module/ha/hadoop-2.7.2/logs/yarn-kris-resourcemanager-hadoop102.out
    
    
    (3)查看服务状态
      bin/yarn rmadmin -getServiceState rm1  //rm1就是hadoop101;  rm2为hadoop102
    [kris@hadoop102 hadoop-2.7.2]$ bin/yarn rmadmin -getServiceState rm1
    active
    [kris@hadoop102 hadoop-2.7.2]$ bin/yarn rmadmin -getServiceState rm2
    standby
    
    ################
    hdfs-ha & yarn-ha
    [kris@hadoop101 hadoop-2.7.2]$ jpsall
    -------hadoop101-------
    14288 ResourceManager
    14402 NodeManager
    13396 NameNode
    14791 Jps
    13114 QuorumPeerMain
    13946 DFSZKFailoverController
    13516 DataNode
    13743 JournalNode
    -------hadoop102-------
    9936 NameNode
    10352 ResourceManager
    9569 JournalNode
    9698 DFSZKFailoverController
    9462 DataNode
    9270 QuorumPeerMain
    10600 Jps
    10202 NodeManager
    -------hadoop103-------
    9073 Jps
    8697 JournalNode
    8522 QuorumPeerMain
    8909 NodeManager
    8590 DataNode

      http://hadoop101:8088/cluster     http://hadoop102:8088/cluster -->它会重定向到hadoop101

      

  • 相关阅读:
    《linux内核设计与实现》读书笔记第十八章
    20135337朱荟潼 Linux第八周学习总结——进程的切换和系统的一般执行过程
    2013337朱荟潼 Linux第四章读书笔记——进程调度
    2013337朱荟潼 Linux&深入理解计算机系统第七章读书笔记——链接
    20135337朱荟潼 Linux第七周学习总结——可执行程序的装载
    2013337朱荟潼 Linux第三章读书笔记——进程管理
    20135337朱荟潼 Linux第六周学习总结——进程的描述和进程的创建
    2013337朱荟潼 Linux第十八章读书笔记——调试
    20135337朱荟潼 Linux第五周学习总结——扒开系统调用的三层皮(下)
    2013337朱荟潼 Linux第五章读书笔记——系统调用
  • 原文地址:https://www.cnblogs.com/shengyang17/p/10367519.html
Copyright © 2011-2022 走看看