zoukankan      html  css  js  c++  java
  • YARN的HA搭建

    YARN的HA搭建

    选择192.168.230.10(node1),192.168.230.14(node5)两台主机作为ResourceManager
    datanode与nodemanager一定要放在一个机器上
    

    YARN的体系架构

    1、配置/opt/modules/hadoop-2.5.1/etc/hadoop/mapred-site.xml:

    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>
    

    2、配置/opt/modules/hadoop-2.5.1/etc/hadoop/yarn-site.xml:

    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>
    

     <!-- 启用 -->
     <property>
       <name>yarn.resourcemanager.ha.enabled</name>
       <value>true</value>
     </property>
     <!-- 集群ID号 -->
     <property>
       <name>yarn.resourcemanager.cluster-id</name>
       <value>cluster1</value>
     </property>
     <property>
       <name>yarn.resourcemanager.ha.rm-ids</name>
       <value>rm1,rm2</value>
     </property>
     <property>
       <name>yarn.resourcemanager.hostname.rm1</name>
       <value>node1</value>
     </property>
     <property>
       <name>yarn.resourcemanager.hostname.rm2</name>
       <value>node5</value>
     </property>
     <property>
       <name>yarn.resourcemanager.zk-address</name>
       <value>node2:2181,node3:2181,node5:2181</value>
     </property>
    

    3、将在node1主机上配置好的yarn-site.xml,mapred-site.xml文件发送到node2、node3、node5主机上

    scp /opt/modules/hadoop-2.5.1/etc/hadoop/yarn-site.xml root@node2:/opt/modules/hadoop-2.5.1/etc/hadoop/
    scp /opt/modules/hadoop-2.5.1/etc/hadoop/mapred-site.xml root@node2:/opt/modules/hadoop-2.5.1/etc/hadoop/
    
    scp /opt/modules/hadoop-2.5.1/etc/hadoop/yarn-site.xml root@node3:/opt/modules/hadoop-2.5.1/etc/hadoop/
    scp /opt/modules/hadoop-2.5.1/etc/hadoop/mapred-site.xml root@node3:/opt/modules/hadoop-2.5.1/etc/hadoop/
    
    scp /opt/modules/hadoop-2.5.1/etc/hadoop/yarn-site.xml root@node5:/opt/modules/hadoop-2.5.1/etc/hadoop/
    scp /opt/modules/hadoop-2.5.1/etc/hadoop/mapred-site.xml root@node5:/opt/modules/hadoop-2.5.1/etc/hadoop/
    

    4、在node1主机上启动yarn

    [root@node1 hadoop-2.5.1]# ./sbin/start-yarn.sh
    

    [root@node1 hadoop-2.5.1]# jps
    

    5、启动单节点ResourceManager

    [root@node5 hadoop-2.5.1]# ./sbin/yarn-daemon.sh start resourcemanager
    

    [root@node5 hadoop-2.5.1]# jps
    

    6、在浏览器地址栏上访问

    http://node1:8088/cluster/nodes
    

    http://node5:8088/cluster/nodes,页面会直接跳转到node1的页面
    

    6.1、杀死node1主机的resourcemanager进程,http://node1:8088/cluster访问不成功

    [root@node1 hadoop-2.5.1]# kill -9 8145
    

    6.2、查看http://node5:8088/cluster发现node5节点的ResourceManager成为Active

    6.3、在node1主机重新单节点启动ResourceManager,发现node1的ResourceManager变成了Standby

    [root@node1 hadoop-2.5.1]# ./sbin/yarn-daemon.sh start resourcemanager
  • 相关阅读:
    Error 1031
    电信短信转移取消
    手机模拟手柄操作电脑游戏
    MySQL 5.7 InnoDB Cluster 部署
    Bitwarden_rs搭建
    脚本:Tomcat日志切割
    报错:The server cannot be started because one or more of the ports are invalid. Open the server editor and correct the invalid ports.
    Linux启动过程详解
    UNIX环境高级编程__针对apue.h找不到的情况以及log错误信息输出的问题
    C++primer_拷贝控制之13.26联系控制成员实现类值行为
  • 原文地址:https://www.cnblogs.com/SparseMatrix/p/5255514.html
Copyright © 2011-2022 走看看