zoukankan      html  css  js  c++  java
  • hadoop高可用HA的配置

    zk3
    zk4
    zk5

    配置hadoop的HA大概可以分为以下几步:

    • 配置zookpeer(namenode之间的通信要靠zk来实现)
    • 配置hadoop的
      • hadoop-env.sh
      • hdfs-site.xml
      • core-site.xml
      • slave
    * hadoop-env.sh
    export JAVA_HOME=/opt/module/jdk1.8.0_144 
    * hdfs-site.xml
    <configuration>
        <!-- 指定数据冗余份数 -->
        <property>
            <name>dfs.replication</name>
            <value>3</value>
        </property>
    
        <!-- 完全分布式集群名称 -->
        <property>
            <name>dfs.nameservices</name>
            <value>mycluster</value>
        </property>
    
        <!-- 集群中NameNode节点都有哪些 -->
        <property>
            <name>dfs.ha.namenodes.mycluster</name>
            <value>nn1,nn2</value>
        </property>
    
        <!-- nn1的RPC通信地址 -->
        <property>
            <name>dfs.namenode.rpc-address.mycluster.nn1</name>
            <value>z04:8020</value>
        </property>
    
        <!-- nn2的RPC通信地址 -->
        <property>
            <name>dfs.namenode.rpc-address.mycluster.nn2</name>
            <value>z05:8020</value>
        </property>
    
        <!-- nn1的http通信地址 -->
        <property>
            <name>dfs.namenode.http-address.mycluster.nn1</name>
            <value>z04:50070</value>
        </property>
    
        <!-- nn2的http通信地址 -->
        <property>
            <name>dfs.namenode.http-address.mycluster.nn2</name>
            <value>z05:50070</value>
        </property>
    
        <!-- 指定NameNode元数据在JournalNode上的存放位置 -->
        <property>
            <name>dfs.namenode.shared.edits.dir</name>
            <value>qjournal://z04:8485;z05:8485;z06:8485/mycluster</value>
        </property>
    
        <!-- 配置隔离机制,即同一时刻只能有一台服务器对外响应 -->
        <property>
            <name>dfs.ha.fencing.methods</name>
            <value>sshfence</value>
        </property>
    
        <!-- 使用隔离机制时需要ssh无秘钥登录-->
        <property>
            <name>dfs.ha.fencing.ssh.private-key-files</name>
            <value>/home/z/.ssh/id_rsa</value>
        </property>
    
        <!-- 声明journalnode服务器存储目录-->
        <property>
            <name>dfs.journalnode.edits.dir</name>
            <value>/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/data/jn</value>
        </property>
    
        <!-- 关闭权限检查-->
        <property>
            <name>dfs.permissions.enable</name>
            <value>false</value>
        </property>
    
        <!-- 访问代理类:client,mycluster,active配置失败自动切换实现方式-->
        <property>
              <name>dfs.client.failover.proxy.provider.mycluster</name>
              <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
        </property>
                
    </configuration>
    
    * core-site.xml
    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://mycluster</value>
        </property>
    
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6/data</value>
        </property>
    </configuration>
    * slaves
    zk04
    zk05
    zk06


  • 相关阅读:
    修复 Visual Studio Error “No exports were found that match the constraint”
    RabbitMQ Config
    Entity Framework Extended Library
    Navisworks API 简单二次开发 (自定义工具条)
    NavisWorks Api 简单使用与Gantt
    SQL SERVER 竖表变成横表
    SQL SERVER 多数据导入
    Devexpress GridControl.Export
    mongo DB for C#
    Devexress XPO xpPageSelector 使用
  • 原文地址:https://www.cnblogs.com/alexzhang92/p/10873152.html
Copyright © 2011-2022 走看看