zoukankan      html  css  js  c++  java
  • hadoop环境搭建笔记

    一、配置Linux

    (1)cat /etc/networks

    (2)cat /etc/sysconfig/network

    (3)vi /etc/udev/rules.d/70-persistent-net.rules

    eth1 改为eth0 

    (4)vi /etc/sysconfig/network-scripts/ifcfg-eth0

    HWADDR改为eth1的地址

    HWADDR=00:0c:29:79:e6:ec

    DNS1=网关地址

    (5)vi /etc/sysconfig/selinux

    #SELINUX=disabled

    (6)加sudo权限

    vi /etc/sudoers

    beifeng ALL=(root)NOPASSWD:ALL

    (7)关闭防火墙

    centos 7:
    systemctl stop firewalld.service #停止
    systemctl disable firewalld.service #禁用
    之前的版本:
    service iptables stop #
    停止
    chkconfig iptables off #禁用

    chkconfig iptables off

    service iptables status

    二、安装jdk、maven、hadoop

    chown beifeng:beifeng /opt/*

    无密钥登陆ssh
    ssh-keygen -t rsa
    ssh-copy-id hostname02
    ssh-copy-id hostname03
    ssh-copy-id hostname04 

    三、配置伪分布式

    1. 修改hadoop-evn.sh

    export JAVA_HOME=/opt/modules/jdk1.7.0_67

    2. 修改yarn-env.sh

    export JAVA_HOME=/opt/modules/jdk1.7.0_67

    3. 修改mapred-env.sh

    export JAVA_HOME=/opt/modules/jdk1.7.0_67

    4. 修改core-site.xml

    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
         <property>
             <name>hadoop.tmp.dir</name>
             <value>/opt/modules/hadoop-2.5.0/data/tmp</value>
         </property>
         <property>
              <name>hadoop.http.staticuser.user</name>
              <value>beifeng</value>
         </property>
    </configuration>

    5. 修改hdfs-site.xml

    <configuration>
            <property>
                    <name>dfs.replication</name>
                    <value>1</value>
            </property>
    
            <property>
                    <name>dfs.namenode.secondary.http-address</name>
                    <value>beifeng-hadoop-02:50090</value>
            </property>
    
            <property>
                    <name>dfs.permissions.enabled</name>
                    <value>false</value>
            </property>
    </configuration>

    6. 修改slaves

    beifeng-hadoop-02

    7. 修改yarn-site.xml

    <configuration>
    
    <!-- Site specific YARN configuration properties -->
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    
            <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>beifeng-hadoop-02</value>
            </property>
    
            <!-- 是否启用日志聚集功能 -->
            <property>
                    <name>yarn.log-aggregation-enable</name>
                    <value>true</value>
            </property>
    
            <!-- 日志保留时间(单位为秒) -->
            <property>
                    <name>yarn.log-aggregation.retain-seconds</name>
                    <value>106800</value>
            </property>
    </configuration>

    8. 修改mapred-site.xml

    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>

    9. 启动服务

    bin/hdfs namenode -format

    sbin/hadoop-daemon.sh start namenode sbin/hadoop-daemon.sh start datanode sbin/yarn-daemon.sh start resourcemanager sbin/yarn-daemon.sh start nodemanager sbin/mr-jobhistory-daemon.sh start historyserver sbin/hadoop-daemon.sh start secondarynamenode

    sbin
    /hadoop-daemon.sh stop namenode sbin/hadoop-daemon.sh stop datanode sbin/yarn-daemon.sh stop resourcemanager sbin/yarn-daemon.sh stop nodemanager sbin/mr-jobhistory-daemon.sh stop historyserver
    sbin/hadoop-daemon.sh stop secondarynamenode

    10. 服务可视化

    hdfs可视化界面: http://beifeng-hadoop-01:50070/dfshealth.html#tab-overview
    yarn可视化界面: http://beifeng-hadoop-01:8088/cluster
    历史服务器可视化界面:http://beifeng-hadoop-01:19888/
    secondarynamenode可视化界面 http://beifeng-hadoop-01:50090/status.html
  • 相关阅读:
    测试篇 尝了一下net5.0桌面开发
    日志篇 vs的文本替换,剔除引号保留数字 将vs2019更新之后无法用ctrl+d
    测试篇 使用 nuget.exe CLI 创建 nuget 包
    日志篇 博客园的下方的女孩透明的,带点击声音的
    测试篇 c#文件类型关联启动程序
    数学篇 求两条直线的交点,说明过程.
    日志篇 原生git笔记
    测试篇 winform Anchor 怎么临时取消关联,窗口边界和控件关联
    cad.net dll动态加载,插件式架构,在dll查找引用了的dll,查找dll依赖,dll热插拔,加载dll运行出错.
    cad.net 图元反应器,图元事件,要加在提交数据库之后哟
  • 原文地址:https://www.cnblogs.com/yeahwell/p/5660434.html
Copyright © 2011-2022 走看看