3、hadoop的分布式安装

1.基本信息

版本	2.7.3
安装机器	三台机器
账号	hadoop
源路径	/opt/software/hadoop-2.7.3.tar.gz
目标路径	/opt/hadoop -> /opt/hadoop-2.7.3
依赖关系	无

2.安装过程

1).切换到hadoop账户，通过tar -zxvf命令将hadoop解压缩至目的安装目录：

[root@bgs-5p173-wangwenting opt]# su hadoop

[hadoop@bgs-5p173-wangwenting opt]$ cd /opt/software

[hadoop@bgs-5p173-wangwenting software]$  tar -zxvf hadoop-${version}.tar.gz  -C /opt

[hadoop@bgs-5p173-wangwenting software]$ cd /opt

[hadoop@bgs-5p173-wangwenting opt]$ ln -s /opt/hadoop-${version} /opt/hadoop

2).创建tmpdir目录：

[hadoop@bgs-5p173-wangwenting opt]$ cd  /opt/hadoop

[hadoop@bgs-5p173-wangwenting hadoop]$ mkdir -p tmpdir

3).配置hadoop-env.sh文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$ mkdir -p /opt/hadoop/pids

[hadoop@bgs-5p173-wangwenting hadoop]$ vi hadoop-env.sh

在hadoop-env.sh文件中添加如下配置：

export JAVA_HOME=/opt/java

export HADOOP_PID_DIR=/opt/hadoop/pids

4.配置mapred-env.sh文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$ vim mapred-env.sh

在mapred-env.sh文件中添加如下配置：

export JAVA_HOME=/opt/java

5.配置core-site.xml文件

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$  vim core-site.xml

在core-site.xml文件中添加如下配置：

<configuration>

<property>

//namenode的临时工作目录

        <name>hadoop.tmp.dir</name>

        <value>/opt/hadoop/tmpdir</value>

    </property>

<property>

//hdfs的入口，告诉namenode在那个机器上，端口号是什么。

        <name>fs.defaultFS</name>

        <value>hdfs://bgs-5p173-wangwenting:9000</value>

    </property>

    <property>

        <name>io.file.buffer.size</name>

        <value>131072</value>

    </property>

    <property>

        <name>fs.trash.interval</name>

        <value>1440</value>

    </property>

</configuration>

6.配置hdfs-site.xml文件

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$ vim hdfs-site.xml

在hdfs-site.xml文件中添加如下配置：

<configuration>

<property>

//副本数量，一般是小于等于datanode的数量，

        <name>dfs.replication</name>

        <value>2</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/opt/hadoop/tmpdir/dfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.name.dir</name>

        <value>file:/opt/hadoop/tmpdir/dfs/data</value>

    </property>

    <property> 

        <name>dfs.webhdfs.enabled</name> 

        <value>true</value> 

</property>

<property>

        <name>dfs.secondary.http.address</name>

        <value>bgs-5p173-wangwenting:50090</value>

 </property>

</configuration>

7.配置mapred-site.xml文件

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$ vi mapred-site.xml

在mapred-site.xml文件中添加如下配置：

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>bgs-5p173-wangwenting:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>bgs-5p173-wangwenting:19888</value>

    </property>

</configuration>

8.配置yarn-site.xml文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop/

[hadoop@bgs-5p173-wangwenting hadoop]$ vim yarn-site.xml

在yarn-site.xml文件中添加如下配置：

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>bgs-5p173-wangwenting:8030</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>bgs-5p173-wangwenting:8031</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>bgs-5p173-wangwenting:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>bgs-5p173-wangwenting:8033</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>bgs-5p173-wangwenting:8088</value>

    </property>

<!-- Site specific YARN configuration properties -->

</configuration>

9.配置hadoop运行的环境变量

[hadoop@bgs-5p173-wangwenting hadoop]$ vim /etc/profile

export HADOOP_HOME=/opt/hadoop

export PATH=$HADOOP_HOME/bin:$PATH

配置成功后，执行source  /etc/profile使配置生效

[hadoop@bgs-5p173-wangwenting hadoop]$ source /etc/profile

10.修改slaves文件:

[hadoop@bgs-5p173-wangwenting hadoop]$ cd /opt/hadoop/etc/hadoop

[hadoop@bgs-5p173-wangwenting hadoop]$ vim slaves

在slaves文件中添加

//datanode的节点的位置

bgs-5p173-wangwenting

bgs-5p174-wangwenting

bgs-5p175-wangwenting

11.在bgs-5p173-wangwenting上复制hadoop-2.7.3到hadoop@bgs-5p174-wangwenting和hadoop@bgs-5p174-wangwenting机器并按照步骤9修改环境变量并执行以下操作:

[hadoop@bgs-5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop@bgs-5p174-wangwenting:/opt/

[hadoop@bgs-5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop

[hadoop@bgs-5p173-wangwenting hadoop]$ scp -r /opt/hadoop-${version} hadoop@bgs-5p175-wangwenting:/opt/

[hadoop@bgs-5p173-wangwenting hadoop]$ ln -s /opt/hadoop-${version} /opt/hadoop

12.格式化namenode，启动hadoop,并启动jobhistory服务:

[hadoop@bgs-5p173-wangwenting hadoop]$ hadoop namenode -format

[hadoop@bgs-5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/start-all.sh

[hadoop@bgs-5p173-wangwenting hadoop]$ ${HADOOP_HOME}/sbin/mr-jobhistory-daemon.sh start historyserver

start-all.sh包含dfs和yarn两个模块的启动，分别为start-dfs.sh 、 start-yarn.sh，所以dfs和yarn可以单独启动。

注意：如果datanode没有启动起来，看看是不是tmpdir中有之前的脏数据，删除这个目录其他两台机器也要删除。

13.检查每台机器的服务，hadoop@bgs-5p173-wangwenting、hadoop@bgs-5p174-wangwenting、hadoop@bgs-5p175-wangwenting三台机器上分别输入jps：

[hadoop@bgs-5p173-wangwenting ~]$ jps

24429 Jps

22898 ResourceManager

23229 NodeManager

24383 JobHistoryServer

22722 SecondaryNameNode

22488 NameNode

6945 DataNode

[ahdoop@bgs-5p174-wangwenting ~]$ jps

7650 DataNode

7788 NodeManager

8018 Jps

[hadoop@bgs-5p175-wangwenting ~]$ jps

28407 Jps

28038 DataNode

28178 NodeManager

如果三台机器正常输出上述内容，则表示hadoop集群的服务正常工作。

访问hadoop的服务页面：在浏览器中输入如下地址

http://bgs-5p173-wangwenting:8088

http://bgs-5p173-wangwenting:50070

http://bgs-5p173-wangwenting:19888

如能正常反应，则表明上述三类服务正常运行.