Hadoop

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

环境准备

机器: Centos7.6 1810

hadoop:  3.3.0 版本

jdk: 1.8.0_144 版本

域名解析配置

vim /etc/hosts

192.168.220.55 hadoop001

文件解压目录

[root@hadoop001 module]# ls
hadoop  java
[root@hadoop001 module]# pwd
/opt/module

环境变量配置

cat >> /etc/profile <<EOF

#JAVA_HOME
export JAVA_HOME=/opt/module/java
export PATH=$PATH:$JAVA_HOME/bin

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

EOF

source /etc/profile

环境变量查看

[root@hadoop001 bin]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

[root@hadoop001 bin]# hadoop version
Hadoop 3.3.0
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af
Compiled by brahma on 2020-07-06T18:44Z
Compiled with protoc 3.7.1
From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4
This command was run using /opt/module/hadoop/share/hadoop/common/hadoop-common-3.3.0.jar

配置集群

配置 hadoop-env.sh

cd /opt/module/hadoop/etc/hadoop/

修改JAVA_HOME 路径
vim hadoop-env.sh
export JAVA_HOME=/opt/module/java

配置 core-site.xml

vim core-site.xml

    <!-- 指定HDFS中NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop001:9000</value>
    </property>

    <!-- 指定Hadoop运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop/data/tmp</value>
    </property>

启动集群

　　(a) 格式化NameNode（第一次启动时格式化，以后就不要总格式化）

hdfs namenode -format

　　(b)  启动NameNode

hdfs --daemon start namenode

　　(c) 启动DataNode

hdfs --daemon start datanode

查看集群

　　(a) 查看是否启动成功

[root@hadoop001 hadoop]# jps
32736 Jps
32705 DataNode
32585 NameNode

　　

　　(b) web端查看HDFS文件系统

http://hadoop001:9870/dfshealth.html#tab-overview

注意:  hadoop2.x 版本 UI端口 50070

 

配置YARN

vim core-site.xml

    <!-- 指定HDFS中NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop001:9000</value>
    </property>

    <!-- 指定Hadoop运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop/data/tmp</value>
    </property>

启动YARN 

启动RM

yarn --daemon start resourcemanager

启动NM

yarn --daemon start nodemanager

查看集群

查看服务状态

[root@hadoop001 hadoop]# jps
33729 ResourceManager
33141 DataNode
34152 Jps
32585 NameNode
34012 NodeManager

 web端查看RM UI