zoukankan      html  css  js  c++  java

  • Apache Hadoop集群安装（NameNode HA + YARN HA + SPARK + 机架感知）


    1、主机规划

    序号主机名IP地址角色
    1nn-1192.168.9.21NameNode、mr-jobhistory、zookeeper、JournalNode
    2nn-2192.168.9.22Secondary NameNode、JournalNode
    3dn-1192.168.9.23DataNode、JournalNode、zookeeper、ResourceManager、NodeManager
    4dn-2192.168.9.24DataNode、zookeeper、ResourceManager、NodeManager
    5dn-3192.168.9.25DataNode、NodeManager

    集群说明：
    （1）、对于集群规模小于7台和以下的， 可以不做NameNode HA。
    （2）、HA的集群， JournalNode节点要在3个以上， 建议设置成5个节点。JournalNode是轻量级服务， 为了本地性， 其中两个JournalNode和两台NameNode节点复用。其他JournalNode和分散在其他节点上。
    （3）、HA的集群，zookeeper节点要在3个以上， 建议设置成5个或者7个节点。zookeeper可以和DataNode节点复用。
    （4）、HA的集群，ResourceManager建议单独一个节点。对于较大规模的集群，且有空闲的主机资源， 可以考虑设置ResourceManager的HA。

    2、主机环境设置

    2.1 配置JDK


    卸载OpenJDK：
    1. --查看java版本
    2. [root@dtgr ~]# java -version
    3. java version "1.7.0_45"
    4. OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
    5. OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)--查看安装源

    7. [root@dtgr ~]# rpm -qa | grep java
    8. java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64-- 卸载
    9. [root@dtgr ~]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

    11. --验证是否卸载成功
    12. [root@dtgr ~]# rpm -qa | grep java
    13. [root@dtgr ~]# java -version
    14. -bash: /usr/bin/java: 没有那个文件或目录

    安装jdk：
    1. -- 下载并解压java源码包
    2. [root@dtgr java]# mkdir /usr/local/java
    3. [root@dtgr java]# mv jdk-7u79-linux-x64.tar.gz /usr/local/java
    4. [root@dtgr java]# cd /usr/local/java
    5. [root@dtgr java]# tar xvf jdk-7u79-linux-x64.tar.gz
    6. [root@dtgr java]# ls
    7. jdk1.7.0_79  jdk-7u79-linux-x64.tar.gz
    8. [root@dtgr java]# 
    10. --- 添加环境变量
    11. [root@dtgr java]# vim /etc/profile
    12. [root@dtgr java]# tail /etc/profile
    13. export JAVA_HOME=/usr/local/java/jdk1.7.0_79
    14. export JRE_HOME=/usr/local/java/jdk1.7.0_79/jre
    15. export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
    16. export PATH=$JAVA_HOME/bin:$PATH
    18. -- 生效环境变量
    19. [root@dtgr ~]# source /etc/profile
    21. -- 验证
    22. [root@dtgr ~]# java -version
    23. java version "1.7.0_79"
    24. Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
    25. Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
    26. [root@dtgr ~]# javac -version
    27. javac 1.7.0_79

    2.2 修改主机名和配置主机名解析

    在所有节点按照规划修改主机名， 并将主机名加入/etc/hosts文件。
    修改主机名：
    1. [root@dn-3 ~]# cat /etc/sysconfig/network
    2. NETWORKING=yes
    3. HOSTNAME=dn-3
    4. [root@dn-3 ~]# hostname dn-3

    配置/etc/hosts， 并分发到所有节点：
    1. [root@dn-3 ~]# cat /etc/hosts
    2. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    3. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    4. 192.168.9.21 nn-1
    5. 192.168.9.22 nn-2
    6. 192.168.9.23 dn-1
    7. 192.168.9.24 dn-2
    8. 192.168.9.25 dn-3

    2.3 新建hadoop账户

    用户和组均为hadoop， 密码为hadoop， home目录为/hadoop。
    1. [root@dn-3 ~]# useradd -d /hadoop hadoop

    2.4 配置ntp时钟同步

    将nn-1主机作为时钟源）
    #vi  /etc/ntp.conf
    #server 0.centos.pool.ntp.org
    #server 1.centos.pool.ntp.org
    #server 2.centos.pool.ntp.org
    server nn-1

    配置ntp服务自启动
    #chkconfig ntpd on
    启动ntp服务
    #service ntpd start

    2.5 关闭防火墙iptables和selinux

    （1）、关闭iptables
    1. [root@dn-3 ~]# service iptables stop
    2. [root@dn-3 ~]# chkconfig iptables off
    3. [root@dn-3 ~]# chkconfig --list | grep iptables
    4. iptables       	0:关闭	1:关闭	2:关闭	3:关闭	4:关闭	5:关闭	6:关闭
    5. [root@dn-3 ~]#

    （2）、关闭selinux
    1. [root@dn-3 ~]# setenforce 0
    2. setenforce: SELinux is disabled
    3. [root@dn-3 ~]# vim /etc/sysconfig/selinux 
           SELINUX=disabled

    2.6 设置ssh无密码登陆

    （1）、在所有节点生成密钥
    所有节点， 切换到hadoop用户下，生成密钥，一路回车：
    1. [hadoop@nn-1 ~]$ ssh-keygen -t rsa

    （2）、在nn-1上面，将公钥复制到文件authorized_keys中：
    命令：$ ssh  主机名   'cat ./.ssh/id_rsa.pub' >> authorized_keys
    将上面的命令的主机名替换成实际的主机名， 在nn-1上面将所有的主机都执行一次，包括自己， 如下示例：
    1. [hadoop@nn-1 ~]$ ssh nn-1 'cat ./.ssh/id_rsa.pub' >> authorized_keys
    2. hadoop@nn-1's password:

    （3）、设置权限
    1. [hadoop@nn-1 .ssh]$ chmod 644 authorized_keys

    （4）、将authorized_keys分发到所有节点: $HOME/.ssh/ 。
    如下示例：
    1. [hadoop@nn-1 .ssh]$ scp authorized_keys hadoop@nn-2:/hadoop/.ssh/

    3、安装配置Hadoop


    说明： 先在nn-1上面修改配置， 配置完毕批量分发到其他节点。

    3.1 上传hadoop、zookeeper安装包

    复制安装包到/hadoop目录下。
    解压安装包： [hadoop@nn-1 ~]$ tar -xzvf hadoop2-js-0121.tar.gz

    3.2 修改hadoop-env.sh

    1. export JAVA_HOME=/usr/local/java/jdk1.7.0_79
    2. export  HADOOP_HEAPSIZE=2000
    3. export  HADOOP_NAMENODE_INIT_HEAPSIZE=10000
    4. export  HADOOP_OPTS="-server $HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
    5. export HADOOP_NAMENODE_OPTS="-Xmx15000m -Xms15000m -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger
    6. =${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"

    3.3 修改core-site.xml

    1. <configuration>
    2.     <property>
    3.         <name>fs.defaultFS</name>
    4.         <value>hdfs://dpi</value>
    5.     </property>
    6.     <property>
    7.         <name>io.file.buffer.size</name>
    8.         <value>131072</value>
    9.     </property>
    10.     <property>
    11.         <name>hadoop.tmp.dir</name>
    12.         <value>file:/hadoop/hdfs/temp</value>
    13.         <description>Abase for other temporary directories.</description>
    14.     </property>
    15.     <property>
    16.         <name>hadoop.proxyuser.hduser.hosts</name>
    17.         <value>*</value>
    18.     </property>
    19.     <property>
    20.         <name>hadoop.proxyuser.hduser.groups</name>
    21.         <value>*</value>
    22.     </property>
    23.         <property>     
    24.         <name>ha.zookeeper.quorum</name>     
    25.         <value>dn-1:2181,dn-2:2181,dn-3:2181</value>    
    26.     </property>
    27. </configuration>

    3.4 修改hdfs-site.xml

    1. <configuration>
    2.     <property> 
    3.         <name>dfs.namenode.secondary.http-address</name>
    4.         <value>nn-1:9001</value>
    5.     </property>
    6.     <property>
    7.         <name>dfs.namenode.name.dir</name>
    8.         <value>file:/hadoop/hdfs/name</value>
    9.     </property>
    10.     <property>
    11.         <name>dfs.datanode.data.dir</name>
    12.         <value>file:/hadoop/hdfs/data,file:/hadoopdata/hdfs/data</value>
    13.     </property>
    14.     <property>
    15.         <name>dfs.replication</name>
    16.         <value>3</value>
    17.     </property>
    18.     <property>
    19.         <name>dfs.webhdfs.enabled</name>
    20.         <value>true</value>
    21.     </property>
    22.     <property>
    23.         <name>dfs.nameservices</name>
    24.         <value>dpi</value>
    25.     </property>
    26.     <property>
    27.         <name>dfs.ha.namenodes.dpi</name>
    28.         <value>nn-1,nn-2</value>
    29.     </property>
    30.     <property>
    31.         <name>dfs.namenode.rpc-address.dpi.nn-1</name>
    32.         <value>nn-1:9000</value>
    33.     </property>
    34.     <property>    
    35.         <name>dfs.namenode.http-address.dpi.nn-1</name>    
    36.         <value>nn-1:50070</value>    
    37.     </property>
    38.     <property>    
    39.         <name>dfs.namenode.rpc-address.dpi.nn-2</name>    
    40.         <value>nn-2:9000</value>    
    41.     </property>
    42.     <property>    
    43.         <name>dfs.namenode.http-address.dpi.nn-2</name>    
    44.         <value>nn-2:50070</value>    
    45.     </property>
    46.     <property>
    47.         <name>dfs.namenode.servicerpc-address.dpi.nn-1</name>
    48.         <value>nn-1:53310</value>
    49.     </property>
    50.     <property>
    51.         <name>dfs.namenode.servicerpc-address.dpi.nn-2</name>
    52.         <value>nn-2:53310</value>
    53.     </property>
    54.     <property>  
    55.         <name>dfs.ha.automatic-failover.enabled</name>  
    56.         <value>true</value>  
    57.     </property>  
    58.     <property>
    59.         <name>dfs.namenode.shared.edits.dir</name>
    60.         <value>qjournal://nn-1:8485;nn-2:8485;dn-1:8485/dpi</value>
    61.     </property>
    62.     <property>
    63.         <name>dfs.client.failover.proxy.provider.dpi</name>
    64.         <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> 
    65.     </property>
    66.     <property>    
    67.         <name>dfs.journalnode.edits.dir</name>    
    68.         <value>/hadoop/hdfs/journal</value>    
    69.     </property>
    70.     <property>    
    71.         <name>dfs.ha.fencing.methods</name>    
    72.         <value>sshfence</value>    
    73.     </property>
    74.     <property>    
    75.         <name>dfs.ha.fencing.ssh.private-key-files</name>    
    76.         <value>/hadoop/.ssh/id_rsa</value>    
    77.     </property>
    78. </configuration>
    新建配置文件中的目录：
    1. mkdir -p /hadoop/hdfs/name
    2. mkdir -p /hadoop/hdfs/data
    3. mkdir -p /hadoop/hdfs/temp
    4. mkdir -p /hadoop/hdfs/journal
    5. 授权：chmod 755 /hadoop/hdfs
    6. mkdir -p /hadoopdata/hdfs/data
    7. chmod 755 /hadoopdata/hdfs

    属主和属组修改为：hadoop:hadoop


    3.5 修改mapred-site.xml

    1. <configuration>
    2.     <property>
    3.         <name>mapreduce.framework.name</name>
    4.         <value>yarn</value>
    5.     </property>
    6.     <property>
    7.         <name>mapreduce.jobhistory.address</name>
    8.         <value>nn-1:10020</value>
    9.     </property>
    10.     <property>
    11.         <name>mapreduce.jobhistory.webapp.address</name>
    12.         <value>nn-1:19888</value>
    13.     </property>
    14. </configuration>


    3.6 修改yarn-site.xml

    启用yarn ha功能， 根据规划， dn-1和dn-2为ResourceManager节点
    1. <configuration>
    2.  <!-- Site specific YARN configuration properties -->
    3.         <property>
    4.                 <name>yarn.nodemanager.aux-services</name>
    5.                 <value>mapreduce_shuffle</value>
    6.         </property>
    7.         <property>
    8.                 <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    9.                 <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    10.         </property>
    11.         <property>
    12.             <name>yarn.resourcemanager.ha.enabled</name>
    13.             <value>true</value>
    14.         </property>
    15.         <property>
    16.             <name>yarn.resourcemanager.ha.rm-ids</name>
    17.             <value>rm1,rm2</value>
    18.         </property>
    19.         <property>
    20.             <name>yarn.resourcemanager.hostname.rm1</name>
    21.             <value>dn-1</value>
    22.         </property>
    23.         <property>
    24.             <name>yarn.resourcemanager.hostname.rm2</name>
    25.             <value>dn-2</value>
    26.         </property>
    27.         <property>
    28.             <name>yarn.resourcemanager.recovery.enabled</name>
    29.             <value>true</value>
    30.         </property>
    31.         <property>
    32.             <name>yarn.resourcemanager.store.class</name>
    33.             <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    34.         </property>
    35.         <property>
    36.             <name>yarn.resourcemanager.zk-address</name>
    37.             <value>dn-1:2181,dn-2:2181,dn-3:2181</value>
    38.             <description>For multiple zk services, separate them with comma</description>
    39.         </property>
    40.         <property>
    41.             <name>yarn.resourcemanager.cluster-id</name>
    42.             <value>yarn-ha</value>
    43.         </property>
    44. </configuration>

    3.7 修改slaves

    将所有的DataNode节点加入到slaves文件中：
    1. dn-1
    2. dn-2
    3. dn-3


    3.8 修改yarn-env.sh

    1. # some Java parameters
    2. # export JAVA_HOME=/home/y/libexec/jdk1.6.0/
    3. if [ "$JAVA_HOME" != "" ]; then
    4.   #echo "run java in $JAVA_HOME"
    5.   JAVA_HOME=/usr/local/java/jdk1.7.0_79
    6. fi
    8. JAVA_HEAP_MAX=-Xmx15000m
    9. YARN_HEAPSIZE=15000
    10. export  YARN_RESOURCEMANAGER_HEAPSIZE=5000
    11. export  YARN_TIMELINESERVER_HEAPSIZE=10000
    12. export  YARN_NODEMANAGER_HEAPSIZE=10000

    3.9 分发配置好的hadoop目录到所有节点

    1. [hadoop@nn-1 ~]$ scp -rp hadoop hadoop@nn-2:/hadoop
    2. [hadoop@nn-1 ~]$ scp -rp hadoop hadoop@dn-1:/hadoop
    3. [hadoop@nn-1 ~]$ scp -rp hadoop hadoop@dn-2:/hadoop
    4. [hadoop@nn-1 ~]$ scp -rp hadoop hadoop@dn-3:/hadoop

    4 安装配置zookeeper

    切换到hadoop目录下面， 根据规划， 三台zookeeper节点为：nn-1, dn-1, dn-2。
    先在nn-1节点配置zookeeper， 然后分发至三个zookeeper节点：

    4.1 在nn-1上传并解压zookeeper


    4.2 修改配置文件/hadoop/zookeeper/conf/zoo.cfg

    1. dataDir=/hadoop/zookeeper/data/
    2. dataLogDir=/hadoop/zookeeper/log/
    3. # the port at which the clients will connect
    4. clientPort=2181
    5. server.1=nn-1:2887:3887
    6. server.2=dn-1:2888:3888
    7. server.3=dn-2:2889:3889

    4.3 从nn-1分发配置的zookeeper目录到其他节点

    1. [hadoop@nn-1 ~]$ scp -rp zookeeper  hadoop@dn-1:/hadoop
    2. [hadoop@nn-1 ~]$ scp -rp zookeeper  hadoop@dn-2:/hadoop

    4.4 在所有zk节点创建目录

    1. [hadoop@dn-1 ~]$ mkdir /hadoop/zookeeper/data/
    2. [hadoop@dn-1 ~]$ mkdir /hadoop/zookeeper/log/

    4.5 修改myid

    在所有zk节点， 切换到目录/hadoop/zookeeper/data，创建myid文件：
    注意：myid文件的内容为zoo.cfg文件中配置的server.后面的数字（即nn-1为1,dn-1为2,dn-2为3）。
    在nn-1节点的myid内容为：
    1. [hadoop@nn-1 data]$ echo 1 > /hadoop/zookeeper/data/myid

    其他zk节点也安要求创建myid文件。


    4.6 设置环境变量

    1. $ echo "export ZOOKEEPER_HOME=/hadoop/zookeeper" >> $HOME/.bash_profile
    2. $ echo "export PATH=$ZOOKEEPER_HOME/bin:$PATH" >> $HOME/.bash_profile
    3. $ source $HOME/.bash_profile


    5 集群启动

    5.1 启动zookeeper

    根据规划， zk的节点为nn-1、dn-1和dn-2， 在这三台节点分别启动zk：

    启动命令：
    1. [hadoop@nn-1 ~]$ /hadoop/zookeeper/bin/zkServer.sh start
    2. JMX enabled by default
    3. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
    4. Starting zookeeper ... STARTED

    查看进程， 可以看到QuorumPeerMain：
    1. [hadoop@nn-1 ~]$ jps
    2. 9382 QuorumPeerMain
    3. 9407 Jps

    查看状态， 可以看到Mode: follower， 说明这是zk的从节点：
    1. [hadoop@nn-1 ~]$ /hadoop/zookeeper/bin/zkServer.sh status
    2. JMX enabled by default
    3. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
    4. Mode: follower

    查看状态， 可以看到Mode: leader， 说明这是zk的leader节点：
    1. [hadoop@dn-1 data]$ /hadoop/zookeeper/bin/zkServer.sh status
    2. JMX enabled by default
    3. Using config: /hadoop/zookeeper/bin/../conf/zoo.cfg
    4. Mode: leader

    5.2 格式化zookeeper集群（只做一次）（机器nn-1上执行）


    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/bin/hdfs zkfc -formatZK
    中间有个交互的步骤， 输入Y：
     
    进入zk， 查看是否创建成功：
    1. [hadoop@nn-1 bin]$ ./zkCli.sh
     

    5.3 启动zkfc（机器nn-1,nn-2上执行）

    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/sbin/hadoop-daemon.sh start zkfc
    2. starting zkfc, logging to /hadoop/hadoop/logs/hadoop-hadoop-zkfc-nn-1.out

    使用jps， 可以看到进程DFSZKFailoverController：
    1. [hadoop@nn-1 ~]$ jps
    2. 9681 Jps
    3. 9638 DFSZKFailoverController
    4. 9382 QuorumPeerMain

     

    5.4 启动journalnode

    根据规划， 启动journalnode节点为nn-1、nn-2和dn-1， 在这三个节点分别使用如下的命令启动服务：
    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/sbin/hadoop-daemon.sh  start  journalnode
    2. starting journalnode, logging to /hadoop/hadoop/logs/hadoop-hadoop-journalnode-nn-1.out

    使用jps命令可以看到进程JournalNode：
    1. [hadoop@nn-1 ~]$ jps
    2. 9714 JournalNode
    3. 9638 DFSZKFailoverController
    4. 9382 QuorumPeerMain
    5. 9762 Jps

    5.5 格式化namenode（机器nn-1上执行）

    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/bin/hadoop namenode -format

    查看日志信息：
     

    5.6 启动namenode（机器nn-1上执行）

    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/sbin/hadoop-daemon.sh  start  namenode
    2. starting namenode, logging to /hadoop/hadoop/logs/hadoop-hadoop-namenode-nn-1.out
    使用jps命令可以看到进程NameNode：
    1. [hadoop@nn-1 ~]$ jps
    2. 9714 JournalNode
    3. 9638 DFSZKFailoverController
    4. 9382 QuorumPeerMain
    5. 10157 NameNode
    6. 10269 Jps

    5.7 格式化secondnamnode（机器nn-2上执行）

    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs namenode -bootstrapStandby
    部分日志如下：
     

    5.8 启动namenode（机器nn-2上执行）

    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/sbin/hadoop-daemon.sh  start  namenode
    2. starting namenode, logging to /hadoop/hadoop/logs/hadoop-hadoop-namenode-nn-2.out
    使用jps命令可以看到进程NameNode：
    1. [hadoop@nn-2 ~]$ jps
    2. 53990 NameNode
    3. 54083 Jps
    4. 53824 JournalNode
    5. 53708 DFSZKFailoverController

    5.9 启动datanode（机器dn-1到dn-3上执行）

    1. [hadoop@dn-1 ~]$ /hadoop/hadoop/sbin/hadoop-daemon.sh  start  datanode
    使用jps可以看到DataNode进程：
    1. [hadoop@dn-1 temp]$ jps
    2. 57007 Jps
    3. 56927 DataNode
    4. 56223 QuorumPeerMain


    5.10 启动resourcemanager

    根据规划，resourcemanager做了HA， 服务在节点dn-1和dn-2上面， 在dn-1和dn-2上面启动resourcemanager：
    1. [hadoop@dn-1 ~]$ /hadoop/hadoop/sbin/yarn-daemon.sh  start  resourcemanager
    2. starting resourcemanager, logging to /hadoop/hadoop/logs/yarn-hadoop-resourcemanager-dn-1.out

    使用jps， 可以看到进程ResourceManager：
    1. [hadoop@dn-1 ~]$ jps
    2. 57173 QuorumPeerMain
    3. 58317 Jps
    4. 57283 JournalNode
    5. 58270 ResourceManager
    6. 58149 DataNode

    5.11 启动jobhistory

    根据规划， jobhistory服务在nn-1上面， 使用如下命令启动：
    1. [hadoop@nn-1 ~]$ /hadoop/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver
    2. starting historyserver, logging to /hadoop/hadoop/logs/mapred-hadoop-historyserver-nn-1.out

    使用jps， 可以看到进程JobHistoryServer：
    1. [hadoop@nn-1 ~]$ jps
    2. 11210 JobHistoryServer
    3. 9714 JournalNode
    4. 9638 DFSZKFailoverController
    5. 9382 QuorumPeerMain
    6. 11039 NameNode
    7. 11303 Jps

    5.12 启动NodeManager

    根据规划， dn-1、dn-2和dn-3是nodemanager， 在这三个节点启动NodeManager：
    1. [hadoop@dn-1 ~]$ /hadoop/hadoop/sbin/yarn-daemon.sh start nodemanager
    2. starting nodemanager, logging to /hadoop/hadoop/logs/yarn-hadoop-nodemanager-dn-1.out

    使用jps可以看到进程NodeManager：
    1. [hadoop@dn-1 ~]$ jps
    2. 58559 NodeManager
    3. 57173 QuorumPeerMain
    4. 58668 Jps
    5. 57283 JournalNode
    6. 58270 ResourceManager
    7. 58149 DataNode


    6、安装后查看和验证

    6.1 HDFS相关操作命令

    查看NameNode状态的命令
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs haadmin -getServiceState nn-1

    手工切换，将active的NameNode从nn-1切换到nn-2 。
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs haadmin -DfSHAadmin -failover nn-1 nn-2
     
    NameNode健康检查：
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs haadmin -checkHealth nn-1
     将其中一台NameNode给kill后， 查看健康状态：
     


    查看所有的DataNode列表：
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs dfsadmin -report | more
     
    查看正常DataNode列表:
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs dfsadmin -report -live
    2. 17/03/01 22:49:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Configured Capacity: 224954695680 (209.51 GB)
    4. Present Capacity: 180557139968 (168.16 GB)
    5. DFS Remaining: 179963428864 (167.60 GB)
    6. DFS Used: 593711104 (566.21 MB)
    7. DFS Used%: 0.33%
    8. Under replicated blocks: 2
    9. Blocks with corrupt replicas: 0
    10. Missing blocks: 0
    12. -------------------------------------------------
    13. Live datanodes (3):
    15. Name: 192.168.9.23:50010 (dn-1)
    16. Hostname: dn-1
    17. Rack: /rack2
    18. Decommission Status : Normal
    19. Configured Capacity: 74984898560 (69.84 GB)
    20. DFS Used: 197902336 (188.73 MB)
    21. Non DFS Used: 14869356544 (13.85 GB)
    22. DFS Remaining: 59917639680 (55.80 GB)
    23. DFS Used%: 0.26%
    24. DFS Remaining%: 79.91%
    25. Configured Cache Capacity: 0 (0 B)
    26. Cache Used: 0 (0 B)
    27. Cache Remaining: 0 (0 B)
    28. Cache Used%: 100.00%
    29. Cache Remaining%: 0.00%
    30. Xceivers: 1
    31. Last contact: Wed Mar 01 22:49:42 CST 2017

    查看异常DataNode列表：
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs dfsadmin -report -dead

    获取指定DataNode信息(运行时间及版本等):
    1. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs haadmin -checkHealth nn-2
    2. 17/03/01 22:55:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. [hadoop@nn-2 ~]$ /hadoop/hadoop/bin/hdfs haadmin -checkHealth nn-1
    4. 17/03/01 22:55:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable


    6.2 YARN相关的命令

    查看resourceManager状态的命令：
    1. [hadoop@dn-1 hadoop]$ yarn rmadmin -getServiceState rm1
    2. active
    3. [hadoop@dn-1 hadoop]$ yarn rmadmin -getServiceState rm2
    4. standby

    查看所有的yarn节点：
    1. [hadoop@dn-1 hadoop]$ /hadoop/hadoop/bin/yarn node -all -list
    2. 17/03/01 23:06:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Total Nodes:3
    4.          Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
    5.       dn-2:55506	        RUNNING	        dn-2:8042	                           0
    6.       dn-1:56447	        RUNNING	        dn-1:8042	                           0
    7.       dn-3:37533	        RUNNING	        dn-3:8042	                           0

    查看正常的yarn节点：
    1. [hadoop@dn-1 hadoop]$ /hadoop/hadoop/bin/yarn node -list
    2. 17/03/01 23:07:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Total Nodes:3
    4.          Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
    5.       dn-2:55506	        RUNNING	        dn-2:8042	                           0
    6.       dn-1:56447	        RUNNING	        dn-1:8042	                           0
    7.       dn-3:37533	        RUNNING	        dn-3:8042	                           0

    查看指定节点的信息：
    /hadoop/hadoop/bin/yarn node -status <NodeId>
    1. [hadoop@dn-1 hadoop]$ /hadoop/hadoop/bin/yarn node -status dn-2:55506
    2. 17/03/01 23:08:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Node Report : 
    4. 	Node-Id : dn-2:55506
    5. 	Rack : /default-rack
    6. 	Node-State : RUNNING
    7. 	Node-Http-Address : dn-2:8042
    8. 	Last-Health-Update : 星期三 01/三月/17 11:06:21:373CST
    9. 	Health-Report : 
    10. 	Containers : 0
    11. 	Memory-Used : 0MB
    12. 	Memory-Capacity : 8192MB
    13. 	CPU-Used : 0 vcores
    14. 	CPU-Capacity : 8 vcores
    15. 	Node-Labels :

    查看当前运行的MapReduce任务：
    1. [hadoop@dn-2 ~]$ /hadoop/hadoop/bin/yarn application -list
    2. 17/03/01 23:10:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1
    4.                 Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
    5. application_1488375590901_0004	     QuasiMonteCarlo	           MAPREDUCE	    hadoop	   default	           RUNNING	         UNDEFINED


    6.3 使用自带的例子测试

    1. [hadoop@dn-1 ~]$ cd hadoop/
    2. [hadoop@dn-1 hadoop]$ 
    3. [hadoop@dn-1 hadoop]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 200

    1. [hadoop@dn-1 hadoop]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 200
    2. Number of Maps  = 2
    3. Samples per Map = 200
    4. 17/02/28 01:51:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    5. Wrote input for Map #0
    6. Wrote input for Map #1
    7. Starting Job
    8. 17/02/28 01:51:15 INFO input.FileInputFormat: Total input paths to process : 2
    9. 17/02/28 01:51:15 INFO mapreduce.JobSubmitter: number of splits:2
    10. 17/02/28 01:51:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1488216892564_0001
    11. 17/02/28 01:51:16 INFO impl.YarnClientImpl: Submitted application application_1488216892564_0001
    12. 17/02/28 01:51:16 INFO mapreduce.Job: The url to track the job: http://dn-1:8088/proxy/application_1488216892564_0001/
    13. 17/02/28 01:51:16 INFO mapreduce.Job: Running job: job_1488216892564_0001
    14. 17/02/28 01:51:24 INFO mapreduce.Job: Job job_1488216892564_0001 running in uber mode : false
    15. 17/02/28 01:51:24 INFO mapreduce.Job:  map 0% reduce 0%
    16. 17/02/28 01:51:38 INFO mapreduce.Job:  map 100% reduce 0%
    17. 17/02/28 01:51:49 INFO mapreduce.Job:  map 100% reduce 100%
    18. 17/02/28 01:51:49 INFO mapreduce.Job: Job job_1488216892564_0001 completed successfully
    19. 17/02/28 01:51:50 INFO mapreduce.Job: Counters: 49
    20. 	File System Counters
    21. 		FILE: Number of bytes read=50
    22. 		FILE: Number of bytes written=326922
    23. 		FILE: Number of read operations=0
    24. 		FILE: Number of large read operations=0
    25. 		FILE: Number of write operations=0
    26. 		HDFS: Number of bytes read=510
    27. 		HDFS: Number of bytes written=215
    28. 		HDFS: Number of read operations=11
    29. 		HDFS: Number of large read operations=0
    30. 		HDFS: Number of write operations=3
    31. 	Job Counters 
    32. 		Launched map tasks=2
    33. 		Launched reduce tasks=1
    34. 		Data-local map tasks=2
    35. 		Total time spent by all maps in occupied slots (ms)=25604
    36. 		Total time spent by all reduces in occupied slots (ms)=7267
    37. 		Total time spent by all map tasks (ms)=25604
    38. 		Total time spent by all reduce tasks (ms)=7267
    39. 		Total vcore-seconds taken by all map tasks=25604
    40. 		Total vcore-seconds taken by all reduce tasks=7267
    41. 		Total megabyte-seconds taken by all map tasks=26218496
    42. 		Total megabyte-seconds taken by all reduce tasks=7441408
    43. 	Map-Reduce Framework
    44. 		Map input records=2
    45. 		Map output records=4
    46. 		Map output bytes=36
    47. 		Map output materialized bytes=56
    48. 		Input split bytes=274
    49. 		Combine input records=0
    50. 		Combine output records=0
    51. 		Reduce input groups=2
    52. 		Reduce shuffle bytes=56
    53. 		Reduce input records=4
    54. 		Reduce output records=0
    55. 		Spilled Records=8
    56. 		Shuffled Maps =2
    57. 		Failed Shuffles=0
    58. 		Merged Map outputs=2
    59. 		GC time elapsed (ms)=419
    60. 		CPU time spent (ms)=6940
    61. 		Physical memory (bytes) snapshot=525877248
    62. 		Virtual memory (bytes) snapshot=2535231488
    63. 		Total committed heap usage (bytes)=260186112
    64. 	Shuffle Errors
    65. 		BAD_ID=0
    66. 		CONNECTION=0
    67. 		IO_ERROR=0
    68. 		WRONG_LENGTH=0
    69. 		WRONG_MAP=0
    70. 		WRONG_REDUCE=0
    71. 	File Input Format Counters 
    72. 		Bytes Read=236
    73. 	File Output Format Counters 
    74. 		Bytes Written=97
    75. Job Finished in 35.466 seconds
    76. Estimated value of Pi is 3.17000000000000000000

    6.4 查看NameNode

     链接分别为：

    192.168.9.21和192.168.9.22分别为NameNode和Secondary NameNode的地址。
     
     



    6.5 查看NameNode 的HA切换是否正常

    将nn-1上状态为active的NameNode进程kill， 查看nn-2上的NameNode能否从standby切换为active：
     

     


    6.6 查看RM页面


     



    查看节点信息， 192.168.9.23为Resource服务所在的active节点。
     


    运行测试任务， 查看YARN HA能否自动切换：
    1. [hadoop@dn-2 hadoop]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 200

    运行到中间的时候， 将rm1上的rm给kill掉， 查看切换是否正常：
     
    查看standby的HA state变成了active：


    查看下面的日志， 程序运行期间由于rm1被kill， 程序报错， 然后Trying to fail over immediately， 最终程序运行成功。
     
    1. [hadoop@dn-2 hadoop]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar pi 2 200
    3. Number of Maps  = 2
    4. Samples per Map = 200
    5. 17/02/28 02:11:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    6. Wrote input for Map #0
    7. Wrote input for Map #1
    8. Starting Job
    9. 17/02/28 02:11:12 INFO input.FileInputFormat: Total input paths to process : 2
    10. 17/02/28 02:11:12 INFO mapreduce.JobSubmitter: number of splits:2
    11. 17/02/28 02:11:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1488216892564_0002
    12. 17/02/28 02:11:13 INFO impl.YarnClientImpl: Submitted application application_1488216892564_0002
    13. 17/02/28 02:11:13 INFO mapreduce.Job: The url to track the job: http://dn-1:8088/proxy/application_1488216892564_0002/
    14. 17/02/28 02:11:13 INFO mapreduce.Job: Running job: job_1488216892564_0002
    15. 17/02/28 02:11:18 INFO retry.RetryInvocationHandler: Exception while invoking getApplicationReport of class ApplicationClientProtocolPBClientImpl over rm1. Trying to fail over immediately.
    16. java.io.EOFException: End of File Exception between local host is: "dn-2/192.168.9.24"; destination host is: "dn-1":8032; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
    17. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    18. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    19. 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    20. 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    21. 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    22. 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    23. 	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    24. 	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    25. 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    26. 	at com.sun.proxy.$Proxy14.getApplicationReport(Unknown Source)
    27. 	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
    28. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    29. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    30. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    31. 	at java.lang.reflect.Method.invoke(Method.java:606)
    32. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    33. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    34. 	at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)
    35. 	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)
    36. 	at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)
    37. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)
    38. 	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)
    39. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:422)
    40. 	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575)
    41. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:325)
    42. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322)
    43. 	at java.security.AccessController.doPrivileged(Native Method)
    44. 	at javax.security.auth.Subject.doAs(Subject.java:415)
    45. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    46. 	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
    47. 	at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:610)
    48. 	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1355)
    49. 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1317)
    50. 	at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
    51. 	at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
    52. 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    53. 	at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
    54. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    55. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    56. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    57. 	at java.lang.reflect.Method.invoke(Method.java:606)
    58. 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    59. 	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    60. 	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    61. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    62. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    63. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    64. 	at java.lang.reflect.Method.invoke(Method.java:606)
    65. 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    66. 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    67. Caused by: java.io.EOFException
    68. 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
    69. 	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)
    70. 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)
    71. 17/02/28 02:11:18 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
    72. 17/02/28 02:11:18 INFO retry.RetryInvocationHandler: Exception while invoking getApplicationReport of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over after sleeping for 40859ms.
    73. java.net.ConnectException: Call From dn-2/192.168.9.24 to dn-2:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    74. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    75. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    76. 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    77. 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    78. 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    79. 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    80. 	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    81. 	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    82. 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    83. 	at com.sun.proxy.$Proxy14.getApplicationReport(Unknown Source)
    84. 	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
    85. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    86. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    87. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    88. 	at java.lang.reflect.Method.invoke(Method.java:606)
    89. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    90. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    91. 	at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)
    92. 	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)
    93. 	at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)
    94. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)
    95. 	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)
    96. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:422)
    97. 	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575)
    98. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:325)
    99. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322)
    100. 	at java.security.AccessController.doPrivileged(Native Method)
    101. 	at javax.security.auth.Subject.doAs(Subject.java:415)
    102. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    103. 	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
    104. 	at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:610)
    105. 	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1355)
    106. 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1317)
    107. 	at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
    108. 	at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
    109. 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    110. 	at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
    111. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    112. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    113. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    114. 	at java.lang.reflect.Method.invoke(Method.java:606)
    115. 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    116. 	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    117. 	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    118. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    119. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    120. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    121. 	at java.lang.reflect.Method.invoke(Method.java:606)
    122. 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    123. 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    124. Caused by: java.net.ConnectException: 拒绝连接
    125. 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    126. 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    127. 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    128. 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    129. 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    130. 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
    131. 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
    132. 	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
    133. 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    134. 	at org.apache.hadoop.ipc.Client.call(Client.java:1438)
    135. 	... 43 more
    136. 17/02/28 02:11:59 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm1
    137. 17/02/28 02:11:59 INFO retry.RetryInvocationHandler: Exception while invoking getApplicationReport of class ApplicationClientProtocolPBClientImpl over rm1 after 2 fail over attempts. Trying to fail over after sleeping for 17213ms.
    138. java.net.ConnectException: Call From dn-2/192.168.9.24 to dn-1:8032 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    139. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    140. 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    141. 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    142. 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    143. 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    144. 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    145. 	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    146. 	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    147. 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    148. 	at com.sun.proxy.$Proxy14.getApplicationReport(Unknown Source)
    149. 	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
    150. 	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    151. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    152. 	at java.lang.reflect.Method.invoke(Method.java:606)
    153. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    154. 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    155. 	at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)
    156. 	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:399)
    157. 	at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:302)
    158. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:153)
    159. 	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:322)
    160. 	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:422)
    161. 	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:575)
    162. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:325)
    163. 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322)
    164. 	at java.security.AccessController.doPrivileged(Native Method)
    165. 	at javax.security.auth.Subject.doAs(Subject.java:415)
    166. 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    167. 	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
    168. 	at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:610)
    169. 	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1355)
    170. 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1317)
    171. 	at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
    172. 	at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
    173. 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    174. 	at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
    175. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    176. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    177. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    178. 	at java.lang.reflect.Method.invoke(Method.java:606)
    179. 	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    180. 	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    181. 	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    182. 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    183. 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    184. 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    185. 	at java.lang.reflect.Method.invoke(Method.java:606)
    186. 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    187. 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    188. Caused by: java.net.ConnectException: 拒绝连接
    189. 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    190. 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    191. 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    192. 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    193. 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    194. 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
    195. 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
    196. 	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
    197. 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    198. 	at org.apache.hadoop.ipc.Client.call(Client.java:1438)
    199. 	... 42 more
    200. 17/02/28 02:12:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
    201. 17/02/28 02:12:18 INFO mapreduce.Job: Job job_1488216892564_0002 running in uber mode : false
    202. 17/02/28 02:12:18 INFO mapreduce.Job:  map 0% reduce 0%
    203. 17/02/28 02:12:22 INFO mapreduce.Job:  map 100% reduce 0%
    204. 17/02/28 02:12:28 INFO mapreduce.Job:  map 100% reduce 100%
    205. 17/02/28 02:12:28 INFO mapreduce.Job: Job job_1488216892564_0002 completed successfully
    206. 17/02/28 02:12:28 INFO mapreduce.Job: Counters: 49
    207. 	File System Counters
    208. 		FILE: Number of bytes read=50
    209. 		FILE: Number of bytes written=326931
    210. 		FILE: Number of read operations=0
    211. 		FILE: Number of large read operations=0
    212. 		FILE: Number of write operations=0
    213. 		HDFS: Number of bytes read=510
    214. 		HDFS: Number of bytes written=215
    215. 		HDFS: Number of read operations=11
    216. 		HDFS: Number of large read operations=0
    217. 		HDFS: Number of write operations=3
    218. 	Job Counters 
    219. 		Launched map tasks=2
    220. 		Launched reduce tasks=1
    221. 		Data-local map tasks=2
    222. 		Total time spent by all maps in occupied slots (ms)=22713
    223. 		Total time spent by all reduces in occupied slots (ms)=3213
    224. 		Total time spent by all map tasks (ms)=22713
    225. 		Total time spent by all reduce tasks (ms)=3213
    226. 		Total vcore-seconds taken by all map tasks=22713
    227. 		Total vcore-seconds taken by all reduce tasks=3213
    228. 		Total megabyte-seconds taken by all map tasks=23258112
    229. 		Total megabyte-seconds taken by all reduce tasks=3290112
    230. 	Map-Reduce Framework
    231. 		Map input records=2
    232. 		Map output records=4
    233. 		Map output bytes=36
    234. 		Map output materialized bytes=56
    235. 		Input split bytes=274
    236. 		Combine input records=0
    237. 		Combine output records=0
    238. 		Reduce input groups=2
    239. 		Reduce shuffle bytes=56
    240. 		Reduce input records=4
    241. 		Reduce output records=0
    242. 		Spilled Records=8
    243. 		Shuffled Maps =2
    244. 		Failed Shuffles=0
    245. 		Merged Map outputs=2
    246. 		GC time elapsed (ms)=233
    247. 		CPU time spent (ms)=12680
    248. 		Physical memory (bytes) snapshot=517484544
    249. 		Virtual memory (bytes) snapshot=2548441088
    250. 		Total committed heap usage (bytes)=260186112
    251. 	Shuffle Errors
    252. 		BAD_ID=0
    253. 		CONNECTION=0
    254. 		IO_ERROR=0
    255. 		WRONG_LENGTH=0
    256. 		WRONG_MAP=0
    257. 		WRONG_REDUCE=0
    258. 	File Input Format Counters 
    259. 		Bytes Read=236
    260. 	File Output Format Counters 
    261. 		Bytes Written=97
    262. Job Finished in 76.447 seconds
    263. Estimated value of Pi is 3.17000000000000000000
    264. [hadoop@dn-2 hadoop]$


    7、安装Spark


    规划， 在现有的Hadoop集群安装spark集群：
    master节点： nn-1
    worker节点： nn-2、dn-1、dn-2、dn-3。

    7.1 安装配置Scala

    上传安装包到nn-1的/hadoop目录下面，解压：
    1. [hadoop@nn-1 ~]$ tar -xzvf spark-1.6.0-bin-hadoop2.6.tgz
    环境变量后面统一配置。

    7.2 安装spark


    上传安装包spark-1.6.0-bin-hadoop2.6.tgz到nn-1的目录/hadoop下面， 解压
    1. [hadoop@nn-1 ~]$ tar -xzvf spark-1.6.0-bin-hadoop2.6.tgz

    进入目录：/hadoop/spark-1.6.0-bin-hadoop2.6/conf
    复制生成文件spark-env.sh和slaves：
    1. [hadoop@nn-1 conf]$ pwd
    2. /hadoop/spark-1.6.0-bin-hadoop2.6/conf
    3. [hadoop@nn-1 conf]$ cp spark-env.sh.template spark-env.sh
    4. [hadoop@nn-1 conf]$ cp slaves.template  slaves
    编辑spark-env.sh， 加入如下内容：
    1. export JAVA_HOME=/usr/local/java/jdk1.7.0_79
    2. export SCALA_HOME=/hadoop/scala-2.11.7
    3. export SPARK_HOME=/hadoop/spark-1.6.0-bin-hadoop2.6
    4. export SPARK_MASTER_IP=nn-1
    5. export SPARK_WORKER_MEMORY=2g
    6. export HADOOP_CONF_DIR=/hadoop/hadoop/etc/hadoop
    SPARK_WORKER_MEMORY根据实际情况配置。

    编辑spark-env.sh， 加入如下内容：slaves
    1. nn-2
    2. dn-1
    3. dn-2
    4. dn-3
    slaves指定的是worker节点。

    7.3 配置环境变量

    1. [hadoop@nn-1 ~]$ vim .bash_profile
    追加如下内容：
    1. export HADOOP_HOME=/hadoop/hadoop
    2. export SCALA_HOME=/hadoop/scala-2.11.7
    3. export SPARK_HOME=/hadoop/spark-1.6.0-bin-hadoop2.6
    4. export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

    7.4 分发上面配置好的scala和spark目录到其他节点

    1. [hadoop@nn-1 bin]$ cd /hadoop
    2. [hadoop@nn-1 ~]$ scp -rp spark-1.6.0-bin-hadoop2.6 hadoop@dn-1:/hadoop
    3. [hadoop@nn-1 ~]$ scp -rp scala-2.11.7 hadoop@dn-1:/hadoop

    7.5 启动Spark集群

    1. [hadoop@nn-1 ~]$ /hadoop/spark-1.6.0-bin-hadoop2.6/sbin/start-all.sh

    在nn-1和其他slaves节点查看进程：
    在nn-1节点， 可以看到Master进程：
    1. [hadoop@nn-1 ~]$ jps
    2. 2473 JournalNode
    3. 2541 NameNode
    4. 4401 Jps
    5. 2399 DFSZKFailoverController
    6. 2687 JobHistoryServer
    7. 2775 Master
    8. 2351 QuorumPeerMain

    在slaves节点可以看到Worker进程：
    1. [hadoop@dn-1 ~]$ jps
    2. 2522 NodeManager
    3. 3449 Jps
    4. 2007 QuorumPeerMain
    5. 2141 DataNode
    6. 2688 Worker
    7. 2061 JournalNode
    8. 2258 ResourceManager

    查看spark页面：

     

    7.6 运行测试案例

    ./bin/spark-submit --class org.apache.spark.examples.SparkPi 

                       --master yarn --deploy-mode cluster 

                       --driver-memory 100M

                       --executor-memory 200M

                       --executor-cores 1 

                       --queue default 

                       lib/spark-examples*.jar 10

    或者：

    ./bin/spark-submit --class org.apache.spark.examples.SparkPi 

                       --master yarn --deploy-mode cluster 

                       --executor-cores 1 

                       --queue default 

                       lib/spark-examples*.jar 10


     
     
     



    8、配置机架感知

    在nn-1和nn-2节点的配置文件/hadoop/hadoop/etc/hadoop/core-site.xml加入如下配置：
    1. <property>  
    2.     <name>topology.script.file.name</name>  
    3.     <value>/hadoop/hadoop/etc/hadoop/RackAware.py</value>  
    4. </property>
    新增文件：/hadoop/hadoop/etc/hadoop/RackAware.py，内容如下：
    1. #!/usr/bin/python
    2. #-*-coding:UTF-8 -*-
    3. import sys
    5. rack = {"dn-1":"rack2",
    6.         "dn-2":"rack1",
    7.         "dn-3":"rack1",
    8.         "192.168.9.23":"rack2",
    9.         "192.168.9.24":"rack1",
    10.         "192.168.9.25":"rack1",
    11. }
    14. if __name__=="__main__":
    15.     print "/" + rack.get(sys.argv[1],"rack0")
    设置权限：
    1. [root@nn-1 hadoop]# chmod +x RackAware.py 
    2. [root@nn-1 hadoop]# ll RackAware.py
    3. -rwxr-xr-x 1 hadoop hadoop 294 3月   1 21:24 RackAware.py

    重启nn-1和nn-2上的NameNode服务：
    1. [hadoop@nn-1 ~]$ hadoop-daemon.sh stop namenode
    2. stopping namenode
    3. [hadoop@nn-1 ~]$ hadoop-daemon.sh start namenode
    4. starting namenode, logging to /hadoop/hadoop/logs/hadoop-hadoop-namenode-nn-1.out

    查看日志：
    1. [root@nn-1 logs]# pwd
    2. /hadoop/hadoop/logs
    3. [root@nn-1 logs]# vim hadoop-hadoop-namenode-nn-1.log

     


    使用命令查看拓扑：
    1. [hadoop@dn-3 ~]$ hdfs dfsadmin -printTopology
    2. 17/03/02 00:21:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    3. Rack: /rack1
    4.    192.168.9.24:50010 (dn-2)
    5.    192.168.9.25:50010 (dn-3)
    7. Rack: /rack2
    8.    192.168.9.23:50010 (dn-1)
  • 相关阅读:
    蓝鸥0c考试 绝密
    省市县用对象实现
    省市区用字典实现
    3道比较有用的oc练习题
    ios 开发 OC编程 类的扩展 协议 延展和 类目
    ios 开发 OC编程 内存管理
    ios 开发 OC编程 块语法bolck的一些应用
    ios 开发 OC编程 块语法bolck
    中等难度的通讯录.字典 动态分组法
    TestFlight
  • 原文地址：https://www.cnblogs.com/xiaohe001/p/6484462.html
Copyright © 2011-2022 走看看