hadoop2.7.3部署安装

环境：5台linux服务器，两台namenode,5台datanode,(两台namenode为主备模式active standby)（两台namenode都为active）

#数据目录

mkdir -p /data/zk_data
mkdir -p /data/logs/zk_logs/
chown -R media:media /data export
cd /data/zk_data/

centos7.2中的设置方法

#打开文件的最大数

vi /etc/security/limits.d/20-nproc.conf

soft soft nproc 102400

root soft nproc unlimited

soft soft nofile 102400

soft hard nofile 102400

ubuntu中的设置方法：

vi /etc/security/limits.conf

media soft nproc 102400

media soft nproc 102400

media soft nofile 102400

media hard nofile 102400

二、系统环境配置

#root操作 (所有主机)

1.关闭selinux，firewall

vim /etc/selinux/config

#将SELINUX=enforcing改为SELINUX=disabled

#临时关闭setenforce 0

systemctl stop firewalld.service

systemctl disable firewalld.service

2.更改主机名

vi /etc/hostname

分别改名为node1 node2 node3 node4 node5 .......

3.配置host解析

192.168.0.179 hadoop01

192.168.0.189 hadoop02

192.168.0.190 hadoop03

192.168.0.191 hadoop04

192.168.0.200 hadoop05

4.添加hadoop用户和组

groupadd hadoop

useradd -g hadoop hadoop

echo "密码" | passwd --stdin hadoop

6.切换到hadoop用户

su - hadoop

#hadoop操作

7.配置免密码登陆（hadoop01）

ssh-keygen #一路回车

cat .ssh/id_rsa.pub > authorized_keys

chmod 600 authorized_keys

将其他主机上的authorized_keys文件内容添加到本机authorized_keys中。

然后将文件传给其他主机

先将其他4台机器上的秘钥拷贝到node1上

scp authorized_keys2 media@node1:/home/media

scp authorized_keys3 media@node1:/home/media

scp authorized_keys4 media@node1:/home/media

scp authorized_keys5 media@node1:/home/media

然后再node1上操作

cat authorized_keys2 >> authorized_keys

cat authorized_keys3 >> authorized_keys

cat authorized_keys4 >> authorized_keys

cat authorized_keys5 >> authorized_keys

最后再将合成好的文件分发到其他node上。

38 scp authorized_keys media@node2:/home/media/.ssh

39 scp authorized_keys media@node3:/home/media/.ssh
40 scp authorized_keys media@node4:/home/media/.ssh
41 scp authorized_keys media@node5:/home/media/.ssh

8.在登陆时不输入yes (所有主机)

vi .ssh/config

StrictHostKeyChecking ask 改成 StrictHostKeyChecking no

三、JDK安装

软件目录： /home/hadoop/

1.安装jdk

#root用户（所有主机）

tar zxf jdk1.8.0_101.tar.gz

#vim /etc/profile

export JAVA_HOME=/home/hadoop/jdk1.8.0_101

export CLASSPATH=.:$JAVA_HOME/lib/

export JRE_HOME=/home/hadoop/jdk1.8.0_101/jre

export PATH=:$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

#source /etc/profile

#java -version #检查，出现如下信息表示成功

java version "1.8.0_101"

Java(TM) SE Runtime Environment (build 1.8.0_101-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

四、安装zookeeper

tar zxf zookeeper-3.4.8.tar.gz

cd zookeeper-3.4.8/conf

cp zoo_sample.cfg zoo.cfg

vim zoo.cfg

dataDir=/home/hadoop/zookeeper-3.4.8/zkdata

server.1=hadoop01:2888:3888

server.2=hadoop02:2888:3888

server.3=hadoop03:2888:3888

server.4=hadoop04:2888:3888

server.5=hadoop05:2888:3888

cd /data/zk_data

echo 1 > myid

　zkServer.sh start zkServer.sh status

ZooKeeper JMX enabled by default
Using config: /home/media/app/zookeeper-3.4.8/bin/../conf/zoo.cfg
Mode: follower

将zookeeper-3.4.8文件拷贝到其他主机，并更改mydi里面的数字

五、安装hadoop

tar -zxf hadoop-2.7.3.tar.gz -C /home/hadoop/

Hadoop配置文件目录

/home/soft/app/hadoop-2.7.3/etc/hadoop

Hadoop创建的目录：(配置文件中会用到这个路径)

mkdir -p /data/hadoop_data/hdfs

mkdir -p /data/hadoop_data/tmp

mkdir -p /data/hadoop_data/hdfs/name

mkdir -p /data/hadoop_data/hdfs/data/disk1

chown -R soft:soft /data/

设置hadoop特有的环境变量

1./hadoop-2.7.3/etc/hadoop/hadoop-env.sh
2./hadoop-2.7.3/etc/hadoop/yarn-env.sh
3./hadoop-2.7.3/etc/hadoop/core-site.xml
4./hadoop-2.7.3/etc/hadoop/hdfs-site.xml
5./hadoop-2.7.3/etc/hadoop/mapred-site.xml
6./hadoop-2.7.3/etc/hadoop/yarn-site.xml

#配置hadoop-env.sh

 1 # Licensed to the Apache Software Foundation (ASF) under one
 2 # or more contributor license agreements.  See the NOTICE file
 3 # distributed with this work for additional information
 4 # regarding copyright ownership.  The ASF licenses this file
 5 # to you under the Apache License, Version 2.0 (the
 6 # "License"); you may not use this file except in compliance
 7 # with the License.  You may obtain a copy of the License at
 8 #
 9 #     http://www.apache.org/licenses/LICENSE-2.0
10 #
11 # Unless required by applicable law or agreed to in writing, software
12 # distributed under the License is distributed on an "AS IS" BASIS,
13 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 # See the License for the specific language governing permissions and
15 # limitations under the License.
16 
17 # Set Hadoop-specific environment variables here.
18 
19 # The only required environment variable is JAVA_HOME.  All others are
20 # optional.  When running a distributed configuration it is best to
21 # set JAVA_HOME in this file, so that it is correctly defined on
22 # remote nodes.
23 
24 # The java implementation to use.
25 export JAVA_HOME=/home/soft/app/jdk1.8.0_101
26 
27 # The jsvc implementation to use. Jsvc is required to run secure datanodes
28 # that bind to privileged ports to provide authentication of data transfer
29 # protocol.  Jsvc is not required if SASL is configured for authentication of
30 # data transfer protocol using non-privileged ports.
31 #export JSVC_HOME=${JSVC_HOME}
32 
33 export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
34 
35 # Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
36 for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
37   if [ "$HADOOP_CLASSPATH" ]; then
38     export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
39   else
40     export HADOOP_CLASSPATH=$f
41   fi
42 done
43 
44 # The maximum amount of heap to use, in MB. Default is 1000.
45 #export HADOOP_HEAPSIZE=
46 #export HADOOP_NAMENODE_INIT_HEAPSIZE=""
47 
48 # Extra Java runtime options.  Empty by default.
49 export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
50 
51 # Command specific options appended to HADOOP_OPTS when specified
52 export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
53 export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
54 
55 export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
56 
57 export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
58 export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"
59 
60 # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
61 export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
62 #HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"
63 
64 # On secure datanodes, user to run the datanode as after dropping privileges.
65 # This **MUST** be uncommented to enable secure HDFS if using privileged ports
66 # to provide authentication of data transfer protocol.  This **MUST NOT** be
67 # defined if SASL is configured for authentication of data transfer protocol
68 # using non-privileged ports.
69 export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
70 
71 # Where log files are stored.  $HADOOP_HOME/logs by default.
72 #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
73 
74 # Where log files are stored in the secure data environment.
75 export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
76 
77 ###
78 # HDFS Mover specific parameters
79 ###
80 # Specify the JVM options to be used when starting the HDFS Mover.
81 # These options will be appended to the options specified as HADOOP_OPTS
82 # and therefore may override any similar flags set in HADOOP_OPTS
83 #
84 # export HADOOP_MOVER_OPTS=""
85 
86 ###
87 # Advanced Users Only!
88 ###
89 
90 # The directory where pid files are stored. /tmp by default.
91 # NOTE: this should be set to a directory that can only be written to by 
92 #       the user that will run the hadoop daemons.  Otherwise there is the
93 #       potential for a symlink attack.
94 export HADOOP_PID_DIR=${HADOOP_PID_DIR}
95 export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
96 
97 # A string representing this instance of hadoop. $USER by default.
98 export HADOOP_IDENT_STRING=$USER
99 export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/home/soft/app/hadoop-2.7.3/lib/native"

View Code

#编辑core-site.xml

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3 <!--
 4   Licensed under the Apache License, Version 2.0 (the "License");
 5   you may not use this file except in compliance with the License.
 6   You may obtain a copy of the License at
 7 
 8     http://www.apache.org/licenses/LICENSE-2.0
 9 
10   Unless required by applicable law or agreed to in writing, software
11   distributed under the License is distributed on an "AS IS" BASIS,
12   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13   See the License for the specific language governing permissions and
14   limitations under the License. See accompanying LICENSE file.
15 -->
16 
17 <!-- Put site-specific property overrides in this file. -->
18 
19 <configuration>
20         <property>
21                 <name>fs.defaultFS</name>
22                 <value>hdfs://ns</value>
23         </property>
24   <property>
25     <name>hadoop.tmp.dir</name>
26     <value>/data/hadoop_data/tmp</value>
27   </property>
28 <!-- 指定zookeeper地址 -->
29         <property>
30         <name>ha.zookeeper.quorum</name>
31         <value>node1:2181,node2:2181,node3:2181,node4:2181</value>
32         </property>
33 <property>
34 <name>fs.hdfs.impl</name>
35 <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
36 <description>The FileSystem for hdfs: uris.</description>
37 </property>
38 <property>
39   <name>hadoop.proxyuser.hue.hosts</name>
40   <value>*</value>
41 </property>
42 <property>
43   <name>hadoop.proxyuser.hue.groups</name>
44   <value>*</value>
45 </property>
46 </configuration>

View Code

#编辑hdfs-site.xml

  1 <?xml version="1.0" encoding="UTF-8"?>
  2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3 <!--
  4   Licensed under the Apache License, Version 2.0 (the "License");
  5   you may not use this file except in compliance with the License.
  6   You may obtain a copy of the License at
  7 
  8     http://www.apache.org/licenses/LICENSE-2.0
  9 
 10   Unless required by applicable law or agreed to in writing, software
 11   distributed under the License is distributed on an "AS IS" BASIS,
 12   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 13   See the License for the specific language governing permissions and
 14   limitations under the License. See accompanying LICENSE file.
 15 -->
 16 
 17 <!-- Put site-specific property overrides in this file. -->
 18 
 19 <configuration>
 20 <property>
 21   <!-- 开几个备份 -->
 22     <name>dfs.replication</name>
 23     <value>3</value>
 24   </property>
 25   <property>
 26     <name>dfs.namenode.name.dir</name>
 27     <value>file:///data/hadoop_data/hdfs/name</value>
 28   </property>
 29   <property>
 30     <name>dfs.datanode.data.dir</name>
 31     <value>/data/hadoop_data/hdfs/data/disk1</value>
 32         <final>true</final>
 33   </property>  
 34   <property>
 35     <name>dfs.nameservices</name>
 36     <value>ns</value>
 37   </property>
 38   <property>
 39     <name>dfs.ha.namenodes.ns</name>
 40     <value>node1,node2</value>
 41   </property>
 42   <property>
 43     <name>dfs.namenode.rpc-address.ns.node1</name>
 44     <value>node1:8020</value>
 45   </property>
 46   <property>
 47     <name>dfs.namenode.rpc-address.ns.node2</name>
 48     <value>node2:8020</value>
 49   </property>
 50   <property>
 51     <name>dfs.namenode.http-address.ns.node1</name>
 52     <value>node1:50070</value>
 53   </property>
 54   <property>
 55     <name>dfs.namenode.http-address.ns.node2</name>
 56     <value>node2:50070</value>
 57   </property>
 58   <property>
 59     <name>dfs.namenode.shared.edits.dir</name>
 60     <value>qjournal://node1:8485;node2:8485;node3:8485;node4:8485/ns</value>
 61   </property>
 62   <property>
 63     <name>dfs.client.failover.proxy.provider.ns</name>
 64     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
 65   </property>
 66   <property>
 67     <name>dfs.ha.fencing.methods</name>
 68     <value>sshfence</value>
 69   </property>
 70   <property>
 71     <name>dfs.ha.fencing.ssh.private-key-files</name>
 72     <value>/home/soft/.ssh/id_rsa</value>
 73   </property>
 74   <property>
 75     <name>dfs.journalnode.edits.dir</name>
 76     <value>/data/hadoop_data/journal</value>
 77   </property>
 78 <!--开启基于Zookeeper及ZKFC进程的自动备援设置,监视进程是否死掉 -->
 79     <property>
 80         <name>dfs.ha.automatic-failover.enabled</name>
 81         <value>true</value>
 82     </property>
 83     <property>
 84         <name>ha.zookeeper.quorum</name>
 85         <value>node1:2181,node2:2181,node3:2181,node4:2181</value>
 86     </property>
 87     <property>
 88         <!--指定ZooKeeper超时间隔，单位毫秒 -->
 89         <name>ha.zookeeper.session-timeout.ms</name>
 90         <value>2000</value>
 91     </property>
 92 <property>
 93     <name>fs.hdfs.impl.disable.cache</name>
 94     <value>true</value>
 95 </property>
 96 <property>
 97   <name>dfs.webhdfs.enabled</name>
 98   <value>true</value>
 99 </property>
100 </configuration>

View Code

#编辑mapred-site.xml

 1 <?xml version="1.0"?>
 2 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 3 <!--
 4 Licensed under the Apache License, Version 2.0 (the "License");
 5 you may not use this file except in compliance with the License.
 6 You may obtain a copy of the License at
 7 
 8 http://www.apache.org/licenses/LICENSE-2.0
 9 
10 Unless required by applicable law or agreed to in writing, software
11 distributed under the License is distributed on an "AS IS" BASIS,
12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 See the License for the specific language governing permissions and
14 limitations under the License. See accompanying LICENSE file.
15 -->
16 
17 <!-- Put site-specific property overrides in this file. -->
18 
19 <configuration>
20 <property>
21 <name>mapreduce.framework.name</name>
22 <value>yarn</value>
23 </property>
24 <property>
25 <name>mapreduce.map.memory.mb</name>
26 <value>8000</value>
27 </property>
28 <property>
29 <name>mapreduce.reduce.memory.mb</name>
30 <value>8000</value>
31 </property>
32 <property>
33 <name>mapreduce.map.java.opts</name>
34 <value>-Xmx8000m</value>
35 </property>
36 <property>
37 <name>mapreduce.reduce.java.opts</name>
38 <value>-Xmx8000m</value>
39 </property>
40 <property>
41 <name>mapred.task.timeout</name>
42 <value>1800000</value> <!-- 30 minutes -->
43 </property>
44 <property> 
45 <name>mapreduce.jobhistory.webapp.address</name> 
46 <value>node1:19888</value> 
47 </property>
48 <property>
49 <name>mapreduce.jobhistory.address</name>
50 <value>node1:10020</value>
51 </property>
52 </configuration>

View Code

#编辑yarn-site.xml

 1 <?xml version="1.0"?>
 2 <!--
 3   Licensed under the Apache License, Version 2.0 (the "License");
 4   you may not use this file except in compliance with the License.
 5   You may obtain a copy of the License at
 6 
 7     http://www.apache.org/licenses/LICENSE-2.0
 8 
 9   Unless required by applicable law or agreed to in writing, software
10   distributed under the License is distributed on an "AS IS" BASIS,
11   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12   See the License for the specific language governing permissions and
13   limitations under the License. See accompanying LICENSE file.
14 -->
15 <configuration>
16     <property>
17         <name>yarn.resourcemanager.connect.retry-interval.ms</name>
18         <value>2000</value>
19     </property>
20     <property>
21         <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
22         <value>true</value>
23     </property>
24 <!-- 开启RM高可靠 -->
25         <property>
26            <name>yarn.resourcemanager.ha.enabled</name>
27            <value>true</value>
28         </property>
29         <!-- 指定RM的cluster id -->
30         <property>
31            <name>yarn.resourcemanager.cluster-id</name>
32            <value>yrc</value>
33         </property>
34         <!-- 指定RM的名字 -->
35         <property>
36            <name>yarn.resourcemanager.ha.rm-ids</name>
37            <value>rm1,rm2</value>
38         </property>
39         <!-- 分别指定RM的地址 -->
40         <property>
41            <name>yarn.resourcemanager.hostname.rm1</name>
42            <value>node1</value>
43         </property>
44         <property>
45            <name>yarn.resourcemanager.hostname.rm2</name>
46            <value>node2</value>
47         </property>
48         <property>
49                 <name>yarn.resourcemanager.ha.id</name>
50                 <value>rm1</value>
51         </property>
52 <!-- 指定zk集群地址 -->
53   <property>
54     <name>yarn.resourcemanager.zk-address</name>
55     <value>node1:2181,node2:2181,node3:2181,node4:2181</value>
56   </property>
57   <property>
58     <name>yarn.nodemanager.aux-services</name>
59     <value>mapreduce_shuffle</value>
60   </property>
61 <property>
62     <name>yarn.log-aggregation-enable</name>
63     <value>true</value>
64   </property>
65 <property>
66     <name>yarn.log-aggregation.retain-seconds</name>
67     <value>86400</value>
68   </property>
69   <property>
70     <name>yarn.nodemanager.vmem-pmem-ratio</name>
71     <value>3</value>
72   </property>
73 <property>
74     <name>yarn.nodemanager.resource.memory-mb</name>
75     <value>57000</value>
76   </property>
77 <property>
78     <name>yarn.scheduler.minimum-allocation-mb</name>
79     <value>4000</value>
80   </property>
81 <property>
82     <name>yarn.scheduler.maximum-allocation-mb</name>
83     <value>10000</value>
84   </property>
85 <property>
86     <name>yarn.resourcemanager.webapp.address</name>
87     <value>node1:8088</value>
88 </property>
89 <property>
90     <name>yarn.web-proxy.address</name>
91     <value>node1:8888</value>
92 </property>
93 </configuration>

View Code

#编辑slaves文件

node1

node2

node3

node4

node5

vi .bashrc

.bashrc

1 export JAVA_HOME=/home/soft/app/jdk1.8.0_101
2 export CLASSPATH=.:$JAVA_HOME/lib/
3 export JRE_HOME=/home/soft/app/jdk1.8.0_101/jre
4 export ZOOKEEPER=/home/soft/app/zookeeper-3.4.8/
5 export  HADOOP_HOME=/home/soft/app/hadoop-2.7.3
6 export  HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
7 export  HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
8 export PATH=:$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:/sbin:$ZOOKEEPER/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:

#将hadoop目录scp到其他主机(.bashrc)

并分别执行source .bashrc

在node2上修改yarn-site.xml文件

property>

        <value>rm2</value>          #这里改成node的id，否则会报端口已占用的错误

<name>yarn.resourcemanager.ha.id</name>

</property><br><span style="color: #333333; font-family: "Microsoft Yahei"">./yarn-daemon.sh start resourcemanager</span>

六、启动

启动所有zookeeper(每台都必须启动)，查看状态是否成功。

zkServer.sh start　　#zookeeper启动

zkServer.sh stop　　#zookeeper停止

zkServer.sh status #zookeeper查看状态

启动journalnode(每台都必须启动)，jps查看状态是否成功。

hadoop-daemons.sh start journalnode #启动的命令

hadoop-daemons.sh stop journalnode #停止的命令

HDFS格式化（如果格式化失败请删除创建的/date目录下面的hadoop目录，然后再创建执行格式化命令）

#hadoop01上执行

hdfs namenode -format #namenode 格式化

hdfs zkfc -formatZK #格式化高可用

sbin/hadoop-daemon.sh start namenode #启动namenode

#备份节点执行

hdfs namenode -bootstrapStandby #同步主节点和备节点之间的元数据

停止所有journalnode，namenode

启动hdfs和yarn相关进程

./start-dfs.sh

./start-yarn.sh

或者

start-all.sh启动所有

stop-all.sh停止所有

备份节点手动启动resourcemanager

yarn-daemon.sh start resourcemanager

七、检查

[root@n1 ~]# jps　

19856 JournalNode
20049 DFSZKFailoverController
20130 ResourceManager
21060 Jps
20261 NodeManager
19655 DataNode
19532 NameNode
2061 QuorumPeerMain

hadoop@node1:~$ jps|wc -l
8

#浏览器访问

http://ip:50070

查看全文