bigdata learning unit one--Hadoop environment setting

1、配置ssh，使集群服务器之间的通讯，不再每次都输入密码进行认证。

2、

[root@hc--uatbeta2 hadoop]# start-all.sh
Starting namenodes on [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [hc--uatbeta2.novalocal]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-08-07 06:56:20,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
[root@hc--uatbeta2 hadoop]#

解决：把所有环节变量都加上   vi hadoop-env.sh

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

--------

[root@hc--uatbeta2 hadoop]# vi core-site.xml
[root@hc--uatbeta2 hadoop]# vi mapred-site.xml
[root@hc--uatbeta2 hadoop]# vi yarn-site.xml

vi hdfs-site.xml

----将hostname 改为IP-------  改为IP也不行，得修改hosts：vi /etc/hosts

2、格式化namenode,

#CD /usr/hadoop/hadoop-3.0.0
# ./bin/hdfs namenode -format

这个报错：

出现未知的主机名的问题，异常信息如下所示：

 [shirdrn@localhost bin]$ hadoop namenode -format
11/06/22 07:33:31 INFO namenode.NameNode: STARTUP_MSG: 

解决方案还是修改hosts；

10.167.202.135:50070  

10.167.202.135:8088

10.167.202.135:50070  不能访问的话

vi  /etc/selinux/config 

修改
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
 
为
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled

　　

设置默认访问端口

mapred-site.xml 添加下面两个
<property>
  <name>mapred.job.tracker.http.address</name>
  <value>0.0.0.0:50030</value>
</property>
<property>
  <name>mapred.task.tracker.http.address</name>
  <value>0.0.0.0:50060</value>
</property>

hdfs-site.xml 添加下面配置
<property>
  <name>dfs.http.address</name>
  <value>0.0.0.0:50070</value>
</property>
然后停止所有进程，删除name、data文件夹下数据重新格式化，重新启动后访问正常

　　

 测试hdfs 是否可以使用：

1、上传文件：hadoop fs -put /root/服务器地址.java hdfs://localhost:9000/    localhost：9000 配置在hadoop-env.sh

2、查看文件：hadoop fs -ls /

3、删除文件：hadoop  fs  -rm hdfs://10.167.202.135:9000/服务器地址.java