一丶 脚本命令
1. xsync: 同步三台主机的文件 修改一台文件后 使用该命令 可以实现多台的同步修改
使用方法 xsync +文件路径 +目标文件路径
#!/bin/bash
#1. 判断参数
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. 遍历集群所有机器
for host in hadoop102 hadoop103 hadoop104
do
echo ================= $host ================
#3. 遍历所有目录,挨个发送
for file in $@
do
#4 判断文件是否存在
if [ -e $file ]
then
#5. 获取父目录
pdir=$(cd -P $(dirname $file); pwd)
#6. 获取当前文件名称
fname=$(basename $file)
ssh $host "mkdir -p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done
2. mycluster.sh: 群起或者群停 Hadoop 结点
使用 : mycluster.sh start/stop
#!/bin/bash
if [ $# -lt 1 ]
then
echo "Input Args Error!!!!"
exit
fi
case $1 in
start)
echo "========== START HDFS =========="
ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/start-dfs.sh
echo "========== START YARN =========="
ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/start-yarn.sh
;;
stop)
echo "========== STOP YARN =========="
ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/stop-yarn.sh
echo "========== STOP HDFS =========="
ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/stop-dfs.sh
;;
*)
echo "Input Args Error!!!!"
;;
esac
3. myjps: 查看三个结点的数据
执行操作:myjps.sh
#!/bin/bash
if [ $# -lt 1 ]
then
echo "Input Args Error!!!!"
exit
fi
case $1 in
start)
echo "========== START HDFS =========="
ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/start-dfs.sh
echo "========== START YARN =========="
ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/start-yarn.sh
;;
stop)
echo "========== STOP YARN =========="
ssh hadoop103 /opt/module/hadoop-3.1.3/sbin/stop-yarn.sh
echo "========== STOP HDFS =========="
ssh hadoop102 /opt/module/hadoop-3.1.3/sbin/stop-dfs.sh
;;
*)
echo "Input Args Error!!!!"
;;
esac
[atguigu@hadoop102 bin]$ cat myjps.sh
#!/bin/bash
for i in hadoop102 hadoop103 hadoop104
do
echo "========== $i jps =========="
ssh $i /opt/module/jdk1.8.0_212/bin/jps
done
二丶HDFS 操作简记
1. hadoop fs -help 帮助命令: 列出所有命令帮助信息
hadoop fs -help (具体命令) 可以看到具体命令帮助信息
2.hadoop fs -ls [hdfs当中的目录]: 显示目录下的所有文件和文件的权限
hadoop fs -ls -R [hdfs当中的目录]: 显示目录下的所有文件
3. hadoop fs -cat [hdfs 文件路径] : 查看文件内容
4.hadoop fs -mkdir [hdfs 存储路径] : 创建目录
hadoop fs -mkdir -p [hdfs 存储路径] : 创建多级目录
5.hadoop fs -rm [文件路径] :删除文件
hadoop fs -rm -r [文件路径] :删除文件夹
6.hadoop fs -put [文件路径]: 复制文件
7. hadoop fs -cp [URI] [URI]: 将文件从源路径复制到目标路径
三丶IDEA 连接Hadoop 需要的Moven项目
<dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <version>2.12.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>3.1.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-common</artifactId> <version>3.1.3</version> </dependency> </dependencies>