hadoop操作

常用命令：

https://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html

ls命令

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://machine:9000/home/cloudops/jia/v0001/0001/src/20151230211015.0030381.rank.0001

touch命令

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -touch hdfs://machine:9000/home/cloudops/jia/mytest 错误示例
/usr/bin/hadoop/software/hadoop/bin/hadoop fs -touchz hdfs://machine:9000/home/cloudops/jia/mytest 正确示例

mv命令

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -mv hdfs://machine:9000/home/cloudops/jia/tmp4/* hdfs://machine:9000/home/cloudops/jia/tmp3/

mkdir命令

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -mkdir hdfs://machine:9000/home/cloudops/jia/tmp4/tmp4.2

拷贝

本地拷贝到远程
/usr/bin/hadoop/software/hadoop/bin/hadoop fs -copyFromLocal /home/hdp-jia/a hdfs://w-namenode1v:9000/home/cloudops/jia/tmp3/

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -copyToLocal hdfs://w-namenode1v:9000/home/cloudops/jia/tmp3/ {localDir}

删除

/usr/bin/hadoop/software/hadoop/bin/hadoop fs -rm -skipTrash hdfs://w-namenode1v:9000/home/cloudops/jia/mytest

更多的shell语法：http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_shell.html

场景1：删除失败，再删除

1，创建；2，查看；3，删除；4，查看，5，删除，6，查看
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -touchz hdfs://w-namenode1v:9000/home/cloudops/jia/mytest
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://w-namenode1v:9000/home/cloudops/jia/my*
Found 1 items
-rw-r--r--   3 hdp-jia cloudops          0 2015-12-30 21:19 /home/cloudops/jia/mytest
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -rm hdfs://w-namenode1v:9000/home/cloudops/jia/mytest
15/12/30 21:22:30 WARN fs.TrashPolicyDefault: Can't create trash directory: hdfsold://w-namenode1v:9000/user/hdp-jia/.Trash/Current/home/cloudops/jia
Problem with Trash.org.apache.hadoop.security.AccessControlException: Permission denied: user=hdp-jia, access=WRITE, inode="user":work:supergroup:rwxr-xr-x. Consider using -skipTrash option
rm: Failed to move to trash: /home/cloudops/jia/mytest
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://w-namenode1v:9000/home/cloudops/jia/my*
Found 1 items
-rw-r--r--   3 hdp-jia cloudops          0 2015-12-30 21:19 /home/cloudops/jia/mytest
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -rm -skipTrash hdfs://w-namenode1vt:9000/home/cloudops/jia/mytest
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://w-namenode1v:9000/home/cloudops/jia/my*
ls: Cannot access hdfs://w-namenode1v:9000/home/cloudops/jia/my*: No such file or directory.

场景2：移动文件夹（如果源文件夹下面有A目录，目标文件夹下已经有A目录了，则mv失败；如果目标文件夹下没有A目录，则mv成功）

[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://w-namenode1v:9000/home/cloudops/jia/tmp3/
Found 3 items
-rw-r--r-- 3 hdp-jia cloudops 14375 2015-12-30 20:41 /home/cloudops/jia/tmp3/a
drwxr-xr-x - hdp-jia cloudops 0 2015-12-30 20:42 /home/cloudops/jia/tmp3/tmp4.1
drwxr-xr-x - hdp-jia cloudops 0 2015-12-30 20:54 /home/cloudops/jia/tmp3/tmp4.2
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -ls hdfs://w-namenode1v:9000/home/cloudops/jia/tmp4/
Found 1 items
drwxr-xr-x - hdp-jia cloudops 0 2015-12-30 20:56 /home/cloudops/jia/tmp4/tmp4.2
[hdp-jia@client90v ~]$ /usr/bin/hadoop/software/hadoop/bin/hadoop fs -mv hdfs://w-namenode1v:9000/home/cloudops/jia/tmp4/* hdfs://w-namenode1v:9000/home/cloudops/jia/tmp3/
mv: Failed to rename hdfs://w-namenode1v:9000/home/cloudops/jia/tmp4/tmp4.2 to hdfs://w-namenode1v:9000/home/cloudops/jia/tmp3