1、执行脚本程序报如下所示的错误:
1 [hadoop@slaver1 script_hadoop]$ hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree /home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 /home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19 2 18/05/20 17:09:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 3 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C 4 at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) 5 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 6 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 7 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 8 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 9 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 10 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) 11 at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498) 12 at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207) 13 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 14 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 15 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 16 at java.lang.reflect.Method.invoke(Method.java:606) 17 at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 18 [hadoop@slaver1 script_hadoop]$ hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19 19 18/05/20 17:10:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C 21 at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) 22 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 23 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 24 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 25 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 26 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 27 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) 28 at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498) 29 at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207) 30 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 31 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 32 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 33 at java.lang.reflect.Method.invoke(Method.java:606) 34 at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 35 [hadoop@slaver1 script_hadoop]$
2、猛一看,WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable,百度一搜这个错误,啥编译32/64位的问题,也有说改配置文件的问题,也许有的可能是这样改就解决问题了,而我想说,出现问题,看看报错原因,也许就可以很快的解决问题了。
然后呢,看下代码,是如下所示导致的错误,将下面的注释了,代码换成上面的就可以使用脚本程序执行代码了:
FileInputFormat.setInputPaths(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); //FileInputFormat.setInputPaths(job, new Path("C:/weblog/output")); //FileOutputFormat.setOutputPath(job, new Path("c:/weblog/pageviews"));