zoukankan      html  css  js  c++  java
  • MapReduce分布式缓存程序,无法在Windows下的Eclipse中执行问题解决

    在写mapreduce程序中经常要用到hadoop自动的分布式缓存DistributedCache(新版本已经换新的API),但是在windows下Eclipse中执行,会出现类似如下错误:

    2016-03-03 10:53:21,424 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2016-03-03 10:53:22,152 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1019)) - session.id is deprecated. Instead, use dfs.metrics.session-id
    2016-03-03 10:53:22,152 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
    2016-03-03 10:53:24,366 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
    2016-03-03 10:53:26,447 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
    2016-03-03 10:53:26,487 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 4
    2016-03-03 10:53:30,876 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:4
    2016-03-03 10:53:31,065 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1862629830_0001
    2016-03-03 10:53:31,133 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/staging/root1862629830/.staging/job_local1862629830_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
    2016-03-03 10:53:31,142 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/staging/root1862629830/.staging/job_local1862629830_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
    2016-03-03 10:53:31,953 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(207)) - Creating symlink: mphadoop-rootmapredlocal1456973611218part-r-00003 <- F:JavaWork2014mapreduce/part-r-00003
    2016-03-03 10:53:32,004 WARN [main] fs.FileUtil (FileUtil.java:symLink(824)) - Fail to create symbolic links on Windows. The default security settings in Windows disallow non-elevated administrators and all non-administrators from creating symbolic links. This behavior can be changed in the Local Security Policy management console
    2016-03-03 10:53:32,004 WARN [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(209)) - Failed to create symlink: mphadoop-rootmapredlocal1456973611218part-r-00003 <- F:JavaWork2014mapreduce/part-r-00003
    2016-03-03 10:53:32,005 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(171)) - Localized hdfs://node1:8020/usr/output/weibo1/part-r-00003 as file:/tmp/hadoop-root/mapred/local/1456973611218/part-r-00003
    2016-03-03 10:53:32,012 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(207)) - Creating symlink: mphadoop-rootmapredlocal1456973611219part-r-00000 <- F:JavaWork2014mapreduce/part-r-00000
    2016-03-03 10:53:32,050 WARN [main] fs.FileUtil (FileUtil.java:symLink(824)) - Fail to create symbolic links on Windows. The default security settings in Windows disallow non-elevated administrators and all non-administrators from creating symbolic links. This behavior can be changed in the Local Security Policy management console
    2016-03-03 10:53:32,052 WARN [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(209)) - Failed to create symlink: mphadoop-rootmapredlocal1456973611219part-r-00000 <- F:JavaWork2014mapreduce/part-r-00000
    2016-03-03 10:53:32,052 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(171)) - Localized hdfs://node1:8020/usr/output/weibo2/part-r-00000 as file:/tmp/hadoop-root/mapred/local/1456973611219/part-r-00000
    2016-03-03 10:53:32,172 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1862629830_0001/job_local1862629830_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
    2016-03-03 10:53:32,177 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1862629830_0001/job_local1862629830_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
    2016-03-03 10:53:32,182 INFO [main] mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
    2016-03-03 10:53:32,183 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1862629830_0001
    2016-03-03 10:53:32,184 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
    2016-03-03 10:53:32,190 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    2016-03-03 10:53:32,236 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
    2016-03-03 10:53:32,237 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1862629830_0001_m_000000_0
    2016-03-03 10:53:32,262 INFO [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
    2016-03-03 10:53:32,306 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@da44a9a
    2016-03-03 10:53:32,310 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(733)) - Processing split: hdfs://node1:8020/usr/output/weibo1/part-r-00001:0+195718
    2016-03-03 10:53:32,319 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
    2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
    2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
    2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
    2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600

    ................................
    2016-03-03 10:53:32,614 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
    2016-03-03 10:53:32,626 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
    2016-03-03 10:53:32,734 WARN [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local1862629830_0001
    java.lang.Exception: java.io.FileNotFoundException: part-r-00003 (系统找不到指定的文件。)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
    Caused by: java.io.FileNotFoundException: part-r-00003 (系统找不到指定的文件。)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at java.io.FileReader.<init>(FileReader.java:58)
    at com.laoxiao.mr.tf.LastMapper.setup(LastMapper.java:49)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    2016-03-03 10:53:33,186 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Job job_local1862629830_0001 running in uber mode : false
    2016-03-03 10:53:33,188 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1362)) - map 0% reduce 0%
    2016-03-03 10:53:33,191 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1375)) - Job job_local1862629830_0001 failed with state FAILED due to: NA
    2016-03-03 10:53:33,197 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 0

    经过对错误的仔细分析后,发现问题出现在本地机器上无法把缓存中的数据拷贝的到本地磁盘,具体错误参考上面标红的文字

    解决方法

    1.直接把程序打成jar包,然后在Linux服务器下利用hadoop jar命令运行

    2.或者关闭windows7 UAC,然后就可以在Eclipse下本地测试模式跑mapreduce

    另外如果用上面的两种方法还出现类似如下错误:

    java.lang.Exception: java.io.FileNotFoundException: file: mphadoop-rootmapredlocal1456979002661 b_dim_city.dat (文件名、目录名或卷标语法不正确。)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
    Caused by: java.io.FileNotFoundException: file: mphadoop-rootmapredlocal1456979002661 b_dim_city.dat (文件名、目录名或卷标语法不正确。)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at java.io.FileReader.<init>(FileReader.java:58)
    at MapJoin.MapSideJoin_LeftOuterJoin$LeftOutJoinMapper.setup(MapSideJoin_LeftOuterJoin.java:100)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    那么可能的原因是hdfs上传的文件名后缀引起的,比如.dat之类的,可以把文件后缀删除后试试

  • 相关阅读:
    TCP/IP和HTTP的举例理解
    c#中栈和堆的理解
    c#设计模式之单例模式
    JSON.stringify实例应用—将对象转换成JSON类型进行AJAX异步传值
    JSON.stringify初识
    c# 过滤字符串中的重复字符
    C#中jQuery Ajax实例(二)
    C#中jQuery Ajax实例(一)
    jQuery动态对表格Table进行添加或删除行以及修改列值操作
    asp.net中控件的Attributes用法
  • 原文地址:https://www.cnblogs.com/QTHelper/p/5237949.html
Copyright © 2011-2022 走看看