zoukankan      html  css  js  c++  java
  • Hadop使用Partitioner后,结果还是一个文件,怎样解决??

    近期看了一下partitioner。于是照着写了一个列子。最后发现程序并没有将结果分开写入对应的文件,结果还是一个文件,于是乎感觉是不是没实用集群去执行程序,发现control中还是本地执行的代码:

    <span style="font-size:12px;">2015-08-09 09:53:02,193 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hadoop-cau/mapred/staging/cau1745029252/.staging/job_local1745029252_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
    2015-08-09 09:53:02,195 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hadoop-cau/mapred/staging/cau1745029252/.staging/job_local1745029252_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
    2015-08-09 09:53:02,632 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hadoop-cau/mapred/local/localRunner/cau/job_local1745029252_0001/job_local1745029252_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
    2015-08-09 09:53:02,634 WARN  [main] conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/hadoop-cau/mapred/local/localRunner/cau/job_local1745029252_0001/job_local1745029252_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
    2015-08-09 09:53:02,657 INFO  [main] mapreduce.Job (Job.java:submit(1272)) - The url to track the job: http://localhost:8080/
    2015-08-09 09:53:02,659 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1317)) - Running job: job_local1745029252_0001
    2015-08-09 09:53:02,662 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(323)) - OutputCommitter set in config null
    2015-08-09 09:53:02,679 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(341)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    2015-08-09 09:53:02,810 INFO  [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:run(389)) - Waiting for map tasks</span>


    于是乎,想着打包到集群去执行看看,结果节点报了各种错!


    <span style="font-size:12px;">15/08/09 10:02:41 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1438842873940_0002/
    15/08/09 10:02:41 INFO mapreduce.Job: Running job: job_1438842873940_0002
    15/08/09 10:02:54 INFO mapreduce.Job: Job job_1438842873940_0002 running in uber mode : false
    15/08/09 10:02:54 INFO mapreduce.Job:  map 0% reduce 0%
    15/08/09 10:02:55 INFO mapreduce.Job: Task Id : attempt_1438842873940_0002_m_000000_0, Status : FAILED
    Container launch failed for container_1438842873940_0002_01_000002 : java.lang.IllegalArgumentException: java.net.UnknownHostException: slave7
    	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
    	at org.apache.hadoop.security.SecurityUtil.setTokenService(SecurityUtil.java:356)
    	at org.apache.hadoop.yarn.util.ConverterUtils.convertFromYarn(ConverterUtils.java:237)
    	at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:218)
    	at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtocolProxy.java:196)
    	at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    	at java.lang.Thread.run(Thread.java:722)
    Caused by: java.net.UnknownHostException: slave7</span>

    终于问题还是没解决,哪位大侠指点一下吧。

    小弟还没入门。在此叩谢!!!

  • 相关阅读:
    CV2图像操作
    Sobel边缘检测
    matlat之KDTreeSearcher()函数
    linux shell 将多行文件转换为一行
    (转)Shell脚本编程--Uniq命令
    (转)iptables简介
    (转)linux passwd批量修改用户密码
    (转)linux sort 命令详解
    (转)Linux命令之md5sum
    (转)shell实例浅谈之产生随机数七种方法
  • 原文地址:https://www.cnblogs.com/yangykaifa/p/6937829.html
Copyright © 2011-2022 走看看