zoukankan      html  css  js  c++  java
  • mapreduce 之 shuffle错误

    错误信息

    reduce容器报的错误信息如下:

    2020-07-01 14:06:19,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#78
    	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
    	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
    	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:422)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)
    Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
    	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:395)
    	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:310)
    	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:291)
    	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330)
    	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)
    

    其他日志信息

    2020-07-01 14:06:19,404 INFO [fetcher#60] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#60 - MergeManager returned status WAIT ...
    2020-07-01 14:06:19,404 INFO [fetcher#8] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx118:13562 freed by fetcher#8 in 0ms
    2020-07-01 14:06:19,404 INFO [fetcher#60] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx162:13562 freed by fetcher#60 in 0ms
    2020-07-01 14:06:19,404 INFO [fetcher#6] org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#6 - MergeManager returned status WAIT ...
    2020-07-01 14:06:19,404 INFO [fetcher#6] org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl: xxx144:13562 freed by fetcher#6 in 2ms
    

    从信息可以看出来,错误的原因是由于reduce从map拷贝数据的过程当中失败的,并且还是在merge阶段.

    解决办法:

    修改mapred的参数

    mapreduce.reduce.shuffle.memory.limit.percent=0.1 # 默认是0.25 可以在代码里面设置,也可以在mapred-site.xml文件配置,这个参数的意思是说reduce每次fetceh数据的时候,达到reduce jvm内存的百分之多少的时候,就把数据写入到磁盘
    
  • 相关阅读:
    (网页)中的简单的遮罩层
    (后端)shiro:Wildcard string cannot be null or empty. Make sure permission strings are properly formatted.
    (网页)jQuery的时间datetime控件在AngularJs中使用实例
    Maven Myeclipse 搭建项目
    MyBatis 环境搭建 (一)
    java 常用方法
    XML 基础
    JS BOM
    js 事件
    js 的使用原则
  • 原文地址:https://www.cnblogs.com/yjt1993/p/13219447.html
Copyright © 2011-2022 走看看