zoukankan      html  css  js  c++  java
  • Mapreduce报错:Split metadata size exceeded 10000000

    报错信息:

    Failure Info:Job initialization failed: java.io.IOException: Split metadata size exceeded 10000000.   
    Aborting job job_201205162059_1073852 at   
    org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:48) at   
    org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:817) at   
    org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:711) at   
    org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4028) at   
    org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) at   
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at   
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)   

    出错原因:该job的job.splitmetainfo文件大小超过限制;

    1. job.splitmetainfo,该文件记录split的元数据信息,job split ----> HDFS block && slave node

      存放路径位于:${hadoop.tmp.dir}/mapred/staging/${user.name}/.staging/jobId/

    2. 参数mapreduce.jobtracker.split.metainfo.maxsize控制该文件的最大大小,默认为:10000000(10M)

  • 相关阅读:
    单点登录原理与简单实现
    关系型数据库中的关键字、主关键字和候选关键字
    无向图的顶点连通度
    memcmp()直接比较两个数组的大小
    静态字典树
    动态字典树
    poj 1149
    poj 2112 floyd+Dinic最大流+二分最小值
    POJ 1698 (二分图的多重匹配)
    网络流算法
  • 原文地址:https://www.cnblogs.com/wangxiaowei/p/3343298.html
Copyright © 2011-2022 走看看