zoukankan      html  css  js  c++  java
  • mapreduce中reduce没有执行

     hadoop执行mapreduce过程reduce不执行原因

    1.如果你的map过程中没有context.write()是不执行reduce过程的;
    2.如果你的map过程中context.write()的map后的的部分数据出现问题,
    不符合reduce接受的数据也会不执行reduce
    比如说你的日志文件中有一个空行是不符合reduce的接受数据reduce不执行;

     原:(会读到空行,不符合reduce的接受数据,reduce不执行)

    public static class Map extends Mapper<Object,Text,Text,IntWritable>{    //将输入输出作为string类型,对应Text类型
                private static Text newKey=new Text();    //每一行作为一个数据  
                public void map(Object key, Text value, Context context) throws IOException, InterruptedException{   
                    String line=value.toString();//转为字符串类型
                    System.out.println(line);
                        String arr[]=line.split(" ");//splite是按照输入的值拆分成数组
                        newKey.set(arr[0]);
                        int click=Integer.parseInt(arr[1]);
                        context.write(newKey,new IntWritable(click)); 
                        System.out.println(newKey+"  "+new IntWritable(click));                
                 } 
             }   

     加判断语句:

    public static class Map extends Mapper<Object,Text,Text,IntWritable>{    //将输入输出作为string类型,对应Text类型
                private static Text newKey=new Text();    //每一行作为一个数据  
                public void map(Object key, Text value, Context context) throws IOException, InterruptedException{   
                    String line=value.toString();//转为字符串类型
                    System.out.println(line);
                    if(!("".equals(line)))//增加控制语句,使得line为”“时能够停止。否则不符合reduce接受的数据不会执行reduce
                    {
                        String arr[]=line.split(" ");//splite是按照输入的值拆分成数组
                        newKey.set(arr[0]);
                        int click=Integer.parseInt(arr[1]);
                        context.write(newKey,new IntWritable(click)); 
                        System.out.println(newKey+"  "+new IntWritable(click));
                    }
                 } 
             }   
  • 相关阅读:
    广域网(ppp协议、HDLC协议)
    0120. Triangle (M)
    0589. N-ary Tree Preorder Traversal (E)
    0377. Combination Sum IV (M)
    1074. Number of Submatrices That Sum to Target (H)
    1209. Remove All Adjacent Duplicates in String II (M)
    0509. Fibonacci Number (E)
    0086. Partition List (M)
    0667. Beautiful Arrangement II (M)
    1302. Deepest Leaves Sum (M)
  • 原文地址:https://www.cnblogs.com/sengzhao666/p/11808268.html
Copyright © 2011-2022 走看看