zoukankan      html  css  js  c++  java
  • mapreduce新旧api对比

    对比:hadoop版本1.x 新版,hadoop版本0.x 旧版

    1.新api引用包一般是mapreduce ,旧版api引用的包一般是mapred

    2.新api使用Job,旧版api使用JobConf

    3.新api类的包名使用mapreduce,旧版api使用mapred

    4.新api使用job.waitForCompletion(true)提交作业,旧版api使用JobClient.runJob(job);

    5.新api:extends Mapper,旧版api:extends MapRedcueBase implements Mapper

    6.新api:使用上下文Context ctx ctx.write(),旧版api:使用OutputCollector<>collector Reporter  collector.collect();

    -------------------------

    新版:

    public class kpi extends Configured implements Tool

    {

    static class MyMapper extends Mapper<LongWritable, Text, Text, kpiwritable>

    {

    protected void map(LongWritable k1, Text v1, Context context)

    context.write(k2, v2);

    }

    static class MyReducer extends Reducer<Text, kpiwritable, Text, kpiwritable>

    {

    protected void reduce(Text k2, java.lang.Iterable<kpiwritable> v2s, Context ctx)

    context.write(k2, v2);

    }

    public int run(String[] arg0)

    {Job job=new Job(conf,"kpi");

    job.waitForCompletion(true);
    return 0;

    }

    public static void main(String[] args) throws Exception

    {
    ToolRunner.run(new kpi(), args);
    }

    }

    ---------------------------------------------------------

    旧版:

    public class old_api

    {

    static class MyMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, LongWritable>{

    public void map(LongWritable k1, Text v1,
    OutputCollector<Text, LongWritable> collector, Reporter reporter){

    collector.collect(new Text(word), new LongWritable(1));

    }

    }

    static class MyReducer extends MapReduceBase implements Reducer<Text, LongWritable, Text, LongWritable>{

    public void reduce(Text k2, Iterator<LongWritable> v2s,
    OutputCollector<Text, LongWritable> collector, Reporter reporter){

    collector.collect(k2, new LongWritable(times));

    }

    }

    public static void main(String[] args) {

    final JobConf job = new JobConf(conf , old_api.class);

    JobClient.runJob(job);

    }

    }

  • 相关阅读:
    通过Navicat导入SQLServer的MDF文件和LDF文件
    ubantu系统出现登录界面死循环处理办法
    ubantu系统修改权限失败,导致只能客人会话登录解决办法
    redis基础
    ubantu安装MySQL,并未出现设置root密码的提示--》少年,请不要乱改密码!
    ngx_http_access_module模块说明
    一些常用的ngx_http_core_module介绍
    八、location匹配规则
    七、nginx的虚拟主机配置
    六、nginx的配置文件说明
  • 原文地址:https://www.cnblogs.com/mlj5288/p/4479265.html
Copyright © 2011-2022 走看看