zoukankan      html  css  js  c++  java
  • 统计 MapReduce 输出路径修改。



    1. 先在上一篇MR 的104 行加入代码。jobConf.setOutputFormat(MyMultipleFilesTextOutputFormat.class); 用意是自定义 job 的输出格式:
    2.         上一篇 MR 代码:
    3.         http://www.cnblogs.com/rocky24/p/f7a27b79fa8e5dfdc22fb535cadb86bc.html

    1. 1 继承 MultipleOutputFormat 实现抽象类的接口方法 getBaseRecordWriter 负责将键值对写入到文件系统。
    2. 2 重写 generateFileNameForKeyValue 方法。 定义不同的输出文件名。

      1. /**
      2. *
      3. */
      4. public static class MyMultipleFilesTextOutputFormat extends MultipleOutputFormat<Text, IntWritable> {
      5. private TextOutputFormat<Text, IntWritable> output = null;
      6. // 明确定义使用哪个 recordwriter类
      7. @Override
      8. protected org.apache.hadoop.mapred.RecordWriter<Text, IntWritable> getBaseRecordWriter(
      9. FileSystem fs, JobConf job, String name, Progressable progress)
      10. throws IOException {
      11. final TextOutputFormat<Text, IntWritable> textOutputFormat = new TextOutputFormat<Text, IntWritable>();
      12. if (output == null) {
      13. output = new TextOutputFormat<Text, IntWritable>();
      14. }
      15. return textOutputFormat.getRecordWriter(fs, job, name, progress);
      16. }
      17. // 重写方法, 将生成输出文件文件名的方法进行重写
      18. @Override
      19. protected String generateFileNameForKeyValue(Text key,IntWritable value, String name) {
      20. //输出的文件名就是k3的值
      21. final String keyString = key.toString();
      22. if(keyString.contains("download")) {
      23. return "download";
      24. } else if(keyString.contains("upload")) {
      25. return "upload";
      26. } else if(keyString.contains("debug")) {
      27. return "debug";
      28. } else {
      29. return "others";
      30. }
      31. }
      32. }





    God has given me a gift. Only one. I am the most complete fighter in the world. My whole life, I have trained. I must prove I am worthy of someting. rocky_24
  • 相关阅读:
    二叉树遍历
    NO.35 2021/12/13(06:50)[周一]
    NO.29 2021/11/30(06:30)[周二]
    NO.22 2021/11/19(06:15) [周五]
    The .NET ORM Architec
    C#格式字符串
    C# Attribute
    .net DLL反编译文件
    【Beta】Scrum meeting1
    【Alpha】Scrum meeting 6
  • 原文地址:https://www.cnblogs.com/rocky24/p/fd5054c9f2af692d999917b2e2d0d81f.html
Copyright © 2011-2022 走看看