zoukankan      html  css  js  c++  java
  • Hadoop- MapReduce在实际应用中常见的调优

    1、Reduce Task Number

    通常来说一个block就对应一个map任务进行处理,reduce任务如果人工不去设置干预的话就一个reduce。reduce任务的个数可以通过在程序中设置  job.setNumReduceTasks(个数); ,也可在配置文件上设置reduce任务个数,默认为1, 或者在代码config中配置

    Configuration configuration = new Configuration();
    configuration.set("mapreduce.job.reduces","2");//这个数字根据实际测试和调试来决定

    2、Map Task 输出压缩

    默认一个块对应一个map任务进行,没办法干预,那么就可以从输出的结果去优化,将结果压缩,如设置Map Task 输出压缩的格式:

    Configuration configuration = new Configuration();
    configuration.set("mapreduce.map.output.codec","org.apache.hadoop.io.compress.SnappyCodec")

    3、shuffle phase 参数

    mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles.
    mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks.
    mapreduce.map.sort.spill.percent 0.80 The soft limit in the serialization buffer. Once reached, a thread will begin to spill the contents to disk in the background. Note that collection will not block if this threshold is exceeded while a spill is already in progress, so spills may be larger than this threshold when it is set to less than .5
    mapreduce.map.cpu.vcores 1 The number of virtual cores to request from the scheduler for each map task.
    mapreduce.reduce.memory.mb 1024 The amount of memory to request from the scheduler for each reduce task.
    mapreduce.reduce.cpu.vcores 1 The number of virtual cores to request from the scheduler for each reduce task.
  • 相关阅读:
    poj 3277 City Horizon (线段树 扫描线 矩形面积并)
    HDU 1255 覆盖的面积 (扫描线 线段树 离散化 矩形面积并)
    Codeforces Round #260 (Div. 2)
    poj 1151 Atlantis (离散化 + 扫描线 + 线段树 矩形面积并)
    CF1237F Balanced Domino Placements
    CF954H Path Counting
    AT2395 [ARC071C] TrBBnsformBBtion
    AT2400 [ARC072B] Alice&Brown
    AT2401 [ARC072C] Alice in linear land
    [国家集训队]阿狸和桃子的游戏
  • 原文地址:https://www.cnblogs.com/RzCong/p/7676169.html
Copyright © 2011-2022 走看看