zoukankan      html  css  js  c++  java
  • Hadoop- MapReduce在实际应用中常见的调优

    1、Reduce Task Number

    通常来说一个block就对应一个map任务进行处理,reduce任务如果人工不去设置干预的话就一个reduce。reduce任务的个数可以通过在程序中设置  job.setNumReduceTasks(个数); ,也可在配置文件上设置reduce任务个数,默认为1, 或者在代码config中配置

    Configuration configuration = new Configuration();
    configuration.set("mapreduce.job.reduces","2");//这个数字根据实际测试和调试来决定

    2、Map Task 输出压缩

    默认一个块对应一个map任务进行,没办法干预,那么就可以从输出的结果去优化,将结果压缩,如设置Map Task 输出压缩的格式:

    Configuration configuration = new Configuration();
    configuration.set("mapreduce.map.output.codec","org.apache.hadoop.io.compress.SnappyCodec")

    3、shuffle phase 参数

    mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles.
    mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1MB, which should minimize seeks.
    mapreduce.map.sort.spill.percent 0.80 The soft limit in the serialization buffer. Once reached, a thread will begin to spill the contents to disk in the background. Note that collection will not block if this threshold is exceeded while a spill is already in progress, so spills may be larger than this threshold when it is set to less than .5
    mapreduce.map.cpu.vcores 1 The number of virtual cores to request from the scheduler for each map task.
    mapreduce.reduce.memory.mb 1024 The amount of memory to request from the scheduler for each reduce task.
    mapreduce.reduce.cpu.vcores 1 The number of virtual cores to request from the scheduler for each reduce task.
  • 相关阅读:
    农民没有钱,怎样从种植养殖方面创业?
    有几万闲钱,投资什么好?
    2019适合创业的项目有哪些呢
    1981年的五块钱相当于现在多少钱的购买力?
    现阶段有50万资金,想去开一个店,从事什么行业好?
    投资40万左右适合做什么?
    阿里联手百联打造新零售,耗巨资10亿成立便利店
    90后,是被逼创业的,大家觉得呢?
    小本创业,具体怎么办?
    (三)调用web服务
  • 原文地址:https://www.cnblogs.com/RzCong/p/7676169.html
Copyright © 2011-2022 走看看