zoukankan      html  css  js  c++  java
  • Partitioning, Shuffle and sort

    Partitioning, Shuffle and sort  what happened?

    - Partitioning
    Partitioning is the process of determining which reducer instance will receive which intermediate keys and values. Each mapper must determine for all of its output (key, value) pairs which reducer will receive them. It is necessary that for any key, regardless of which mapper instance generated it, the destination partition is the same
    Problem: how dose the hadoop make it? Use a hash function ? what is the function? 
    here is code~
    1 public class HashPartitioner<K, V> extends Partitioner<K, V> {
    2     public int getPartition(K key, V value, int numReduceTasks) {
    3         return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
    4     }
    5 }

    解释:将key均匀分布在ReduceTasks上,举例如果Key为Text的话,Text的hashcode方法跟String的基本一致,都是采用的Horner公式计算,得到一个int,string太大的话这个int值可能会溢出变成负数,所以与上Integer.MAX_VALUE(即0111111111111111),然后再对reduce个数取余,这样就可以让相同key分布在一个节点上,并且较为均匀的分布在reduce上

    Horner规则:算法导论上有介绍这个,百度之

    think about BloomFilter~ 保证这个任务任务分发的均匀是关键,所以要设计优秀的hash函数是关键

    - Shuffle
    After the first map tasks have completed, the nodes may still be performing several more map tasks each. But they also begin exchanging the intermediate outputs from the map tasks to where they are required by the reducers. This process of moving map outputs to the reducers is known as shuffling.
     
    - Sort
    Each reduce task is responsible for reducing the values associated with several intermediate keys. The set of intermediate keys on a single node is automatically sorted by Hadoop before they are presented to the Reducer 
  • 相关阅读:
    awk去重以某列重复的行
    awk 统计文件中按照某列统计某列的和(sum)
    使用jdk压缩war包
    histoty显示时间戳
    awk统计文件中某关键词出现次数
    Jbox帮助文档,默认的属性含义
    net之session漫谈及分布式session解决方案
    StackExchange.Redis 基本使用 (一) (转)
    Sql Server 表创建以及Ef浅谈
    数据验证(自定义特性)
  • 原文地址:https://www.cnblogs.com/cherri/p/3286294.html
Copyright © 2011-2022 走看看