zoukankan      html  css  js  c++  java
  • Why Sqls can be run via MapReduce?

      in 《Core Model of Sql》, I showed that there are three driven operations in SQL: Group, Filter, Join. In this article, I'll illustrate why sqls can be executed by a MapReduce engine, by showing that all the 3 operations can be implemented via MapReduce.  A typical MapReduce goes this way:

      Map:       each file row                                                -> (intermediary key, intermediary value)

      Reduce:  (intermediary key, List of intermediary value) -> (final key, final value)

    Now I show the 3 operations one by one.

    Group:

      Map:      each table row                                               ->(columns of "group by", other columns)

      Reduce: (columns of "group by", List of other columns)  ->(columns of "group by", apply aggregate function to "List of other columns")

    Filter:

      Map:      each table row -> if this row does not pass the where condition, then drop it

                                                  else (1, selected columns)

      Reduce:  No reduce is needed

    Join:

      Split tableA into N partitions, split tableB into M partitions. Then there are N*M input pairs

      For each input pair(part_i_A,part_j_B):

        for rowA in part_i_A:

          for rowB in part_j_B:

            generate (1, rowA+rowB)

  • 相关阅读:
    关于git修改和查看用户名邮箱
    Spring深入理解(三)
    Spring深入理解(二)
    Spring深入理解(一)
    jeecg开源快速开发
    关于面向对象的三大特性和五大基本原则
    关于Excel导入导出POI工具类
    关于Hanoi算法
    LOJ 530 最小倍数(数论)
    BZOJ 4242 水壶(BFS建图+最小生成树+树上倍增)
  • 原文地址:https://www.cnblogs.com/linghuaichong/p/4366996.html
Copyright © 2011-2022 走看看