zoukankan      html  css  js  c++  java
  • spark中的shuffle算子

    官网的话
    什么是Shuffle

    In Spark, data is generally not distributed across partitions to be in the necessary place for a specific operation.
    During computations, a single task will operate on a single partition - thus,
    to organize all the data for a single reduceByKey reduce task to execute,
    Spark needs to perform an all-to-all operation. It must read from all partitions to find all the values for all keys,
    and then bring together values across partitions to compute the final result for each key - this is called the shuffle.

    我直接复制了整段话,其实用概括起来就是:

    把不同节点的数据拉取到同一个节点的过程就叫做Shuffle

    有哪些Shuffle算子
    Operations which can cause a shuffle include
    repartition operations like repartition and coalesce,
    ‘ByKey operations (except for counting) like groupByKey and reduceByKey,
    and join operations like cogroup and join.


    这一句话完美总结了Spark中Shuffle算子的分类:

    重分区算子
    (repartition ,coalesce)
    ByKey算子
    (groupByKey ,reduceByKey)
    Join算子
    (cogroup ,join)
    详细总结三类Shuffle算子
    其实官网写那几个就是最常用的了

    重分区算子
    repartition
    coalesce


    ByKey算子
    groupByKey
    reduceByKey
    aggregateByKey
    combineByKey
    sortByKey
    sortBy


    Join算子
    cogroup
    join
    leftOuterJoin
    intersection
    subtract
    subtractByKey
    (姑且把后面三个也放到Join类算子)
    后记
    官网说了三类,这里再加一类:

    去重算子
    distinct

    原文链接:https://blog.csdn.net/Android_xue/article/details/102806676

  • 相关阅读:
    性能优化之
    gruntjs开发实例
    从数组里随机获取N项
    调试工具-fiddler:本地资源替换线上调试
    HTML5-canvas实例:2D折线数据图与2D扇形图
    移动前端兼容性笔记
    Less开发指南(三)- 代码文件跟踪调试
    像纸质笔记本一样给div,textarea添加行的分割线
    基于视图的增删改查操作(颠覆传统思维吧)
    恶劣的百度推广人员
  • 原文地址:https://www.cnblogs.com/guoyu1/p/14456245.html
Copyright © 2011-2022 走看看