zoukankan      html  css  js  c++  java
  • spark sortBy&&sortByKey

    sortByKey和sortBy都是transforamation算子;

    sortByKey   源码如下:

      def sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length)
          : RDD[(K, V)] = self.withScope
      {
        val part = new RangePartitioner(numPartitions, self, ascending)
        new ShuffledRDD[K, V, V](self, part)
          .setKeyOrdering(if (ascending) ordering else ordering.reverse)
      }

    参数一:ascending 默认是True正序排列

    参数二:分区数

    即可以对标准RDD进行排序,也可以按照指定指定键值对pair Rdd进行排序

    sortBy   源码如下

    def sortBy[K](
          f: (T) => K,
          ascending: Boolean = true,
          numPartitions: Int = this.partitions.length)
          (implicit ord: Ordering[K], ctag: ClassTag[K]): RDD[T] = withScope {
        this.keyBy[K](f)
            .sortByKey(ascending, numPartitions)
            .values
      }

    参数一: 输入一个匿名函数,匿名函数的返回值是排序的关键字

    参数二:ascending 默认是True正序排列

    参数三:分区数

    案例对wordcount结果进行排序

    数据

    hadoop hadoop hadoop
    spark spark spark spark spark
    hadoop hadoop hadoop
    spark spark spark spark spark
    hive hive hive hive
    hadoop hadoop hadoop
    spark spark spark spark spark
    hadoop hadoop hadoop
    spark spark spark spark spark
    apache apache apache apache
    hadoop hadoop hadoop
    spark spark spark spark spark
    hive hive hive hive
    hadoop hadoop hadoop
    spark spark spark spark spark
    apache apache apache apache

    sortByKey

    val data:RDD[String]=spark.sparkContext.textFile(datapath,2)
    data.flatMap(x=>x.split(" ")).map((_,1)).reduceByKey(_+_).sortByKey(false).collect.foreach(println(_))

    sortBykey

    val data:RDD[String]=spark.sparkContext.textFile(datapath,2)
    data.flatMap(x=>x.split(" ")).map((_,1)).reduceByKey(_+_).sortBy(x=>x._1,false).collect.foreach(println(_))

    结果

    (spark,30)
    (hive,8)
    (hadoop,18)
    (apache,8)
  • 相关阅读:
    Codeforces Round #498 (Div. 3) E. Military Problem
    codeforces ~ 1009 B Minimum Ternary String(超级恶心的思维题
    二叉排序树
    codeforces ~ 1004 C Sonya and Robots (dp)
    fragment shader的优化
    计算带宽
    trilinear filter
    GPU bubbles
    Dx12 occlusion query
    非意外的PDB错误 OK(0)
  • 原文地址:https://www.cnblogs.com/students/p/14230930.html
Copyright © 2011-2022 走看看