zoukankan      html  css  js  c++  java
  • Spark获取DataFrame中列的几种姿势--col,$,column,apply

    1.doc上的解释(https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Column.html)
     df("columnName")            // On a specific DataFrame.
       col("columnName")           // A generic column no yet associated with a DataFrame.
       col("columnName.field")     // Extracting a struct field
       col("`a.column.with.dots`") // Escape `.` in column names.
       $"columnName"               // Scala short hand for a named column.
       expr("a + 1")               // A column that is constructed from a parsed SQL Expression.
       lit("abc")                  // A column that produces a literal (constant) value.

    2.使用时涉及到的的包

       import spark.implicits._
       import org.apache.spark.sql.functions._
       import org.apache.spark.sql.Column
    3.示例
    scala> val idCol = $"id"
    idCol: org.apache.spark.sql.ColumnName = id
     
    scala> val idCol = col("id")
    idCol: org.apache.spark.sql.Column = id
     
    scala> val idCol = column("id")
    idCol: org.apache.spark.sql.Column = id
    scala> val dataset = spark.range(5).toDF("text")
    dataset: org.apache.spark.sql.DataFrame = [text: bigint]
     
    scala> val textCol = dataset.col("text")
    textCol: org.apache.spark.sql.Column = text
     
    scala> val textCol = dataset.apply("text")
    textCol: org.apache.spark.sql.Column = text
     
    scala> val textCol = dataset("text")
    textCol: org.apache.spark.sql.Column = text
  • 相关阅读:
    ViZDoom深度预测(Depth Prediction)
    刨根问底U3D---从Profile中窥探Unity的内存管理
    关于Android真机调测Profiler
    初探Stage3D(三) 深入研究透视投影矩阵
    初探Stage3D(二) 了解AGAL
    初探Stage3D(一) 3D渲染基础原理
    unity3d优化总结篇
    Unity数据存储路径总结
    CREATE A ENERGY / HEALTH BAR HUD
    CREATE A LOADING SCENE / SPLASH SCREEN
  • 原文地址:https://www.cnblogs.com/itboys/p/10962665.html
Copyright © 2011-2022 走看看