zoukankan      html  css  js  c++  java
  • spark2.1操作json(save/read)

    建筑物配置信息:

    case class BuildingConfig(buildingid: String, building_height: Long, gridcount: Long, gis_display_name: String, wear_loss: Double, path_loss: Double) extends Serializable

    向hdfs写入json文件:

     sql(
          s"""|select buildingid,
              |height,
              |gridcount,
              |collect_list(gis_display_name)[0] as gis_display_name,
              |avg(wear_loss) as wear_loss,
              |avg(path_loss) as path_loss
              |from
              |xxx
              |""".stripMargin)
          .map(s => BuildingConfig(s.getAs[String]("buildingid"), s.getAs[Int]("height"), s.getAs[Long]("gridcount"), s.getAs[String]("gis_display_name"), s.getAs[Double]("wear_loss"), s.getAs[Double]("path_loss")))
          .toDF.write.format("org.apache.spark.sql.json").mode(SaveMode.Overwrite).save(s"/user/my/buidlingconfigjson/${p_city}")

    从hdfs中读取json文件:

     /**
          * scala> buildingConfig.printSchema
          * root
          * |-- building_height: long (nullable = true)
          * |-- buildingid: string (nullable = true)
          * |-- gis_display_name: string (nullable = true)
          * |-- gridcount: long (nullable = true)
          * |-- path_loss: double (nullable = true)
          * |-- wear_loss: double (nullable = true)
          **/
        spark.read.json(s"/user/my/buildingconfigjson/${p_city}")
          .map(s => BuildingConfig(s.getAs[String]("buildingid"), s.getAs[Long]("building_height"), s.getAs[Long]("gridcount"), s.getAs[String]("gis_display_name"), s.getAs[Double]("wear_loss"), s.getAs[Double]("path_loss")))
          .createOrReplaceTempView("building_scene_config")
  • 相关阅读:
    延迟消失菜单
    控制产品上下滚动
    百度音乐全选
    百度文库评分两种代码写法
    选项卡
    搜狐视频
    m 调用传参图片切换
    IIS 7.5站点配置
    jquery plugins —— datatables 搜索后汇总
    jquery plugins —— datatables 增加行号
  • 原文地址:https://www.cnblogs.com/yy3b2007com/p/8564220.html
Copyright © 2011-2022 走看看