zoukankan      html  css  js  c++  java
  • spark2.1操作json(save/read)

    建筑物配置信息:

    case class BuildingConfig(buildingid: String, building_height: Long, gridcount: Long, gis_display_name: String, wear_loss: Double, path_loss: Double) extends Serializable

    向hdfs写入json文件:

     sql(
          s"""|select buildingid,
              |height,
              |gridcount,
              |collect_list(gis_display_name)[0] as gis_display_name,
              |avg(wear_loss) as wear_loss,
              |avg(path_loss) as path_loss
              |from
              |xxx
              |""".stripMargin)
          .map(s => BuildingConfig(s.getAs[String]("buildingid"), s.getAs[Int]("height"), s.getAs[Long]("gridcount"), s.getAs[String]("gis_display_name"), s.getAs[Double]("wear_loss"), s.getAs[Double]("path_loss")))
          .toDF.write.format("org.apache.spark.sql.json").mode(SaveMode.Overwrite).save(s"/user/my/buidlingconfigjson/${p_city}")

    从hdfs中读取json文件:

     /**
          * scala> buildingConfig.printSchema
          * root
          * |-- building_height: long (nullable = true)
          * |-- buildingid: string (nullable = true)
          * |-- gis_display_name: string (nullable = true)
          * |-- gridcount: long (nullable = true)
          * |-- path_loss: double (nullable = true)
          * |-- wear_loss: double (nullable = true)
          **/
        spark.read.json(s"/user/my/buildingconfigjson/${p_city}")
          .map(s => BuildingConfig(s.getAs[String]("buildingid"), s.getAs[Long]("building_height"), s.getAs[Long]("gridcount"), s.getAs[String]("gis_display_name"), s.getAs[Double]("wear_loss"), s.getAs[Double]("path_loss")))
          .createOrReplaceTempView("building_scene_config")
  • 相关阅读:
    Android JNI开发摘录(二)之JNI数组处理
    基于HTTP协议的下载功能实现
    javascript中的事件冒泡和事件捕获
    JavaScript模仿块级作用域
    javascript表单的Ajax 提交插件的使用
    论学历与经验
    使用Palette抽取图片颜色
    通过脚本和EL表达式获取域中数据
    需求分析
    Tomcat简介
  • 原文地址:https://www.cnblogs.com/yy3b2007com/p/8564220.html
Copyright © 2011-2022 走看看