zoukankan      html  css  js  c++  java
  • SparkSQL JSON数据操作(1.3->1.4)

    1.用户自定义schema

    data

    json串格式如下:

    {
        "partner_code": "demo",
        "app_name": "web",
        "person_info": {
            "name": "张三",
            "age": 18
        },
        "items": [
            {
                "item_id": 1,
                "item_name": "王家村",
                "group": "group1"
            },
            {
                "item_id": 2,
                "item_name": "李家澡堂",
                "item_detail": {
                    "platform_count": 2
                },
                "group": "group2"
            }
        ]
    }

    spark1.3

    在spark1.3我们是这样处理的

    //定义schema
    val struct =StructType(
        StructField("partner_code", StringType, true) ::
        StructField("app_name", StringType, true)::
        StructField("person_info",MapType(StringType,StringType,true)) ::                    
        StructField("items",ArrayType(MapType(StringType,StringType,true))) ::
            Nil)
    
     val data = sc.textFile("path/jsonFile")
     val df = sqlContext.jsonRDD(data,struct)
     df.printSchema
     df.show

    spark1.4

    //定义schema
    val struct =StructType(
        StructField("partner_code", StringType, true) ::
        StructField("app_name", StringType, true)::
        StructField("person_info",MapType(StringType,StringType,true)) ::                    
        StructField("items",ArrayType(MapType(StringType,StringType,true))) ::
            Nil)
    
    val df = sqlContext.read.schema(struct).json("path/jsonFile")

    输出结果

    //df.printSchema
    root
     |-- partner_code: string (nullable = true)
     |-- app_name: string (nullable = true)
     |-- person_info: map (nullable = true)
     |    |-- key: string
     |    |-- value: string (valueContainsNull = true)
     |-- items: array (nullable = true)
     |    |-- element: map (containsNull = true)
     |    |    |-- key: string
     |    |    |-- value: string (valueContainsNull = true)
    
    //df.show
    +------------+--------+--------------------+--------------------+
    |partner_code|app_name|         person_info|               items|
    +------------+--------+--------------------+--------------------+
    |        demo|     web|Map(name -> 张三, a...|List(Map(item_id ...|
    +------------+--------+--------------------+--------------------+

    系统自动生成schema

    直接使用自带的解析会更方便,不过那样会产生大量的struct结构,同时如果结构复杂多变将会产生大量的空值。

    //不需要定义schema,系统自动判断生成
    val df = sqlContext.read.json("path/jsonFile")
    df.printSchema
    df.show

    输出结果

    //df.printSchema
    root
     |-- app_name: string (nullable = true)
     |-- items: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- group: string (nullable = true)
     |    |    |-- item_detail: struct (nullable = true)
     |    |    |    |-- platform_count: long (nullable = true)
     |    |    |-- item_id: long (nullable = true)
     |    |    |-- item_name: string (nullable = true)
     |-- partner_code: string (nullable = true)
     |-- person_info: struct (nullable = true)
     |    |-- age: long (nullable = true)
     |    |-- name: string (nullable = true)
    
    //df.show
    +--------+--------------------+------------+-----------+
    |app_name|               items|partner_code|person_info|
    +--------+--------------------+------------+-----------+
    |     web|List([group1,null...|        demo|    [18,张三]|
    +--------+--------------------+------------+-----------+
  • 相关阅读:
    JavaScript原型、闭包、继承和原型链等等总结
    JS创建对象的几种方式整理
    js中 给json对象添加属性和json数组添加元素
    JSON 数组
    httpclient封装
    java 数字和日期处理
    jmeter所有版本下载路径
    idea的使用
    Java环境的搭建
    Axure8.0可用的授权码
  • 原文地址:https://www.cnblogs.com/zhangyunlin/p/6168181.html
Copyright © 2011-2022 走看看