zoukankan      html  css  js  c++  java
  • spark处理jsonFile

    按照spark的说法,这里的jsonFile是特殊的文件:

    Note that the file that is offered as jsonFile is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.

    它是按行分隔多个JSON对象,否则的话就会出错。

    以下是一个jsonFile的内容:

    scala> val path = "examples/src/main/resources/people.json"
    path: String = examples/src/main/resources/people.json
    
    scala> Source.fromFile(path).foreach(print)
    {"name":"Michael"}
    {"name":"Andy", "age":30}
    {"name":"Justin", "age":19}

    可以获取到一个SchemaRDD:

    scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    scala> val jsonFile = sqlContext.jsonFile(path)
    scala> jsonFile.printSchema()
    root
     |-- age: integer (nullable = true)
     |-- name: string (nullable = true)

    针对该SchemaRDD可以做遍历操作:

    jsonFile.filter(row=>{val age=row(0).asInstanceOf[Int];age>=13&&age<=19}).collect

    既然是SchemaRDD,就可以采用SQL:

    scala> jsonFile.registerTempTable("people")
    scala> val teenagers = sqlContext.sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
    scala> teenagers.foreach(println)
  • 相关阅读:
    京东饭粒捡漏V1.15
    京东饭粒捡漏V1.14
    京东饭粒捡漏V1.13
    京东饭粒捡漏V1.1.0
    京东饭粒捡漏V1.0.8
    京东饭粒捡漏V1.0.7
    性能瓶颈分析总结
    Jmeter循环控制
    HttpClient接口测试之会话保持
    Jenkins自动部署Tomcat项目
  • 原文地址:https://www.cnblogs.com/bluejoe/p/5115851.html
Copyright © 2011-2022 走看看