zoukankan      html  css  js  c++  java
  • Spark 覆盖写Hive分区表,只覆盖部分对应分区

    要求Spark版本2.3以上,亲测2.2无效

    配置
    config("spark.sql.sources.partitionOverwriteMode","dynamic")
    注意
    1、saveAsTable方法无效,会全表覆盖写,需要用insertInto,详情见代码
    2、insertInto需要主要DataFrame列的顺序要和Hive表里的顺序一致,不然会数据错误!

    package com.dkl.blog.spark.hive

    import org.apache.spark.sql.SparkSession

    /**

    • Created by dongkelun on 2020/1/16 15:25

    • 博客:Spark 覆盖写Hive分区表,只覆盖部分对应分区

    • 要求Spark版本2.3以上
      */
      object SparkHivePartitionOverwrite {
      def main(args: Array[String]): Unit = {
      val spark = SparkSession
      .builder()
      .appName("SparkHivePartitionOverwrite")
      .master("local")
      .config("spark.sql.parquet.writeLegacyFormat", true)
      .config("spark.sql.sources.partitionOverwriteMode","dynamic")
      .enableHiveSupport()
      .getOrCreate()

      import spark.sql

      val data = Array(("001", "张三", 21, "2018"), ("002", "李四", 18, "2017"))

      val df = spark.createDataFrame(data).toDF("id", "name", "age", "year")
      //创建临时表
      df.createOrReplaceTempView("temp_table")

      val tableName="test_partition"
      //切换hive的数据库
      sql("use test")
      // 1、创建分区表,并写入数据
      df.write.mode("overwrite").partitionBy("year").saveAsTable(tableName)

      spark.table(tableName).show()

      val data1 = Array(("011", "Sam", 21, "2018"))

      val df1 = spark.createDataFrame(data1).toDF("id", "name", "age", "year")
      // df1.write.mode("overwrite").partitionBy("year").saveAsTable(tableName) //不成功,全表覆盖
      // df1.write.mode("overwrite").format("Hive").partitionBy("year").saveAsTable(tableName) //不成功,全表覆盖
      df1.write.mode("overwrite").insertInto(tableName)

      spark.table(tableName).show()

      spark.stop
      }

    }
    结果
    +---+----+---+----+
    | id|name|age|year|
    +---+----+---+----+
    |002| 李四| 18|2017|
    |001| 张三| 21|2018|
    +---+----+---+----+

    +---+----+---+----+
    | id|name|age|year|
    +---+----+---+----+
    |011| Sam| 21|2018|
    +---+----+---+----+

  • 相关阅读:
    (转)Android IPC机制详解
    (转)android 多线程
    (转)android进程间通信:使用AIDL
    (转)如何调用SQLITE工具查看数据库
    (转)Android 自动 打包
    (转)如何手动编译一个APK
    (转)编译Android源码的全过程
    Js打造层拖动实例:网站菜单拖拽移位效果
    JS鼠标悬停时动态翻滚的紫色导航条
    jQuery1.3.2竖向的伸缩菜单
  • 原文地址:https://www.cnblogs.com/tree1123/p/13440952.html
Copyright © 2011-2022 走看看