【spark】spark2升级到spark3，spark3中的包变动记录

背景:

spark3新增动态裁剪。现尝试将spark2升级到spark3

当前版本：spark 2.4.1，scala 2.11.12

目标版本：spark 3.1.1,  scala 2.12.13

异常记录:

• 异常1

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport

出问题的包

 <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
    <version>2.4.1</version>
</dependency>

修正后

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
    <version>3.0.0</version>
</dependency>

异常原因:

spark3.0中的org.apache.spark.sql.sources.DataSourceRegister中serviceLoader加载的类为

org.apache.spark.sql.execution.datasources.v2.csv.CSVDataSourceV2
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider
org.apache.spark.sql.execution.datasources.v2.json.JsonDataSourceV2
org.apache.spark.sql.execution.datasources.noop.NoopDataSource
org.apache.spark.sql.execution.datasources.orc.OrcFileFormat
org.apache.spark.sql.execution.datasources.v2.parquet.ParquetDataSourceV2
org.apache.spark.sql.execution.datasources.v2.text.TextDataSourceV2
org.apache.spark.sql.execution.streaming.ConsoleSinkProvider
org.apache.spark.sql.execution.streaming.sources.RateStreamProvider
org.apache.spark.sql.execution.streaming.sources.TextSocketSourceProvider
org.apache.spark.sql.execution.datasources.binaryfile.BinaryFileFormat

对比之前spark2中

org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider
org.apache.spark.sql.execution.datasources.json.JsonFileFormat
org.apache.spark.sql.execution.datasources.orc.OrcFileFormat
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
org.apache.spark.sql.execution.datasources.text.TextFileFormat
org.apache.spark.sql.execution.streaming.ConsoleSinkProvider
org.apache.spark.sql.execution.streaming.sources.RateStreamProvider
org.apache.spark.sql.execution.streaming.sources.TextSocketSourceProvider

发现部分的Source已发生改变。追踪下来 org/apache/spark/sql/sources 下的v2包都没了

spark2中的KafkaSourceProvider

private[kafka010] class KafkaSourceProvider extends DataSourceRegister
    with StreamSourceProvider
    with StreamSinkProvider
    with RelationProvider
    with CreatableRelationProvider
    with StreamWriteSupport
    with ContinuousReadSupport
    with MicroBatchReadSupport
    with Logging {
  import KafkaSourceProvider._

spark3中的KafkaSourceProvider

private[kafka010] class KafkaSourceProvider extends DataSourceRegister
    with StreamSourceProvider
    with StreamSinkProvider
    with RelationProvider
    with CreatableRelationProvider
    with SimpleTableProvider
    with Logging {
  import KafkaSourceProvider._

---

• 异常2

目前vertica提供的spark暂不支持3.0，需要通过jdbc方式重新实现一版

• 异常3

java.lang.String cannot be cast to java.time.ZonedDateTime

异常源:

<dependency>
    <groupId>com.github.housepower</groupId>
    <artifactId>clickhouse-integration-spark_2.12</artifactId>
    <version>2.5.4</version>
</dependency>

建表语句:

create table default.zwy_test (time DateTime,AMP Float64,NOZP Int32,value Int32,reason String ) ENGINE = MergeTree order by time

写入数据的schema:

root
 |-- time: string (nullable = true)
 |-- AMP: double (nullable = true)
 |-- NOZP: integer (nullable = true)
 |-- value: integer (nullable = true)
 |-- reason: string (nullable = true)

异常原因:

在Spark 3.0中，将值插入具有不同数据类型的表列中时，将根据ANSI SQL标准执行类型强制转换。标准SQL的转换规则参考，其中String转日期已经不属于隐式转换，而且spark2中String会自动转换为日期类型。因此spark2升级到spark3中，需要对String类型通过from_utc_timestamp等函数显式地转换

• 变动1

　　jdbc spark3增加keytab，principal参数，支持kerberos了

spark2到3的变更记录 https://spark.apache.org/docs/3.0.0/core-migration-guide.html