zoukankan      html  css  js  c++  java
  • spark读取写入jdbc.,Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

     df.write
              .option("truncate", "true")
              .option("driver", mysqlDriver)
              .mode(SaveMode.Overwrite).jdbc(url, table, pro)
    

     需要配置某个jdbc类型驱动driver,option("driver", mysqlDriver),不配置的话不会通过url前缀去寻找的,会抛出错误

    1/01/26 14:07:35 INFO LineBufferedStream: stdout: java.lang.InstantiationException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.db.RelationDb.dataFrameInsert(RelationDb.scala:56)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:99)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:26)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:39)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$.run(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp.run(RecommendAnalysisApp.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest$.main(BulkLoadTest.scala:65)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest.main(BulkLoadTest.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:180)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.security.AccessController.doPrivileged(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at javax.security.auth.Subject.doAs(Subject.java:422)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

  • 相关阅读:
    Objective-C系列总结之基础知识
    OC自动释放池autoreleasepool介绍
    OC导入框架方式#import、@import的区别
    OC源文件扩展名
    安装好MySQL后就开始学习如何后台创建自己的数据库吧!
    如何修改Eclipse中的快捷键
    安装MySQL的详细步骤
    Eclipse运行错误:Failed to load the JNI shared library的解决办法
    导航栏返回带的数据
    flutter Container
  • 原文地址:https://www.cnblogs.com/hejunhong/p/14330160.html
Copyright © 2011-2022 走看看