zoukankan      html  css  js  c++  java
  • spark读取写入jdbc.,Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

     df.write
              .option("truncate", "true")
              .option("driver", mysqlDriver)
              .mode(SaveMode.Overwrite).jdbc(url, table, pro)
    

     需要配置某个jdbc类型驱动driver,option("driver", mysqlDriver),不配置的话不会通过url前缀去寻找的,会抛出错误

    1/01/26 14:07:35 INFO LineBufferedStream: stdout: java.lang.InstantiationException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.db.RelationDb.dataFrameInsert(RelationDb.scala:56)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:99)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:26)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:39)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$.run(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp.run(RecommendAnalysisApp.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest$.main(BulkLoadTest.scala:65)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest.main(BulkLoadTest.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:180)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.security.AccessController.doPrivileged(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at javax.security.auth.Subject.doAs(Subject.java:422)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

  • 相关阅读:
    自学Linux命令的四种方法
    POJ 1170 Shopping Offers -- 动态规划(虐心的六重循环啊!!!)
    九度OJ 1447 最短路 1008 最短路径问题
    九度OJ 1024 畅通工程 -- 并查集、贪心算法(最小生成树)
    PHPActiveRecord 学习三
    PHPUnit 组织测试
    PHPActiveRecord validates
    PHPActiveRecord 学习二
    PHPActiveRecord 学习一
    PHP ActiveRecord demo栗子中 关于类名 的问题
  • 原文地址:https://www.cnblogs.com/hejunhong/p/14330160.html
Copyright © 2011-2022 走看看