zoukankan      html  css  js  c++  java
  • spark读取写入jdbc.,Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

     df.write
              .option("truncate", "true")
              .option("driver", mysqlDriver)
              .mode(SaveMode.Overwrite).jdbc(url, table, pro)
    

     需要配置某个jdbc类型驱动driver,option("driver", mysqlDriver),不配置的话不会通过url前缀去寻找的,会抛出错误

    1/01/26 14:07:35 INFO LineBufferedStream: stdout: java.lang.InstantiationException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.db.RelationDb.dataFrameInsert(RelationDb.scala:56)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:99)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.recommend.relation.GoodsRelationAnalysis.analysis(GoodsRelationAnalysis.scala:26)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:39)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$$anonfun$run$3.apply(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp$.run(RecommendAnalysisApp.scala:34)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.RecommendAnalysisApp.run(RecommendAnalysisApp.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest$.main(BulkLoadTest.scala:65)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at com.wanmi.sbc.dw.spark.app.BulkLoadTest.main(BulkLoadTest.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.lang.reflect.Method.invoke(Method.java:498)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:180)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at java.security.AccessController.doPrivileged(Native Method)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at javax.security.auth.Subject.doAs(Subject.java:422)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:178)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    21/01/26 14:07:35 INFO LineBufferedStream: stdout: Caused by: java.lang.NoSuchMethodException: org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.<init>()

  • 相关阅读:
    命令安装mongod作为服务启动
    npm命令
    win服务
    win进程操作
    【数据结构与算法之美】7.排序算法之冒泡、选择、插入
    【leetcode】23.合并K个排序链表
    【leetcode】141.环形链表
    【数据结构与算法之美】6.环形队列
    【数据结构与算法之美】5.基于链表实现队列
    【数据结构与算法之美】4.基于数组实现队列
  • 原文地址:https://www.cnblogs.com/hejunhong/p/14330160.html
Copyright © 2011-2022 走看看