Well,there is no post after my blog has been renovationed,so it is.
Here,
when i use a jdbc format url to connect to a mysql server in spark program.
a error occured , no suitable driver ……..,as follow:
1 2 3 |
def hive_to_mysql(data: DataFrame, tableName: String): Unit = {
data.write.mode(SaveMode.Append).jdbc(url, tableName, mysql_prop)
}
|
I think that the sparkcontext will find a driver from registered driver list according to connection url,
but the error occured.
with i am a beginner so i don’t know how to solve it.
i found a pull request about no suitable driver from spark github,
1 2 3 4 5 6 7 8 |
def getConnector(driver: String, url: String, properties: Properties): () => Connection = {
() => {
try {
- if (driver != null) Utils.getContextOrSparkClassLoader.loadClass(driver)
+ if (driver != null) DriverRegistry.register(driver)
} catch {
case e: ClassNotFoundException => {
logWarning(s"Couldn't find class $driver", e);
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
val upperBound = parameters.getOrElse("upperBound", null)
val numPartitions = parameters.getOrElse("numPartitions", null)
- if (driver != null) Utils.getContextOrSparkClassLoader.loadClass(driver)
+ if (driver != null) DriverRegistry.register(driver)
if (partitionColumn != null
&& (lowerBound == null || upperBound == null || numPartitions == null)) {
@@ -136,7 +136,7 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table, properties)
override def buildScan(requiredColumns: Array[String], filters: Array[Filter]): RDD[Row] = {
- val driver: String = DriverManager.getDriver(url).getClass.getCanonicalName
+ val driver: String = DriverRegistry.getDriverClassName(url)
JDBCRDD.scanTable(
sqlContext.sparkContext,
schema,
|
according to the changes puller made , he create a DriverWrapper extends java.sql.Driver, and reload the jdbc driver with the same driver loaded by sparkcontext .
the key point is that the pull request has been mergen,haha. so i add a line before connectting to mysql server,like this
1 2 3 4 |
def hive_to_mysql(data: DataFrame, tableName: String): Unit = {
DriverRegistry.register("com.mysql.jdbc.Driver")
data.write.mode(SaveMode.Append).jdbc(url, tableName, mysql_prop)
}
|
i think it will succeed ,but but but ,three but is but, it failed, a new error was reported by spark program.
sparkContext can not find the registered driver……………
the the CTO in my company solve the final problem for me.
1 2 3 4 |
Property mysql_prop = new Property()
mysql_prop.add("user","root")
mysql_prop.add("password","password")
mysql_prop.add("driver","com.mysql.jdbc.Driver") //the most important line
|
the most important line had shown we must tell sparkContext the classname of Driver I mean.
why so , it is a bug that the spark job in yarn cluster mode ran by oozie workflow.
In other words,you must put driver key into Property that will be used by mysql connection while the spark job is ran by oozie workflow in yarn cluster mode.
so far, all spark job will finish successfully.