zoukankan      html  css  js  c++  java
  • spark连接mysql数据库

    1.安装启动检查Mysql服务。
    netstat -tunlp (3306)

    2.spark 连接mysql驱动程序

    –cp /usr/local/hive/lib/mysql-connector-java-5.1.40-bin.jar /usr/local/spark/jars

    3.启动 Mysql shell,新建数据库spark,表student。
    select * from student;

    create database spark;
    use spark;
    create table student (id int(4), name char(20), gender char(4), age int(4));
    alter table student change id id int auto_increment primary key;
    insert into student values(1,'Xueqian','F',23);
    insert into student values(2,'Weiliang','M',24);
    insert into student values(3,'Rongcheng','M',26);
    insert into student values(4,'Guanhua','M',27);
    select * from student;

    4.spark读取MySQL数据库中的数据
    spark.read.format("jdbc").option("url", "jdbc:mysql://localhost:3306/spark?useSSL=false") ...  .load()

    spark.read.format("jdbc")
    .option("url", "jdbc:mysql://localhost:3306/spark")
    .option("driver","com.mysql.jdbc.Driver").option("dbtable", "student")
    .option("user", "root")
    .option("password", "1")
    .load()
    .show()

    5.spark向MySQL数据库写入数据
    studentDF.write.format(‘jdbc’).option(…).mode(‘append’).save() 

    from pyspark.sql.types import Row,StructField,StringType,StructType,IntegerType
    
    #下面设置两条数据,表示两个学生的信息
    studentRDD = spark.sparkContext.parallelize(["5 zhao M 26","6 wang M 27"]).map(lambda line:line.split(" "))
    
    #下面创建Row对象,每个Row对象都是rowRDD中的一行
    rowRDD = studentRDD.map(lambda p:Row(p[1].strip(),p[2].strip(),int(p[3])))
    
    # 下面设置模式信息
    schema = StructType([StructField("name",StringType(),True),StructField("gender",StringType(),True),StructField("age",IntegerType(),True)])
    
    #建立起Row对象和模式之间的对应关系,也就是把数据和模式对应起来
    studentDF = spark.createDataFrame(rowRDD,schema)
    
    #写入数据库
    prop = {}
    prop['user'] = 'root'
    prop['password'] = '123456'
    prop['driver'] = 'com.mysql.jdbc.Driver'
    studentDF.write.jdbc("jdbc:mysql://localhost:3306/spark?useSSL=false",'student','append',prop)

     

     

     

  • 相关阅读:
    事务与事务隔离级别
    TNS12535: TNS: 操作超时
    11g的exp导出空表提示EXP00011: SCOTT.TEST1 不存在
    oracle中chr含义
    SQL Server 2008 System Views Map
    SQL Server Execution Plans eBook
    生成建表脚本(V3.0)
    SQL Server 2008 通过配置数据库邮件实现发送邮件功能
    MSSQL2005中的非公开存储过程sp_msdependencies
    SQL Server Tacklebox Free eBook
  • 原文地址:https://www.cnblogs.com/shawncs/p/14826174.html
Copyright © 2011-2022 走看看