spark基于mavn的scala程序

好久没有写博客了，但是发现以前学习的东西，现在想起来就忘记了，太浪费时间了。学习还是要掌握方法。对于我们这样的程序猿，天生的嘴比较笨，写作不仅可以使我们内心得到排解，另外能使我们对一个东西认识的更加透彻。

1, 准备环境

jdk1.8 ,scala-sdk-2.11.8,intellij idea

2.pom文件

<properties>
    <spark.version>2.1.0</spark.version>
    <scala.version>2.11</scala.version>
    <mysql.version>5.1.24</mysql.version>
</properties>


<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_${scala.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>


    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>${mysql.version}</version>
    </dependency>

</dependencies>

<build>
    <plugins>

        <plugin>
            <groupId>org.scala-tools</groupId>
            <artifactId>maven-scala-plugin</artifactId>
            <version>2.15.2</version>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.6.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.19</version>
            <configuration>
                <skip>true</skip>
            </configuration>
        </plugin>

    </plugins>
</build>

3.scala 程序

/**
  *
  * 学习SparkSession
  */
object TestSparkSession {

  def main(args: Array[String]): Unit = {
    val spark = SparkSession
      .builder()
      .appName("Spark SQL basic example")
      .config("spark.some.config.option", "some-value")
      .master("local")
      .getOrCreate()

    val df = spark.read.csv("D:\code\mongo\data\all_record.csv")
    df.show(10)
  }
}

4.运行结果图