zoukankan      html  css  js  c++  java
  • IDEA创建本地Spark程序,并本地运行

     1   IDEA创建maven项目进行测试

    v创建一个新项目,步骤如下:

     

     

     选择“Enable Auto-Import”,加载完后:选择“Enable Auto-Import”,加载完后:

     添加SDK依赖:

     

     

     点击OK  ok

     可以看到scala包加载成功

     再修改pox.xml文件

    <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
      <modelVersion>4.0.0</modelVersion>
      <groupId>com.wangle</groupId>
      <artifactId>LocalSpark</artifactId>
      <version>1.0-SNAPSHOT</version>
    
        <properties>
            <spark.version>2.1.0</spark.version>
            <scala.version>2.11</scala.version>
        </properties>
        <dependencies>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-hive_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_${scala.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
        </dependencies>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.6.0</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.19</version>
                    <configuration>
                        <skip>true</skip>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    
    
    
    </project>

    新建“Demo1.scala”,测试

  • 相关阅读:
    H50055:html 页面加载完后再加载js文件 ,url带有时间戳后缀
    H50054:html 页面自动刷新 http-equiv属性
    WebGL_0015:参数值在一定范围内循环变化
    H50053:switch 判断范围
    WebGL_0014:改变相机的刷新颜色
    WebGL_0013:JQuery获取json文件 读取数据
    WebGL_0012:三维讲解导航模板 Icon方式
    H50052:按钮 禁止 选择 拖拽 右键
    滑动窗口的最大值(队列)
    MapReduce程序编写过程
  • 原文地址:https://www.cnblogs.com/wangleBlogs/p/12002066.html
Copyright © 2011-2022 走看看