zoukankan      html  css  js  c++  java
  • Windows下IntelliJ IDEA中运行Spark Standalone

    前提条件:

    1、Spark Standalone 集群部署完成

    2、Intellij Idea 能够运行 Spark local 模式的程序。 

    源码:

     1 import org.apache.spark.{SparkContext, SparkConf}
     2 import scala.math._
     3 
     4 /**
     5   * Created by Edward on 2016/8/27.
     6   */
     7 object WordCount {
     8   def main(args: Array[String]) {
     9 
    10     val sparkConf = new SparkConf().setAppName("WordCount")
    11       .setMaster("spark://node1:7077").setJars(List("D:\documents\Spark\MyDemo\Test\out\artifacts\spark_sample_jar\Test.jar"))
    12     //val sc = new SparkContext(sparkConf)
    13     val spark = new SparkContext(sparkConf)
    14     val slices = if (args.length > 0) args(0).toInt else 2
    15     val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    16     val count = spark.parallelize(1 until n, slices).map { i =>
    17         val x = random * 2 - 1
    18         val y = random * 2 - 1
    19         if (x*x + y*y < 1) 1 else 0
    20       }.reduce(_ + _)
    21     println("Pi is roughly " + 4.0 * count / n)
    22     spark.stop()
    23   }
    24 }

    这里主要的思想还是将打包的jar提交到集群。

    使用.setJars方法

  • 相关阅读:
    Object-C,NSSet,不可变集合
    NYIST 860 又见01背包
    NYIST 1070 诡异的电梯【Ⅰ】
    HDU 1542 Atlantis
    HDU 4756 Install Air Conditioning
    CodeForces 362E Petya and Pipes
    HDU 4751 Divide Groups
    HDU 3081 Marriage Match II
    UVA 11404 Palindromic Subsequence
    UVALIVE 4256 Salesmen
  • 原文地址:https://www.cnblogs.com/one--way/p/5818989.html
Copyright © 2011-2022 走看看