zoukankan      html  css  js  c++  java
  • Eclipse 编写的spark版WordCount运行在spark上

    1、代码编写

     if (args.length != 3 ){
          println("usage is org.test.WordCount <master> <input> <output>")
          return
        }
        val sc = new SparkContext(args(0), "WordCount",
        System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_TEST_JAR")))
        val textFile = sc.textFile(args(1))
        val result = textFile.flatMap(line => line.split("\s+"))
            .map(word => (word, 1)).reduceByKey(_ + _)
        result.saveAsTextFile(args(2))

    2、导出jar包,这里我命名为WordCount.jar

    3、运行

    bin/spark-submit --master spark://master:7077 /home/zkpk/Desktop/WordCount.jar hdfs://master:9000/dedup_in hdfs://master:9000/dedup_out

    先记录下,第三部还没执行,抽空试试

     
  • 相关阅读:
    H
    饭卡(0 1背包)
    J
    H
    E
    3991: [SDOI2015]寻宝游戏
    CF 1051 F. The Shortest Statement
    CF 643 E. Bear and Destroying Subtrees
    4567: [Scoi2016]背单词
    4946: [Noi2017]蔬菜
  • 原文地址:https://www.cnblogs.com/wyhong/p/4126013.html
Copyright © 2011-2022 走看看