zoukankan      html  css  js  c++  java
  • spark stream初探

    spark带了一个NetworkWordCount测试程序,用以统计来自某TCP连接的单词输入:

    /usr/local/spark/bin/run-example streaming.NetworkWordCount localhost 9999

    再启动netcat:
    nc -lk 9999

    尝试输入一些单词:

    hello world
    damn it

    可以看到NetworkWordCount产生如下输出:

    -------------------------------------------
    Time: 1425866862000 ms
    -------------------------------------------
    (world,1)
    (hello,1)
    -------------------------------------------
    Time: 1425866877000 ms
    -------------------------------------------
    (damn,1)
    (it,1)

    也可以手动在shell里输入NetworkWordCount的代码:

    scala> :paste
    // Entering paste mode (ctrl-D to finish)
    
    import org.apache.spark._
    import org.apache.spark.streaming._
    import org.apache.spark.streaming.StreamingContext._
    
    // Create a local StreamingContext with two working thread and batch interval of 1 second.
    // The master requires 2 cores to prevent from a starvation scenario.
    
    val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
    val ssc = new StreamingContext(conf, Seconds(1))
    // Create a DStream that will connect to hostname:port, like localhost:9999
    val lines = ssc.socketTextStream("localhost", 9999)
    // Split each line into words
    val words = lines.flatMap(_.split(" "))
    val pairs = words.map(word => (word, 1))
    val wordCounts = pairs.reduceByKey(_ + _)
    
    // Print the first ten elements of each RDD generated in this DStream to the console
    wordCounts.print()
    ssc.start()             // Start the computation
    ssc.awaitTermination()  // Wait for the computation to terminate

    执行后,即可在屏幕上得到类似的输出。

  • 相关阅读:
    office 2007
    关于网站爬虫的爬取
    华为交换机默认密码大合集
    一些不错的网站工具箱
    HDU 2023 求平均成绩
    HDU 2056 Rectangles
    HDU 2022 海选女主角
    HDU 2045 不容易系列之(3)—— LELE的RPG难题
    HDU 2042 不容易系列之二
    HDU 2044 一只小蜜蜂...
  • 原文地址:https://www.cnblogs.com/bluejoe/p/5115854.html
Copyright © 2011-2022 走看看