zoukankan      html  css  js  c++  java
  • Spark在Hadoop集群上的配置(spark-1.1.0-bin-hadoop2.4)

    运行Spark服务,需要在每个节点上部署Spark。

    可以先从主节点上将配置修改好,然后把Spark直接scp到其他目录。

    关键配置

    修改conf/spark-env.sh文件:

    export JAVA_HOME=/usr/java/latest
    export HADOOP_CONF_DIR=/opt/hadoop-2.4.1/etc/hadoop/
    export SPARK_MASTER_IP=master
    

    以上是必要的几个配置,详细的配置说明,请参见官网的Document。

    修改conf/slaves,slave节点配置,将worker节点的主机名直接添加进去即可。

    启动集群

    sbin/start-all.sh

    jps查看本机java进程,主节点应该有Master进程,worker节点应该有个Worker进程。

    WebUI地址:http://master:8080

    测试Spark,bin/run-example SparkPi,正常的话,可以看到以下测试结果:

    ...
    14/11/11 22:11:25 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4052 ms on localhost (1/2)
    14/11/11 22:11:25 INFO scheduler.DAGScheduler: Stage 0 (reduce at SparkPi.scala:35) finished in 4.130 s
    14/11/11 22:11:25 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 69 ms on localhost (2/2)
    14/11/11 22:11:25 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
    14/11/11 22:11:25 INFO spark.SparkContext: Job finished: reduce at SparkPi.scala:35, took 4.613856515 s
    Pi is roughly 3.1431
    14/11/11 22:11:26 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/metrics/json,null}
    14/11/11 22:11:26 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}
    14/11/11 22:11:26 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/,null}
    14/11/11 22:11:26 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/static,null}
    14/11/11 22:11:26 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/executors/json,null}
    ...
    
  • 相关阅读:
    hdu 5101 Select
    hdu 5100 Chessboard
    cf B. I.O.U.
    cf C. Inna and Dima
    cf B. Inna and Nine
    cf C. Counting Kangaroos is Fun
    Radar Installation 贪心
    spfa模板
    Sequence
    棋盘问题
  • 原文地址:https://www.cnblogs.com/jasondan/p/4089330.html
Copyright © 2011-2022 走看看