zoukankan      html  css  js  c++  java
  • Spark HA 的搭建

    接hadoop HA的搭建,因为你zookeeper已经部署完成,所以直接安装spark就可以

    tar –xzf spark-1.6.1-bin-hadoop2.6.tgz -C ../services

    -bash-4.1$ ln -sv services/spark-1.6.1-bin-hadoop2.6/ spark
    

    修改spark-env.sh的相应信息

    -bash-4.1$ vim spark-env.sh
    
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=MSJTVL-DSJC-H03:2181,MSJTVL-DSJC-H04:2181,MSJTVL-DSJC-H05:2181 -Dspark.deploy.zookeeper.dir=/hadoop/spark"
    export JAVA_HOME=/opt/java/jdk1.8.0_91
    export SCALA_HOME=/hadoop/services/scala-2.10.5
    #export SPARK_WORKER_CORES=5
    export SPARK_WORKER_MEMORY=5g
    export HADOOP_HOME=/hadoop/hadoop
    export HADOOP_CONF_DIR=/hadoop/hadoop/etc/hadoop
    

    在slaves中增加work的配置信息

    -bash-4.1$ vim slaves

      MSJTVL-DSJC-H03
      MSJTVL-DSJC-H04
      MSJTVL-DSJC-H05

    启动spark

    -bash-4.1$ ./start-all.sh 
    starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-iZ2zefn1rjw3rfejj5hhwbZ.out
    iZ2zehhwq5a6tmvi3wg17iZ: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-iZ2zehhwq5a6tmvi3wg17iZ.out
    iZ2zee62ni1rdbg34t5mydZ: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-iZ2zee62ni1rdbg34t5mydZ.out
    iZ2zee62ni1rdbg34t5mycZ: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-iZ2zee62ni1rdbg34t5mycZ.out
    -bash-4.1$ jps
    9905 DFSZKFailoverController
    31076 Jps
    9621 NameNode
    9995 ResourceManager
    30990 Master
    

    在另一个节点上,启动新的master

    -bash-4.1$ ./start-master.sh 
    starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-iZ2zefn1rjw3rfejj5hhwaZ.out
    -bash-4.1$ 
    

    登录网页可以查看到相应的状态,web的端口是8080 

      

      

  • 相关阅读:
    nginx last break等
    Jmeter
    nginx location规则
    解决性能瓶颈的一些思路
    spring 一些总结
    idea快捷键
    可读的jvm gc日志时间
    redis 搭建集群
    windows下使用Python来修改文件时间戳
    Python获得文件时间戳
  • 原文地址:https://www.cnblogs.com/tian880820/p/5960130.html
Copyright © 2011-2022 走看看