zoukankan      html  css  js  c++  java
  • spark standalone zookeeper HA部署方式

    虽然spark master挂掉的几率很低,不过还是被我遇到了一次。以前在spark standalone的文章中也介绍过standalone的ha,现在详细说下部署流程,其实也比较简单。

    一.机器

    zookeeper集群

    zk1:2181
    zk2:2181
    zk3:2181
    

    spark master

    spark-m1
    spark-m2
    

    spark worker

    若干
    

    二.步骤

    1.进入spark-m1
    修改conf/spark-env.sh

    vi spark-env.sh
    export SPARK_MASTER_IP=spark-m1
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zk1:2181,zk2:2181,zk3:2181 -Dspark.deploy.zookeeper.dir=/spark"   

    启动master和slaves

    ./sbin/start-master.sh
    ./sbin/start-slaves.sh

    2.进入spark-m2

    修改conf/spark-env.sh

    vi spark-env.sh
    export SPARK_MASTER_IP=spark-m2
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zk1:2181,zk2:2181,zk3:2181 -Dspark.deploy.zookeeper.dir=/spark"   

    启动master和slaves

    ./sbin/start-master.sh
    ./sbin/start-slaves.sh

    三.检测

    在spark-m1的web ui中可以看到状态

    spark-m2中可以看到处于STANDBY状态

    application提交时,master改为

    --master spark://spark-m1:7077,spark-m2:7077
    

    spark shell 测试

    在spark-m1中启动spark Shell

    spark-shell --master spark://spark-m1:7077,spark-m2:7077
    

    连接后关闭spark-m1 master

    ./bin/stop-master.sh
    

    发现spark-shell不会断开而是转到spark-m2的master上继续执行(该过程持续大概1分钟,woker会重新注册到spark-m2上),同时spark-m2变为alive状态。

    可以在spark-m2的master日志中看到:

    15/08/17 14:45:35 INFO ZooKeeperLeaderElectionAgent: We have gained leadership
    15/08/17 14:45:36 INFO Master: I have been elected leader! New state: RECOVERING
    15/08/17 14:45:36 INFO Master: Trying to recover worker:...
    15/08/17 14:45:36 INFO Master: Trying to recover worker: ...
    15/08/17 14:45:36 INFO Master: Trying to recover worker: ...
    ......
    15/08/17 14:45:36 INFO Master: Worker has been re-registered: worker-...
    15/08/17 14:45:36 INFO Master: Worker has been re-registered: worker-...
    15/08/17 14:45:36 INFO Master: Worker has been re-registered: worker-...
    ...
    15/08/17 14:45:36 INFO Master: Recovery complete - resuming operations!

    部署结束

  • 相关阅读:
    在ASP.NET MVC2的Html.BeginForm中定义Form的id和name
    MVC中使用Ajax提交数据 Jquery Ajax方法传值到action
    用XML编写EXCEL文件,XML的写法注意事项,可以C#+xslt导出Excel
    DataGrid中删除分页最后一条记录时PageIndex错误的解决方法
    el 表达式函数对数组的处理
    solr 3.5 配置及应用(三)
    centos 6.2 syslogng的配置
    CentOS 6.2 安装tripwire2.4.2.2配置
    日志服务器的配置
    solr 3.5 配置及应用(一)
  • 原文地址:https://www.cnblogs.com/zhangyunlin/p/6168174.html
Copyright © 2011-2022 走看看