zoukankan      html  css  js  c++  java
  • Spark Standalone cluster try

    Spark Standalone cluster

    node*
    -- stop firewalld
    systemctl stop firewalld
    systemctl disable firewalld
    -- tar spark
    cd /opt
    tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz
    cd spark-2.4.0-bin-hadoop2.7
    -- cp application & application data
    ftp spark.test-1.0.jar -> /opt/spark-2.4.0-bin-hadoop2.7
    ftp words_count.txt -> /opt/spark-2.4.0-bin-hadoop2.7/data

    node1
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-master.sh

    node2
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node3
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node4
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node?
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./bin/spark-submit --class xyz.fz.spark.WordsCount --master spark://node1:7077 spark.test-1.0.jar

    spark result: Lines with Basics: 2, lines with Programming: 2

     

  • 相关阅读:
    葡萄庄园 [图论]
    硬币游戏 [博弈论, 思维题]
    烹饪 [容斥]
    BZOJ1597 [Usaco2008 Mar]土地购买 [斜率优化]
    TCP IP协议
    soap协议
    xml的语法规则
    fiddler的使用
    常见默认端口
    智能休眠时间的使用
  • 原文地址:https://www.cnblogs.com/xiayudashan/p/10497205.html
Copyright © 2011-2022 走看看