zoukankan      html  css  js  c++  java
  • Spark Standalone cluster try

    Spark Standalone cluster

    node*
    -- stop firewalld
    systemctl stop firewalld
    systemctl disable firewalld
    -- tar spark
    cd /opt
    tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz
    cd spark-2.4.0-bin-hadoop2.7
    -- cp application & application data
    ftp spark.test-1.0.jar -> /opt/spark-2.4.0-bin-hadoop2.7
    ftp words_count.txt -> /opt/spark-2.4.0-bin-hadoop2.7/data

    node1
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-master.sh

    node2
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node3
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node4
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./sbin/start-slave.sh spark://node1:7077

    node?
    cd /opt/spark-2.4.0-bin-hadoop2.7
    ./bin/spark-submit --class xyz.fz.spark.WordsCount --master spark://node1:7077 spark.test-1.0.jar

    spark result: Lines with Basics: 2, lines with Programming: 2

     

  • 相关阅读:
    【转】 上海交大ACM队长建议
    好资源
    待做
    分治思想
    周末看的东西
    [UVa11988] Broken Keyboard (a.k.a. Beiju Text)
    UVa 题目分类
    [UVa11729] Commando War
    [LA3135] Arugus
    [UVa11995] I Can Guess the Data Structure!
  • 原文地址:https://www.cnblogs.com/xiayudashan/p/10497205.html
Copyright © 2011-2022 走看看