zoukankan      html  css  js  c++  java
  • Spark部署配置

    前提是已经安装了Hadoop

    ============================ SetUp Spark=============================
    Configuration
    spark-env.sh
    HADOOP_CONF_DIR=/opt/data02/hadoop-2.6.0-cdh5.4.0/etc/hadoop
    JAVA_HOME=/opt/modules/jdk1.7.0_67
    SCALA_HOME=/opt/modules/scala-2.10.4
    #######################################################
    SPARK_MASTER_IP=hadoop-spark.dragon.org
    SPARK_MASTER_PORT=7077
    SPARK_MASTER_WEBUI_PORT=8080
    SPARK_WORKER_CORES=1
    SPARK_WORKER_MEMORY=1000m
    SPARK_WORKER_PORT=7078
    SPARK_WORKER_WEBUI_PORT=8081
    SPARK_WORKER_INSTANCES=1
    slaves
    hadoop-spark.dragon.org
    spark-defaults.conf
    spark.master spark://hadoop-spark.dragon.org:7077
    Start Spark
    Start Master
    sbin/start-master.sh
    Start Slaves
    sbin/start-slaves.sh
    WEB UI
    http://hadoop-spark.dragon.org:8080

    ============================ Test Spark=============================

    scala> val rdd=sc.textFile("hdfs://hadoop-spark.dragon.org:8020/user/hadoop/data/wc.input")

    scala> rdd.cache()

    scala> val wordcount=rdd.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_)

    scala> wordcount.take(10)

    scala> val wordsort=wordcount.map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1))

    scala> wordsort.take(10)



  • 相关阅读:
    java常用api
    常用命令
    mysql常用命令
    特性
    centos ubuntu 软件安装
    WebStorm创建Vue项目记录
    登录oracle官网下载资料账号可以使用(保存)(转)
    java学习之路—JDBC—DBUtils
    Linux从入门到精通(第4章--桌面环境)
    Linux从入门到精通(第2章--Linux安装)
  • 原文地址:https://www.cnblogs.com/ilinuxer/p/5119904.html
Copyright © 2011-2022 走看看