zoukankan      html  css  js  c++  java
  • 搭建spark集群

    搭建spark集群 spark1.6和hadoop2.6
    1.准备hadoop环境;
    2.准备下载包;
    3.解压安装包:
    tar -xf spark-1.6.0-bin-hadoop2.6.tgz
    4.修改配置文件
    目录:/apps/app/spark1.6-hadoop2.6/conf
    a.修改spark-env.sh:
    cp spark-env.sh.template spark-env.sh
    vim spark-env.sh
    添加:
    export JAVA_HOME=/apps/app/dk/jdk1.7.0_79
    export SCALA_HOME=/apps/app/dk/scala-2.10.4
    export SPARK_DRIVER_MEMORY=1g
    export SPARK_WORKER_MEMORY=512m
    export SPARK_WORKER_CORES=2
    export SPARK_EXECUTOR_MEMORY=512m
    export SPARK_EXECUTOR_CORES=2
    export HADOOP_HOME=/apps/app/hadoop-2.6.0
    export HADOOP_CONF_DIR=/apps/app/hadoop-2.6.0/etc/hadoop
    b.修改spark-defaults.conf
    cp spark-defaults.conf.template spark-defaults.conf
    vim spark-defaults.conf
    spark.eventLog.enabled           true
    spark.eventLog.dir               hdfs://kylin01:9000/historyserverforspark
    spark.history.fs.logDirectory    hdfs://kylin01:9000/historyserverforspark
    spark.yarn.historyServer.address    kylin01:18880
    spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
    c.修改:hive-site.xml
    <configuration>
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://kylin06:9083</value>
    </property>
    <property>
    <name>hive.server2.thrift.min.worker.threads</name>
    <value>5</value>
    <description>Minimum number of Thrift worker threads</description>
    </property>

    <property>
    <name>hive.server2.thrift.max.worker.threads</name>
    <value>50</value>
    <description>Maximum number of Thrift worker threads</description>
    </property>

    <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
    <description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>
    </property>

    <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>kylin06</value>
    <description>Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST</description>
    </property>
    </configuration>
    d.修改:slaves
    cp slaves.template slaves
    vim slaves

    视频地址:http://v.qq.com/x/search/?q=%E5%A4%A7%E6%95%B0%E6%8D%AE%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA&stag=0&smartbox_ab=

  • 相关阅读:
    用户控件赋值
    计算一串数字中每个数字出现的次数
    如何理解c和c++的复杂类型声明
    STM32 NVIC学习
    stm32f10x_flash.c中文版
    IBM中国研究院Offer之感言——能力是一种态度
    对于STM32别名区的理解 (转载)
    STM32时钟学习之STM3210X_RCC.H解读
    STM32 DMA
    STM32 内部时钟输出PA.8(MCO)
  • 原文地址:https://www.cnblogs.com/gnool/p/6115548.html
Copyright © 2011-2022 走看看