zoukankan      html  css  js  c++  java
  • 搭建spark集群

    搭建spark集群 spark1.6和hadoop2.6
    1.准备hadoop环境;
    2.准备下载包;
    3.解压安装包:
    tar -xf spark-1.6.0-bin-hadoop2.6.tgz
    4.修改配置文件
    目录:/apps/app/spark1.6-hadoop2.6/conf
    a.修改spark-env.sh:
    cp spark-env.sh.template spark-env.sh
    vim spark-env.sh
    添加:
    export JAVA_HOME=/apps/app/dk/jdk1.7.0_79
    export SCALA_HOME=/apps/app/dk/scala-2.10.4
    export SPARK_DRIVER_MEMORY=1g
    export SPARK_WORKER_MEMORY=512m
    export SPARK_WORKER_CORES=2
    export SPARK_EXECUTOR_MEMORY=512m
    export SPARK_EXECUTOR_CORES=2
    export HADOOP_HOME=/apps/app/hadoop-2.6.0
    export HADOOP_CONF_DIR=/apps/app/hadoop-2.6.0/etc/hadoop
    b.修改spark-defaults.conf
    cp spark-defaults.conf.template spark-defaults.conf
    vim spark-defaults.conf
    spark.eventLog.enabled           true
    spark.eventLog.dir               hdfs://kylin01:9000/historyserverforspark
    spark.history.fs.logDirectory    hdfs://kylin01:9000/historyserverforspark
    spark.yarn.historyServer.address    kylin01:18880
    spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
    c.修改:hive-site.xml
    <configuration>
    <property>
    <name>hive.metastore.uris</name>
    <value>thrift://kylin06:9083</value>
    </property>
    <property>
    <name>hive.server2.thrift.min.worker.threads</name>
    <value>5</value>
    <description>Minimum number of Thrift worker threads</description>
    </property>

    <property>
    <name>hive.server2.thrift.max.worker.threads</name>
    <value>50</value>
    <description>Maximum number of Thrift worker threads</description>
    </property>

    <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
    <description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>
    </property>

    <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>kylin06</value>
    <description>Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST</description>
    </property>
    </configuration>
    d.修改:slaves
    cp slaves.template slaves
    vim slaves

    视频地址:http://v.qq.com/x/search/?q=%E5%A4%A7%E6%95%B0%E6%8D%AE%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA&stag=0&smartbox_ab=

  • 相关阅读:
    A1066 Root of AVL Tree (25 分)
    A1099 Build A Binary Search Tree (30 分)
    A1043 Is It a Binary Search Tree (25 分) ——PA, 24/25, 先记录思路
    A1079; A1090; A1004:一般树遍历
    A1053 Path of Equal Weight (30 分)
    A1086 Tree Traversals Again (25 分)
    A1020 Tree Traversals (25 分)
    A1091 Acute Stroke (30 分)
    A1103 Integer Factorization (30 分)
    A1032 Sharing (25 分)
  • 原文地址:https://www.cnblogs.com/gnool/p/6115548.html
Copyright © 2011-2022 走看看