zoukankan      html  css  js  c++  java
  • 搭建单机CDH环境,并更新spark环境

    搭建单机CDH环境,并更新spark环境

    1,安装VMWare Player,http://dlsw.baidu.com/sw-search-sp/soft/90/13927/VMware_player_7.0.0_2305329.1420626349.exe

    2,启动BIOS虚拟化,http://www.cnblogs.com/stono/p/8323516.html

    3,下载CDH QuickStart版本,https://downloads.cloudera.com/demo_vm/vmware/cloudera-quickstart-vm-5.12.0-0-vmware.zip

    4,用vmware player启动CDH,内存8G,CPU4个;root密码cloudera

    5,重新安装spark,下载命令 wget http://apache.mirrors.tds.net/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz

    下载的时候多下载几次,开始可能出现404问题;

    6,下载后配置spark,

    tar xzvf spark-2.0.0-bin-hadoop2.7.tgz 
    cd spark-2.0.0-bin-hadoop2.7
    
    
    vi /etc/profile.d/spark2.sh
    export SPARK_HOME=/home/cloudera/spark-2.0.0-bin-hadoop2.7
    export PATH=$PATH:/home/cloudera/spark-2.0.0-bin-hadoop2.7/bin
    
    
    cp conf/spark-env.sh.template conf/spark-env.sh
    cp conf/spark-defaults.conf.template conf/spark-defaults.conf
    
    vi conf/spark-env.sh
    export HADOOP_CONF_DIR=/etc/hadoop/conf
    export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
    
    cp /etc/hive/conf/hive-site.xml conf/
    
    修改conf/log4j.properties中的日志级别为ERROR
  • 相关阅读:
    Hive安装教程
    HBase安装教程
    Hadoop集群搭建
    Redis集群安装详细步骤
    Python绘图工具turtle库的使用
    python程序语法元素分析
    Selenium请求库爬取京东商品实例
    python爬虫入门
    python入门
    pytest fixture场景一:参数传入
  • 原文地址:https://www.cnblogs.com/stono/p/8323727.html
Copyright © 2011-2022 走看看