zoukankan      html  css  js  c++  java
  • 《OD大数据实战》Hive环境搭建

    一、搭建hadoop环境

    《OD大数据实战》hadoop伪分布式环境搭建

    二、Hive环境搭建

    1. 准备安装文件

    下载地址:

    http://archive.cloudera.com/cdh5/cdh/5/

    hive-0.13.1-cdh5.3.6.tar.gz

    2. 解压

    tar -zxvf hive-0.13.1-cdh5.3.6.tar.gz -C /opt/modules/cdh/

    3. 修改配置

    cd /opt/modules/cdh/hive-0.13.1-cdh5.3.6/conf
    mv hive-env.sh.template hive-env.sh
    mv hive-default.xml.template hive-site.xml
    mv hive-exec-log4j.properties.template hive-exec-log4j.properties 
    mv hive-log4j.properties.template hive-log4j.properties

    1)修改hive-env.sh

    #增加一行
    export JAVA_HOME=/opt/modules/jdk1.7.0_67
    
    HADOOP_HOME=/opt/modules/cdh/hadoop-2.5.0-cdh5.3.6
    
    export HIVE_CONF_DIR=/opt/modules/cdh/hive-0.13.1-cdh5.3.6/conf

    2)修改hive-log4j.properties

    hive.log.dir=/opt/modules/cdh/hive-0.13.1-cdh5.3.6/logs

    3)修改hive-exec-log4j.properties 

    hive.log.dir=/opt/modules/cdh/hive-0.13.1-cdh5.3.6/logs

    4)修改hive-site.xml

    <property>
      <name>hive.lazysimple.extended_boolean_literal</name>
      <value>false</value>
      <description>
        LazySiimpleSerde uses this properties to determine if it treats 'T', 't', 'F', 'f',
        '1', and '0' as extened, legal boolean literal, in addition to 'TRUE' and 'FALSE'.
        The default is false, which means only 'TRUE' and 'FALSE' are treated as legal
        boolean literal.
      </description>  
    </property>
    
    <property>
      <name>hive.mapjoin.optimized.hashtable</name>
      <value>true</value>
      <description>Whether Hive should use memory-optimized hash table for MapJoin. Only works on Tez, because memory-optimized hashtable cannot be serialized.</description>
    </property>

    4. 验证hive环境结果

    bin/hive
    dfs -ls /;

    三、mysql环境搭建

    1. 目标是安装mysql 5.1.17

    2. 在官网下载yum源

    http://dev.mysql.com/downloads/repo/yum/

    http://repo.mysql.com//mysql57-community-release-el6-8.noarch.rpm

    3. 安装yum源到/etc/yum.repos.d/目录

    sudo rpm -Uvh mysql57-community-release-el6-8.noarch.rpm

    cd /etc/yum.repos.d/

    4. 修改yum源配置

    修改文件:mysql-community.repo 和mysql-community-resource.repo

    5.6 enable = 1

    5.7 enable = 0

    5. 安装mysql

    sudo yum -y install mysql-community-server

    6. mysql安全性设置

    sudo mysql_secure_installation

    grant all privileges on *.* to 'root'@'%' identified by 'beifeng' with grant option

    7. 验证mysql安装结果

    进入命令行: mysql -uroot -p

    四、本地mysql作为metastore模式

    1. copy mysql驱动到${HIVE_HOME}/lib中

    cp mysql-connector-java-5.1.27-bin.jar /opt/modules/cdh/hive-0.13.1-cdh5.3.6/lib/

    2. 修改hive-site.xml

    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost:3306/cdh_hive_local_hive?createDatabaseIfNotExist=true</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
      <description>username to use against metastore database</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>beifeng</value>
      <description>password to use against metastore database</description>
    </property>

    3. 运行bin/hive命令

    4. 查看mysql数据库,发现多了一个cdh_hive_local_hive数据库 

    五、远程mysql作为metastore模式

    1. copy mysql驱动到${HIVE_HOME}/lib中

    cp mysql-connector-java-5.1.27-bin.jar /opt/modules/cdh/hive-0.13.1-cdh5.3.6/lib/

    2. 启动metastore服务器

    nohup hive --service metastore > /home/beifeng/hive_metastore.run.log 2>&1 &

    系统日志输出级别: 2 错误,1正常

    查看进程信息: ps -ef | grep HiveMetaStore

    关闭Hive

    kill -9 processId

    kill -9 `ps -ef | grep HiveMetaStore | awk '{print $2'} | head -n 1`

    3. 修改hive-site.xml

    <property>
      <name>hive.metastore.uris</name>
      <value>thrift://beifeng-hadoop-02:9083</value>
      <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost:3306/cdh_hive_remote_hive?createDatabaseIfNotExist=true</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>root</value>
      <description>username to use against metastore database</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>beifeng</value>
      <description>password to use against metastore database</description>
    </property>

    4. 运行bin/hive命令

    5. 查看mysql数据库,发现多了一个cdh_hive_local_hive数据库 

    六、JDBC连接hive

    1. 修改hive-site.xml

    <property>
      <name>hive.server2.thrift.port</name>
      <value>10000</value>
      <description>Port number of HiveServer2 Thrift interface.
      Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>
    </property>
    
    <property>
      <name>hive.server2.thrift.bind.host</name>
      <value>0.0.0.0</value>
      <description>Bind host on which to run the HiveServer2 Thrift interface.
      Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST</description>
    </property>

    2. 启动hiveserver2服务器

    nohup hive --service hiveserver2 > /home/beifeng/hiveserver2.run.log 2>&1 & 
    ps -ef | grep HiveServer2
    netstat -tlnup | grep 10000

    3. 进入beeline客户端

    beeline

    4. 连接hive

    beeline> !connect jdbc:hive2://beifeng-hadoop-02:10000
    scan complete in 5ms
    Connecting to jdbc:hive2://beifeng-hadoop-02:10000
    Enter username for jdbc:hive2://beifeng-hadoop-02:10000: beifeng
    Enter password for jdbc:hive2://beifeng-hadoop-02:10000: *******

    5. 修改配置

    <property>
      <name>hive.server2.long.polling.timeout</name>
      <value>5000</value>
      <description>Time in milliseconds that HiveServer2 will wait, before responding to asynchronous calls that use long polling</description>
    </property>
  • 相关阅读:
    Java实现网易163邮箱好友通讯录的解析功能(带源码)
    wordpress优化第四招 修改评论模板,留住客户,让评论在新的页面打开。
    wordpress优化 使用SAE提供的jquery.js替代wordpress原生的
    出售wordpress的淘宝客主题一套
    做了一个可以生成在线mp3 flash播放器的网站
    wordpress优化第三招 开启gzip减少网页流量
    20多个常用的免费WebService接口
    wordpress优化第一招 压缩css和js减少流量提高博客速度(尤其适用SAE)
    Linux学习笔记10常用操作命令(useradd命令、passwd 命令)
    Linux学习笔记08linux文本处理(cat命令、more命令、head命令、tail命令)
  • 原文地址:https://www.cnblogs.com/yeahwell/p/5726999.html
Copyright © 2011-2022 走看看