zoukankan      html  css  js  c++  java
  • hive部署手册

    安装环境:    

          机器 只需要安装一台机器
          操作系统:Ubuntu 11.04 64操作系统
          hadoop:版本是1.0.2,安装在/usr/local/hadoop
          sun jdk:版本是1.6.0_31 64bit,安装在/usr/local/jdk
          hive:版本是0.8.1,安装在/usr/local/hive

    安装步骤:

    1.下载  

        下载hive:http://labs.mop.com/apache-mirror/hive/hive-0.8.1/hive-0.8.1.tar.gz


    2.安装

    (1)上传hive安装包到机器上,使用root用户登陆:   

     tar -xvf hive-0.8.1.tar.gz

    (2)将解压的hive分别移动并改名为/usr/local/hive

    rm -rf /usr/local/hive
    mv hive-0.8.1  /usr/local/hive

     

    3.配置hive

    (1)修改/usr/local/hive/bin/hive-config.sh

        在文件末尾加入

    export JAVA_HOME=/usr/local/jdk  
    export HIVE_HOME=/usr/local/hive  
    export HADOOP_HOME=/usr/local/hadoop  

    (2) 根据hive-default.xml复制hive-site.xml

    cp /usr/local/hive/conf/hive-default.xml /usr/local/hive/conf/hive-site.xml

    (3)配置hive-site.xml,主要配置项如下:
            hive.metastore.warehouse.dir:(HDFS上的)数据目录
            hive.exec.scratchdir:(HDFS上的)临时文件目录
            hive.metastore.warehouse.dir默认值是/user/hive/warehouse
            hive.exec.scratchdir默认值是/tmp/hive-${user.name}
            以上是默认值,暂时不改。
    (4)改变 /usr/local/hive的目录所有者为hadoop

    chown -R hadoop:hadoop /usr/local/hive

    (5)配置hive的log4j:
          cp /usr/loca/hive/conf/hive-log4j.properties.template  /usr/loca/hive/conf/hive-log4j.properties
       修改/usr/loca/hive/conf/hive-log4j.properties将org.apache.hadoop.metrics.jvm.EventCounter改为org.apache.hadoop.log.metrics.EventCounter
    (6)启动hive

          使用hadoop用户登陆,执行/usr/local/hive/bin/hive
    (7)测试hive

    hive> create TABLE pokes( id INT, name string);  
    hive> SHOW TABLES; 
    hive> select * from pokes; 
    hive> drop table pokes;

    4.优化hive

    默认meta数据库为derby ,为了避免使用默认的Derby数据库(有并发访问和性能的问题),通常还需要配置元数据库为MySQL

    修改配置文件conf/hive-site.xml 

    <property>
      <name>hive.metastore.local</name>
      <value>false</value>
      <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://mysql_server_host:3306/hivedb?createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=latin1</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>mysql_username</value>
      <description>username to use against metastore database</description>
    </property>
    
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>mysql_password</value>
      <description>password to use against metastore database</description>
    </property>
    
    <property>
      <name>hive.stats.dbconnectionstring</name>
      <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=latin1&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value>
      <description>The default connection string for the database that stores temporary hive statistics.</description>
    </property>
    
    <property>
      <name>hive.stats.dbconnectionstring</name>
      <value>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;characterEncoding=utf8&amp;user=mysql_username&amp;password=mysql_password&amp;createDatabaseIfNotExist=true</value>
      <description>The default connection string for the database that stores temporary hive statistics.</description>
    </property>
    
    <property>
      <name>hive.stats.dbclass</name>
      <value>jdbc:mysql</value>
      <description>The default database that stores temporary hive statistics.</description>
    </property>
    
    <property>
      <name>hive.stats.jdbcdriver</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>The JDBC driver for the database that stores temporary hive statistics.</description>
    </property>
    
    <property>
      <name>hive.metastore.uris</name>
      <value>thrift://127.0.0.1:9083</value>
    </property>

    添加metastore启动脚本bin/hive-metastore.sh

    #!/bin/sh
    nohup ./hive --service metastore >> metastore.log 2>&1 &
    echo $! > hive-metastore.pid

    添加hive server启动脚本bin/hive-server.sh

    nohup ./hive --service hiveserver >> hiveserver.log 2>&1 &
    echo $! > hive-server.pid

    启动metastore和hive server

    ./hive-metastore.sh
    ./hive-server.sh

    使用客户端连接和测试

    参考之前的hive测试步骤一一执行。

    如果出现
    FAILED: Error in metadata: javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.
    NestedThrowables:
    com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    或者是
    FAILED: Error in metadata: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

    以上错误,主要是在select操作、drop操作时,就会出错。

    这样的错误,可以修改hivedb数据库编码:

    mysql> alter database hivedb character set latin1;

    然后重启hive metastore和hive server就可以了

    mysql数据库的编码目前最好设置为latin1,否则使用hive会出现莫名其妙的问题

  • 相关阅读:
    macbook 无声音解决方案
    webapck dev server代理请求 json截断问题
    百度卫星地图开启
    服务器 nginx配置 防止其他域名绑定自己的服务器
    记一次nginx php配置的心路历程
    遇到npm报错read ECONNRESET怎么办
    运行svn tortoiseSvn cleanup 命令失败的解决办法
    svn add 命令 递归目录下所有文件
    m4出现Please port gnulib freadahead.c to your platform! Look at the definition of fflush, fread, ungetc on your system, then report this to bug-gnulib."
    Ubuntu下安装GCC,mpc、mpfr、gmp
  • 原文地址:https://www.cnblogs.com/ggjucheng/p/2471341.html
Copyright © 2011-2022 走看看