zoukankan      html  css  js  c++  java
  • hbase与hive集成:hive读取hbase中数据

    1、创建hbase jar包到hive lib目录软连接

    hive需要jar包:
    hive-hbase-handler-0.13.1-cdh5.3.6.jar
    zookeeper-3.4.5-cdh5.3.6.jar
    guava-12.0.1.jar                  --删除hive下低版本jar包,cp hbase中的到hive/lib/下(即需要注意jar包版本)

    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-server-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-server-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-client-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-client-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-protocol-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-protocol-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-it-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-it-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/htrace-core-2.04.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/htrace-core-2.04.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-hadoop2-compat-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-hadoop2-compat-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/hbase-hadoop-compat-0.98.6-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/hbase-hadoop-compat-0.98.6-cdh5.3.6.jar
    ln -s /opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6/lib/high-scale-lib-1.1.1.jar /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/lib/high-scale-lib-1.1.1.jar

    2、配置hive-site.xml文件,加入zookeeper属性:

    hbase.zookeeper.quorum

    3、配置环境变量:

    export HBASE_HOME=/opt/cdh-5.3.6/hbase-0.98.6-cdh5.3.6
    export HADOOP_HOME=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
    HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase mapredcp`:${HBASE_HOME}/conf

    4、hive集成hbase中dept表(外部表:hbase中已经存在的表,想要在hive中使用sql进行分析在hive应该创建外部表):

    CREATE EXTERNAL TABLE hbase_dept(
    deptno string,
    dname string,
    loc string)
    STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
    WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,info:dname,info:loc")
    TBLPROPERTIES ("hbase.table.name" = "dept");

  • 相关阅读:
    meta标签设置(移动端)
    清除浮动
    响应式设计
    堆和堆排序
    O(n^2)以及O(nlogn)时间复杂度的排序算法
    求数组的最大连续子数组和
    HTTP缓存原理
    将两个有序数组合并为一个有序数组
    如何实现居中对齐
    查找字符串中出现最多的字符
  • 原文地址:https://www.cnblogs.com/wcwen1990/p/7643978.html
Copyright © 2011-2022 走看看