zoukankan      html  css  js  c++  java
  • Hive创建HBase,ES外部表

    1、创建HBase外部表

    CREATE EXTERNAL TABLE `ods_women`(
      `rowkey` string COMMENT 'from deserializer', 
      `article` string COMMENT 'from deserializer', 
      `url` string COMMENT 'from deserializer', 
      `web` string COMMENT 'from deserializer', 
      `keyword` string COMMENT 'from deserializer', 
      `acquire_time` string COMMENT 'from deserializer', 
      `article_time` string COMMENT 'from deserializer', 
      `calculate_time` string COMMENT 'from deserializer', 
      `title` string COMMENT 'from deserializer', 
      `english_industry` string COMMENT 'from deserializer')
    STORED BY 
      'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
    WITH SERDEPROPERTIES ( 
    'hbase.columns.mapping'=':key,info:article_word,info:article_url,info:website,info:chinese_keyword,info:acquire_time,info:article_time,info:calculate_time,info:article_title,info:english_industry')
    TBLPROPERTIES ( 'hbase.table.name'='test:ods_women');

    2、创建es外部表

    1)下载相关jar包

    https://www.elastic.co/cn/downloads/past-releases#es-hadoop

    2)hive命令行输入

    add jar /home/jar/elasticsearch-hadoop-5.5.3/dist/elasticsearch-hadoop-5.5.3.jar;

    3)创建表

    drop table if exists dw_women_article_core;
    create external table dw_women_article_core(
    md5id string,
    article_id string,
    keyword string,
    search_keyword string,
    keyword_weight bigint,
    article_title string,
    article_content string,
    web string,
    article_date string,
    status bigint,
    keyword_push string
    )
    STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
    TBLPROPERTIES('es.nodes' = '192.168.2.14:9200',
    'es.index.auto.create' = 'true',
    'es.resource' = 'app_knowledgegraph_new/app_women_article_core',
    'es.mapping.id' = 'md5id',
    'es.mapping.names' = 'md5id:md5id,article_id:article_id,keyword:keyword,search_keyword:search_keyword,keyword_weight:keyword_weight,article_title:article_title,web:web,article_date:article_date,status:status,keyword_push:keyword_push,article_content:article_content',
    'es.nodes.wan.only' = 'true');

    es.index.auto.create (default yes)

    Whether elasticsearch-hadoop should create an index (if its missing) when writing data to Elasticsearch or fail.

    es.nodes.wan.only (default false)

    Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and only connects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

  • 相关阅读:
    [转贴]Nexus7的编译过程。
    Office:SmartArt使用
    jira:导出的Excel无法打开
    TDT:关键字驱动测试方法
    SVN:“SVN”不是内部命令,解决方法
    VS:如何在VS2010中运行控制台程序时停留在控制台显示窗口
    Total Cammander中容合SVN
    NSIS:NSIS基础语法
    AppTimer.exe:程序启动时间测试工具
    MySql安装与GUi安装
  • 原文地址:https://www.cnblogs.com/qfdy123/p/13937686.html
Copyright © 2011-2022 走看看