zoukankan      html  css  js  c++  java
  • Hive创建HBase,ES外部表

    1、创建HBase外部表

    CREATE EXTERNAL TABLE `ods_women`(
      `rowkey` string COMMENT 'from deserializer', 
      `article` string COMMENT 'from deserializer', 
      `url` string COMMENT 'from deserializer', 
      `web` string COMMENT 'from deserializer', 
      `keyword` string COMMENT 'from deserializer', 
      `acquire_time` string COMMENT 'from deserializer', 
      `article_time` string COMMENT 'from deserializer', 
      `calculate_time` string COMMENT 'from deserializer', 
      `title` string COMMENT 'from deserializer', 
      `english_industry` string COMMENT 'from deserializer')
    STORED BY 
      'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
    WITH SERDEPROPERTIES ( 
    'hbase.columns.mapping'=':key,info:article_word,info:article_url,info:website,info:chinese_keyword,info:acquire_time,info:article_time,info:calculate_time,info:article_title,info:english_industry')
    TBLPROPERTIES ( 'hbase.table.name'='test:ods_women');

    2、创建es外部表

    1)下载相关jar包

    https://www.elastic.co/cn/downloads/past-releases#es-hadoop

    2)hive命令行输入

    add jar /home/jar/elasticsearch-hadoop-5.5.3/dist/elasticsearch-hadoop-5.5.3.jar;

    3)创建表

    drop table if exists dw_women_article_core;
    create external table dw_women_article_core(
    md5id string,
    article_id string,
    keyword string,
    search_keyword string,
    keyword_weight bigint,
    article_title string,
    article_content string,
    web string,
    article_date string,
    status bigint,
    keyword_push string
    )
    STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
    TBLPROPERTIES('es.nodes' = '192.168.2.14:9200',
    'es.index.auto.create' = 'true',
    'es.resource' = 'app_knowledgegraph_new/app_women_article_core',
    'es.mapping.id' = 'md5id',
    'es.mapping.names' = 'md5id:md5id,article_id:article_id,keyword:keyword,search_keyword:search_keyword,keyword_weight:keyword_weight,article_title:article_title,web:web,article_date:article_date,status:status,keyword_push:keyword_push,article_content:article_content',
    'es.nodes.wan.only' = 'true');

    es.index.auto.create (default yes)

    Whether elasticsearch-hadoop should create an index (if its missing) when writing data to Elasticsearch or fail.

    es.nodes.wan.only (default false)

    Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and only connects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

  • 相关阅读:
    日期获取以及时间转化
    ddt 接口框架数据处理调用excel 处理
    ddt 测试用例UI运用
    动态验证码处理UI自动化获取处理
    Bug Report For .Net (zz.IS2120@BG57IV3)
    中关村翠湖科技园:高端产业聚集区 (zz.IS2120@BG57IV3.T752270541 .K)
    vc6,windows 7 x64 调试 (IS2120@BG57IV3)
    Excel c#Excel文件的操作[转载]
    NUnit学习 标签、方法 记录与说明
    Excel c#Excel工作进程的创建写 与Excel文件的保存[原创] (20100205 11:09)
  • 原文地址:https://www.cnblogs.com/qfdy123/p/13937686.html
Copyright © 2011-2022 走看看