zoukankan      html  css  js  c++  java
  • Hive创建HBase,ES外部表

    1、创建HBase外部表

    CREATE EXTERNAL TABLE `ods_women`(
      `rowkey` string COMMENT 'from deserializer', 
      `article` string COMMENT 'from deserializer', 
      `url` string COMMENT 'from deserializer', 
      `web` string COMMENT 'from deserializer', 
      `keyword` string COMMENT 'from deserializer', 
      `acquire_time` string COMMENT 'from deserializer', 
      `article_time` string COMMENT 'from deserializer', 
      `calculate_time` string COMMENT 'from deserializer', 
      `title` string COMMENT 'from deserializer', 
      `english_industry` string COMMENT 'from deserializer')
    STORED BY 
      'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
    WITH SERDEPROPERTIES ( 
    'hbase.columns.mapping'=':key,info:article_word,info:article_url,info:website,info:chinese_keyword,info:acquire_time,info:article_time,info:calculate_time,info:article_title,info:english_industry')
    TBLPROPERTIES ( 'hbase.table.name'='test:ods_women');

    2、创建es外部表

    1)下载相关jar包

    https://www.elastic.co/cn/downloads/past-releases#es-hadoop

    2)hive命令行输入

    add jar /home/jar/elasticsearch-hadoop-5.5.3/dist/elasticsearch-hadoop-5.5.3.jar;

    3)创建表

    drop table if exists dw_women_article_core;
    create external table dw_women_article_core(
    md5id string,
    article_id string,
    keyword string,
    search_keyword string,
    keyword_weight bigint,
    article_title string,
    article_content string,
    web string,
    article_date string,
    status bigint,
    keyword_push string
    )
    STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' 
    TBLPROPERTIES('es.nodes' = '192.168.2.14:9200',
    'es.index.auto.create' = 'true',
    'es.resource' = 'app_knowledgegraph_new/app_women_article_core',
    'es.mapping.id' = 'md5id',
    'es.mapping.names' = 'md5id:md5id,article_id:article_id,keyword:keyword,search_keyword:search_keyword,keyword_weight:keyword_weight,article_title:article_title,web:web,article_date:article_date,status:status,keyword_push:keyword_push,article_content:article_content',
    'es.nodes.wan.only' = 'true');

    es.index.auto.create (default yes)

    Whether elasticsearch-hadoop should create an index (if its missing) when writing data to Elasticsearch or fail.

    es.nodes.wan.only (default false)

    Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and only connects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

  • 相关阅读:
    Apache服务器的简单配置与安全策略
    Linux下的ICMP反弹后门:PRISM
    项目年度任务失败总结
    SpringBoot下配置Druid
    ftm国际化解决方案
    SpringBoot自动装配源码解析
    log4j到log4j2升级迁移方案
    Linux常用命令记录
    MySQL安装后无法用root用户访问的问题
    html实体命名
  • 原文地址:https://www.cnblogs.com/qfdy123/p/13937686.html
Copyright © 2011-2022 走看看