zoukankan      html  css  js  c++  java
  • 从MySQL向Greenplum集群中导入数据

    我们要从MySQL当中导出数据到Greenplum当中,按照以下步骤就可以

    1:将MySQL当中的表导出外部文件

    以schema_name.table_name为例

    select 
     product_id, number, name, english_name, purchase_name, system_name, bar_code, category_one, category_two, category_three,
     parent_id, parent_number, brand_id, supplier_id, price, ad_word, give_integral, shelf_life, FROM_UNIXTIME(shelve_date), product_area, country,
     sale_unit, specification, weight, length, width, height, storage_conditions, storage, model, refuse_notes, status, is_promote, 
     is_gift, is_book, is_outgoing, is_presale, is_fragile, is_have, is_cod, is_return, is_oos, is_seasonal, is_multicity, is_package, is_show, click,
     favorite, min_purchase_unit, in_price, refer_in_price, mwaverage_price, is_unique_number, is_batch_number, qs_proportion, shelf_life_proportion, box_specification,
     max_unsalable, advent_shelves, pro_warning, FROM_UNIXTIME(add_time), operator_id,FROM_UNIXTIME( audit_time), remark, price_type, new_tag, product_type, business_model, is_sell, return_policy,
     package, inventory, merchant_number, modified_time ,now()
     from schema_name.table_name  INTO OUTFILE '/tmp/table_name.txt';

    导的时候需要注意,一些字符的转换,对于这张表来说,主要就是在MySQL当中一些时间格式存储的为INT类型,我们需要进行转化后然后导出,而且在Greenplum当中建表的时候会多一个时间字段,我们这里默认导出现在时间。按照以上格式进行导出。

    2:将文件拷贝到Greenplum服务器上,并且创建外部表

    先将文件拷贝到外部表的目录下,这个比较简单,什么方法都可以,然后创建外部表:

    create external  TABLE  schema_name.table_name_ext( product_id int,
    number varchar(10),
    name varchar(100),
    english_name varchar(100),
    purchase_name varchar(100),
    system_name varchar(100),
    bar_code varchar(255),
    category_one int,
    category_two int,
    category_three int,
    parent_id int,
    parent_number int,
    brand_id int,
    supplier_id int,
    price int,
    ad_word varchar(100),
    give_integral int,
    shelf_life int,
    shelve_date timestamp without time zone,
    product_area int,
    country int,
    sale_unit varchar(20),
    specification varchar(255),
    weight decimal(10,2) ,
    length int,
    width int,
    height int,
    storage_conditions varchar(255),
    storage smallint,
    model varchar(20),
    refuse_notes varchar(255),
    status smallint,
    is_promote smallint,
    is_gift smallint,
    is_book smallint,
    is_outgoing smallint,
    is_presale int,
    is_fragile smallint,
    is_have smallint,
    is_cod smallint,
    is_return smallint,
    is_oos smallint,
    is_seasonal smallint,
    is_multicity smallint,
    is_package smallint,
    is_show smallint,
    click int,
    favorite int,
    min_purchase_unit int,
    in_price int,
    refer_in_price int,
    mwaverage_price int,
    is_unique_number int,
    is_batch_number int,
    qs_proportion int,
    shelf_life_proportion DOUBLE PRECISION,
    box_specification varchar(50),
    max_unsalable int,
    advent_shelves int,
    pro_warning int,
    add_time timestamp without time zone,
    operator_id int,
    audit_time timestamp without time zone,
    remark varchar(255),
    price_type smallint,
    new_tag int,
    product_type int,
    business_model smallint,
    is_sell smallint,
    return_policy smallint,
    package varchar(200),
    inventory varchar(200),
    merchant_number int,
    modified_time timestamp without time zone,
    dw_modified_time timestamp without time zone
    )  location(
    'gpfdist://172.16.16.34:9888/table_name.txt' ) 
    FORMAT
    'TEXT' SEGMENT REJECT LIMIT 1000000 rows ;
    这里我们要指定'gpfdist://172.16.16.34:9888/table_name.txt',这个IP地址加上外部表就可以了,后面要把这个文件拷贝到 gpfdist 的目录当中,我们看下启动方式gpfdist -d /tmp -p 9888,也就是要把外部文件拷贝到/tmp目录下才可以。其他的注意列名对应就好
    然后查询一下,一般情况列对上就不会有问题。
    3:导入到Greenplum当中正式表

    先创建一张正式表:

    create table schema_name.table_name ( product_id int,
    number varchar(10),
    name varchar(100),
    english_name varchar(100),
    purchase_name varchar(100),
    system_name varchar(100),
    bar_code varchar(255),
    category_one int,
    category_two int,
    category_three int,
    parent_id int,
    parent_number int,
    brand_id int,
    supplier_id int,
    price int,
    ad_word varchar(100),
    give_integral int,
    shelf_life int,
    shelve_date timestamp without time zone,
    product_area int,
    country int,
    sale_unit varchar(20),
    specification varchar(255),
    weight decimal(10,2) ,
    length int,
    width int,
    height int,
    storage_conditions varchar(255),
    storage smallint,
    model varchar(20),
    refuse_notes varchar(255),
    status smallint,
    is_promote smallint,
    is_gift smallint,
    is_book smallint,
    is_outgoing smallint,
    is_presale int,
    is_fragile smallint,
    is_have smallint,
    is_cod smallint,
    is_return smallint,
    is_oos smallint,
    is_seasonal smallint,
    is_multicity smallint,
    is_package smallint,
    is_show smallint,
    click int,
    favorite int,
    min_purchase_unit int,
    in_price int,
    refer_in_price int,
    mwaverage_price int,
    is_unique_number int,
    is_batch_number int,
    qs_proportion int,
    shelf_life_proportion DOUBLE PRECISION,
    box_specification varchar(50),
    max_unsalable int,
    advent_shelves int,
    pro_warning int,
    add_time timestamp without time zone,
    operator_id int,
    audit_time timestamp without time zone,
    remark varchar(255),
    price_type smallint,
    new_tag int,
    product_type int,
    business_model smallint,
    is_sell smallint,
    return_policy smallint,
    package varchar(200),
    inventory varchar(200),
    merchant_number int,
    modified_time timestamp without time zone,
    dw_modified_time timestamp without time zone
    ) distributed by(product_id);

    然后导入数据:

    insert into schema_name.table_name
    select * from schema_name.table_name_ext

     这样就把外部表数据导出到了内部表,均匀分布在每个segment上。注意schema_name.table_name的结构要和schema_name.table_name_ext是一致的。

  • 相关阅读:
    Android WiFi系统【转】
    TCP服务器端和客户端程序设计【转】
    Ubuntu apt-get update 失败【转】
    AiCloud 2.0 AT开发文档【转】
    【转】简单几步让App Store软件下载快如迅雷 -- 不错!!!
    【转】Mac用户必备!100多款免费实用的苹果Mac软件大搜集
    【转】Mac QQ截图保存在哪里?
    【转】Xcode7.1环境下上架iOS App到AppStore 流程 (Part 三)
    【转】Xcode7.1环境下上架iOS App到AppStore 流程 (Part 二)
    【转】Xcode7.1环境下上架iOS App到AppStore 流程 (Part 一)
  • 原文地址:https://www.cnblogs.com/shengdimaya/p/8762059.html
Copyright © 2011-2022 走看看