zoukankan      html  css  js  c++  java
  • MapD表格导出测试

    gpu

    1.1万行数据测试:

    COPY (SELECT * FROM flights_2008_10k) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new.csv';

    验证:

    create table test (
    ..> flight_year smallint,
    ..> flight_month smallint,
    ..> flight_dayofmonth smallint,
    ..> flight_dayofweek smallint,
    ..> deptime smallint,
    ..> crsdeptime smallint,
    ..> arrtime smallint,
    ..> crsarrtime smallint,
    ..> uniquecarrier text encoding dict,
    ..> flightnum smallint,
    ..> tailnum text encoding dict,
    ..> actualelapsedtime smallint,
    ..> crselapsedtime smallint,
    ..> airtime smallint,
    ..> arrdelay smallint,
    ..> depdelay smallint,
    ..> origin text encoding dict,
    ..> dest text encoding dict,
    ..> distance smallint,
    ..> taxiin smallint,
    ..> taxiout smallint,
    ..> cancelled smallint,
    ..> cancellationcode text encoding dict,
    ..> diverted smallint,
    ..> carrierdelay smallint,
    ..> weatherdelay smallint,
    ..> nasdelay smallint,
    ..> securitydelay smallint,
    ..> lateaircraftdelay smallint,
    ..> dep_timestamp timestamp(0),
    ..> arr_timestamp timestamp(0),
    ..> carrier_name text encoding dict,
    ..> plane_type text encoding dict,
    ..> plane_manufacturer text encoding dict,
    ..> plane_issue_date date,
    ..> plane_model text encoding dict,
    ..> plane_status text encoding dict,
    ..> plane_aircraft_type text encoding dict,
    ..> plane_engine_type text encoding dict,
    ..> plane_year smallint,
    ..> origin_name text encoding dict,
    ..> origin_city text encoding dict,
    ..> origin_state text encoding dict,
    ..> origin_country text encoding dict,
    ..> origin_lat real,
    ..> origin_lon real,
    ..> dest_name text encoding dict,
    ..> dest_city text encoding dict,
    ..> dest_state text encoding dict,
    ..> dest_country text encoding dict,
    ..> dest_lat real,
    ..> dest_lon real,
    ..> origin_merc_x real,
    ..> origin_merc_y real,
    ..> dest_merc_x real,
    ..> dest_merc_y real
    ..> );

    copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new.csv' with (quoted='true');
    Result
    Loaded: 10000 recs, Rejected: 0 recs in 28.290000 secs

    2.20万行数据测试:

    COPY (SELECT * FROM flights_2008_200k) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new1.csv';

    验证:

    copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new1.csv' with (quoted='true');
    Result
    Loaded: 200000 recs, Rejected: 0 recs in 1.924000 secs

    3.700万行数据测试:

    COPY (SELECT * FROM flights_2008_7M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv';

    报错:GPU内存不足:E0628 10:31:13.320468 30484 MapDHandler.cpp:2384] Exception: Query couldn't keep the entire working set of columns in GPU memory
    Exception: Query couldn't keep the entire working set of columns in GPU memor

    cpu

    1.直接测700万行数据:

    COPY (SELECT * FROM flights_2008_7M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv';(花费时间:40-60分钟)

    验证:

    copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv' with (quoted='true');
    Result
    Loaded: 7009728 recs, Rejected: 0 recs in 60.079000 secs

    2.测2800万行数据:

    COPY (SELECT * FROM flights_2008_28M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv';(花费时间:259分钟)

    (%cpu:100   %mem:16)

    验证:

    copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv' with (quoted='true');

    Result
    Loaded: 28248912 recs, Rejected: 0 recs in 344.980000 secs

    (%cpu:3000-4000   %mem:6)

    3..测试1亿行:

    COPY (SELECT * FROM flights_2008_100M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv';

    报错:Thrift: Wed Jun 28 12:38:38 2017 TSocket::write_partial() send() <Host: localhost Port: 9091>Broken pipe
    Thrift: Wed Jun 28 12:38:38 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift: Wed Jun 28 12:38:38 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift error: connect() failed: Connection refused

    第二次尝试:

    Thrift: Wed Jun 28 13:38:09 2017 TSocket::write_partial() send() <Host: localhost Port: 9091>Broken pipe
    Thrift: Wed Jun 28 13:38:09 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift: Wed Jun 28 13:38:09 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
    Thrift error: connect() failed: Connection refused

  • 相关阅读:
    自己写的一个后台模板
    php计算登陆时间差
    友盟推送,php简单代码
    rolling_curl curl批量采集函数留份
    PHP socket服务框架
    JS 画饼图,折线图
    jquery 日期插件
    JS 获取当前页面url,及锚点
    PHP 读取文件,返回二进制流
    各个数据库DataSource配置文件
  • 原文地址:https://www.cnblogs.com/laozhuang/p/7089340.html
Copyright © 2011-2022 走看看