zoukankan      html  css  js  c++  java
  • mysql调优小记

    对于INNODB,主键就是聚集索引,如果没有主键定义,则第一个唯一非空索引被作为聚集索引.
    如果没有主键也没有合适的唯一索引,那么innodb内部会生成一个隐藏的主键作为聚集索引,
    这个隐藏的主键类似一个自增的id(int).

    删除与重新添加主键
    alter table tbname drop primary key;

    alter table tbname add primary key(col1,col2,col3);

    CREATE INDEX IDX_METE_BASE_AREA_STATION_CABINET ON CMS.T_CABINET_ELECTRICITY_HH_HIVE (
    BASE_NAME ASC,
    AREA_NAME ASC,
    STATION_NAME ASC,
    CABINET_NAME ASC
    );

    SELECT DISTINCT DEVICE_NAME ,REGEXP_EXTRACT(DEVICE_NAME,"-(.*)",0)
    FROM T_METE_CONFIG
    WHERE LSC_ID IN (30,31)
    AND DEVICE_TYPE_NAME='UPS配电'
    AND STATION_NAME LIKE '%数据机房%'
    ORDER BY DEVICE_NAME;

    SELECT REGEXP_EXTRACT('ASDFSADF-JOB','-([A-Z])',0) FROM CMS.DUAL LIMIT 1;

    复合分区示例,
    先以时间粒度天为例进行外分区,然后根据地域id进行分区,
    mysql仅支持range和list的子分区,而且子分区只能是hash或key.

    CREATE TABLE t_cabinet_electricity_hh_hive2
    (
       base_id int NOT NULL,
       base_name varchar(20),
       lsc_name varchar(255),
       area_id bigint,
       area_name varchar(255),
       station_id int NOT NULL,
       station_name varchar(255),
       device_type_id varchar(255),
       device_type_name varchar(255),
       device_id varchar(255) NOT NULL,
       device_name varchar(255),
       mete_id int NOT NULL,
       mete_name varchar(255),
       max_report_time varchar(20),
       max_mete_value double,
       min_report_time varchar(20),
       min_mete_value double,
       avg_mete_value double,
       hour_sum_value double,
       hour_times int,
       dt_hh varchar(10) NOT NULL,
       hour_last_value double,
       cabinet_name varchar(255),
       dt varchar(20) NOT NULL,
       lsc_id int NOT NULL,
       PRIMARY KEY (dt,dt_hh,base_id,lsc_id,mete_id,device_id,station_id,mete_name)
    )
    PARTITION BY RANGE(TO_DAYS(DT))
    SUBPARTITION BY HASH(LSC_ID)
    SUBPARTITIONS 6
    (
    PARTITION P0  VALUES LESS THAN (TO_DAYS('20170810')),
    PARTITION P1  VALUES LESS THAN (TO_DAYS('20170811')),
    PARTITION P2  VALUES LESS THAN (TO_DAYS('20170812')),
    PARTITION P3  VALUES LESS THAN (TO_DAYS('20170813')),
    PARTITION P4  VALUES LESS THAN (TO_DAYS('20170814')),
    PARTITION P5  VALUES LESS THAN (TO_DAYS('20170815')),
    PARTITION P6  VALUES LESS THAN (TO_DAYS('20170816')),
    PARTITION P7  VALUES LESS THAN (TO_DAYS('20170817')),
    PARTITION P8  VALUES LESS THAN (TO_DAYS('20170818')),
    PARTITION P9  VALUES LESS THAN (TO_DAYS('20170819')),
    PARTITION P10  VALUES LESS THAN (TO_DAYS('20170820'))
    )
    ;

    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P11  VALUES LESS THAN (TO_DAYS('20170821')));         
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P12  VALUES LESS THAN (TO_DAYS('20170822')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P13  VALUES LESS THAN (TO_DAYS('20170823')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P14  VALUES LESS THAN (TO_DAYS('20170824')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P15  VALUES LESS THAN (TO_DAYS('20170825')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P16  VALUES LESS THAN (TO_DAYS('20170826')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P17  VALUES LESS THAN (TO_DAYS('20170827')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P18  VALUES LESS THAN (TO_DAYS('20170828')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P19  VALUES LESS THAN (TO_DAYS('20170829')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P20  VALUES LESS THAN (TO_DAYS('20170830')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P21  VALUES LESS THAN (TO_DAYS('20170831')));
    ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P22  VALUES LESS THAN (TO_DAYS('20170901')));
            
    --查看分区信息
    select
    partition_name,table_rows
    from information_schema.partitions where table_name='xxx';
     
     
    问题,hive表中的日期类型 yyyy-mm-dd 到mysql中对应的date字段,但是无法导入成功
    其实看sqoop转换的java代码,可以看出这是sqoop的一个bug.,
    里面有类似 (java.sql.Date)value
    转换失败为空,这样进入我含有非空约束的字段就失败了.
    原因是日期格式是不确定的,所以直接转有问题,
    建议是直接设置临时表的字段为varchar,后面带做处理.
     

    sqoop eval --connect jdbc:mysql://xx:3306/cms --username root --password xxx
    --query "truncate table xx.stage_t_xx"

    sqoop export --connect jdbc:mysql://xxx:3306/xxx
    --username root --password xxx --table stage_t_xxe
    --hcatalog-database xxx--hcatalog-table stage_txxx

    sqoop eval --connect jdbc:mysql://xxx:3306/cms --username root --password xxx
    --query "insert into xxx.t_xxx_xx_hh_xx select * from xxx.stage_t_xx_xx_hh_hive"

  • 相关阅读:
    python3 使用opencv 画基本图形
    python 打印 九九表
    Python Date 1–Hello world print
    Linux下 Nginx+vsftpd搭建FTP服务器详细步骤
    Linux 开启端口命令
    No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK Maven异常解决方案
    React Fullpage
    Mint-UI组件 MessageBox为prompt 添加判断条件
    简易搭建本地静态服务器
    Mint-UI Picker 三级联动
  • 原文地址:https://www.cnblogs.com/huaxiaoyao/p/7429710.html
Copyright © 2011-2022 走看看