zoukankan      html  css  js  c++  java
  • MyCat 学习笔记 第六篇.数据分片 之 按月数据分片

    1 应用场景

    Mycat 有很多数据分库规则,接下来几篇就相关觉得常用的规则进行试用与总结。

    一般来说,按自然月份来进行数据分片的规则比较适用于商城订单查询,类似最近1周、2周、3个月内的数据。或是报表类应用。

    这样的数据放在一个片区内省去了数据合并的时间。

    当然按月数据量不要过大就OK。

     

    2 环境说明

    Windows 7   

    本机多数据库 Mysql 5.5.2  

    3306 端口下挂有4个库 : range_db_4、range_db_5、range_db_6、range_db_7

    3310 端口下挂有4个库 : range_db_8、range_db_9、range_db_10、range_db_11

    3 参数配置

    3.1 数据库配置

    mysql 客户端分别进入 3306 和 3310 服务,开始建立物理的schema。

    CREATE SCHEMA `range_db_4` DEFAULT CHARACTER SET utf8 ;
    CREATE SCHEMA `range_db_5` DEFAULT CHARACTER SET utf8 ;
    CREATE SCHEMA `range_db_6` DEFAULT CHARACTER SET utf8 ;
    CREATE SCHEMA `range_db_7` DEFAULT CHARACTER SET utf8 ;

    ...

    3306 上

    mysql> show databases;
    +--------------------+
    | Database |
    +--------------------+
    | information_schema |
    | mycat_sync_test |
    | mysql |
    | performance_schema |
    | range_db_4 |
    | range_db_5 |
    | range_db_6 |
    | range_db_7 |
    +--------------------+
    8 rows in set (0.00 sec)

    3310上

    mysql> show databases;
    +--------------------+
    | Database |
    +--------------------+
    | information_schema |
    | mycat_sync_test |
    | mysql |
    | performance_schema |
    | range_db_10 |
    | range_db_11 |
    | range_db_8 |
    | range_db_9 |
    | traveldata_db_1 |
    | traveldata_db_2 |
    +--------------------+
    10 rows in set (0.00 sec)

    3.2 server.xml 配置

    <!-- 开通test用户访问RANGEDB访问权限  RANGEDB是虚拟schema -->

    <user name="test">
      <property name="password">test</property>
      <property name="schemas">TRDB,RANGEDB</property> 
    </user>

    3.3 schema.xml 配置

    <!-- 设定虚拟 schema  RANGEDB 信息 -->

    <schema name="RANGEDB" checkSQLschema="false" sqlMaxLimit="100">

      <!-- 设定虚拟表 t_range_date 对应至数据结点 dn4:dn11 一共8个数据分片,使用 sharding-by-date  分片规则 -->
      <table name="t_range_date" dataNode="dn4,dn5,dn6,dn7,dn8,dn9,dn10,dn11" rule="sharding-by-date" />
    </schema>

    <!-- 设定数据结点dn4:dn7 对应的host为 3306服务 以及对应的物理schema   -->

    <dataNode name="dn4" dataHost="localhost3306" database="range_db_4" />
    <dataNode name="dn5" dataHost="localhost3306" database="range_db_5" />
    <dataNode name="dn6" dataHost="localhost3306" database="range_db_6" />
    <dataNode name="dn7" dataHost="localhost3306" database="range_db_7" />

    <!-- 设定数据结点dn8:dn11 对应的host为 3310 服务 以及对应的物理schema   -->

    <dataNode name="dn8" dataHost="localhost3310" database="range_db_8" />
    <dataNode name="dn9" dataHost="localhost3310" database="range_db_9" />
    <dataNode name="dn10" dataHost="localhost3310" database="range_db_10" />
    <dataNode name="dn11" dataHost="localhost3310" database="range_db_11" />

    <!-- 设定datahost 3306  目前只配了一台物理机,若要做读写分离可以参考开场第1、2篇内容进行调整 -->

    <dataHost name="localhost3306" maxCon="1000" minCon="10" balance="1"
      writeType="0" dbType="mysql" dbDriver="native" switchType="2" slaveThreshold="100">
      <heartbeat>select user()</heartbeat>
      <writeHost host="hostM3306" url="localhost:3306" user="root" password="root123"></writeHost>
    </dataHost>

    <!-- 设定datahost 3306 -->
    <dataHost name="localhost3310" maxCon="1000" minCon="10" balance="1"
      writeType="0" dbType="mysql" dbDriver="native" switchType="2" slaveThreshold="100">
      <heartbeat>select user()</heartbeat>
      <writeHost host="hostM3310" url="localhost:3310" user="root" password="root123"></writeHost>
    </dataHost>

    3.4 rule.xml 配置

    <!-- 分片字段对应到date_str 分片规则为partbymonth -->

    <tableRule name="sharding-by-date">
      <rule>
        <columns>date_str</columns>
        <algorithm>partbymonth</algorithm>
      </rule>
    </tableRule>

    <!-- 分片规则 partbymonth 的配置  从 2015 -01 -01 开始分片 -->

    <function name="partbymonth" class="org.opencloudb.route.function.PartitionByMonth">

      <property name="dateFormat">yyyy-MM-dd</property>
      <property name="sBeginDate">2015-01-01</property>
    </function>

    3.5 mycat 重新加载配置信息


    访问Mycat 9066 管理口,并重新加载所有参数配置。

    D:inmysqlMySQL_3307in>mysql -utest -ptest -P 9066
    Welcome to the MySQL monitor. Commands end with ; or g.
    Your MySQL connection id is 2
    Server version: 5.5.8-mycat-1.5-beta-20160111170158 MyCat Server (monitor)

    Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.

    Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

    mysql> reload @@config_all;
    Query OK, 1 row affected (0.36 sec)
    Reload config success

    4 数据验证

    4.1 Mycat 建表

    进入 Mycat 8066 服务口,选用 RANGEDB 库,同步create table。 

    D:inmysqlMySQL_3310in>mysql -utest -ptest -P 8066
    Welcome to the MySQL monitor. Commands end with ; or g.
    Your MySQL connection id is 1
    Server version: 5.5.8-mycat-1.5-beta-20160111170158 MyCat Server (OpenCloundDB)

    Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.

    Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

    mysql> use RANGEDB;
    Database changed
    mysql> CREATE TABLE `t_range_date` ( `id` INT NOT NULL, `date` DATE NULL, `date_str` VARCHAR(45) NULL, `context` VARCHAR(45) NULL, PRIMARY KEY (`id`));

    Query OK, 0 rows affected (0.09 sec)

    4.2 数据插入与查询

    由于只建立了8个分片,超出部分就直接抛数组越界异常了。

     mysql> insert into t_range_date (id,date_str,context) values(1,'2015-01-01','month-1-str');

    insert into t_range_date (id,date_str,context) values(2,'2015-02-01','month-2-str');

    insert into t_range_date (id,date_str,context) values(3,'2015-03-01','month-3-str');

    insert into t_range_date (id,date_str,context) values(4,'2015-04-01','month-4-str');

    insert into t_range_date (id,date_str,context) values(5,'2015-05-01','month-5-str');

    insert into t_range_date (id,date_str,context) values(6,'2015-06-01','month-6-str');

    insert into t_range_date (id,date_str,context) values(7,'2015-07-01','month-7-str');

    insert into t_range_date (id,date_str,context) values(8,'2015-08-01','month-8-str');

    insert into t_range_date (id,date_str,context) values(9,'2015-09-01','month-9-str');

    insert into t_range_date (id,date_str,context) values(10,'2015-10-01','month-10-str');

    insert into t_range_date (id,date_str,context) values(11,'2015-11-01','month-11-str');

    Query OK, 1 row affected (0.01 sec)

    Query OK, 1 row affected (0.01 sec)

    Query OK, 1 row affected (0.00 sec)

    Query OK, 1 row affected (0.00 sec)

    Query OK, 1 row affected (0.01 sec)

    Query OK, 1 row affected (0.00 sec)

    Query OK, 1 row affected (0.01 sec)

    Query OK, 1 row affected (0.00 sec)

    ERROR 1064 (HY000): Index: 8, Size: 8
    ERROR 1064 (HY000): Index: 9, Size: 8
    ERROR 1064 (HY000): Index: 10, Size: 8

    mysql> select * from t_range_date;
    +----+------+------------+-------------+
    | id | date | date_str | context |
    +----+------+------------+-------------+
    | 2 | NULL | 2015-02-01 | month-2-str |
    | 4 | NULL | 2015-04-01 | month-4-str |
    | 5 | NULL | 2015-05-01 | month-5-str |
    | 1 | NULL | 2015-01-01 | month-1-str |
    | 3 | NULL | 2015-03-01 | month-3-str |
    | 6 | NULL | 2015-06-01 | month-6-str |
    | 7 | NULL | 2015-07-01 | month-7-str |
    | 8 | NULL | 2015-08-01 | month-8-str |
    +----+------+------------+-------------+
    8 rows in set (0.01 sec)

    4.3 物理库查询

    前 4 个月份的数据进入 3306 服务的物理库

    mysql> select * from range_db_4.t_range_date;

    select * from range_db_5.t_range_date;

    select * from range_db_6.t_range_date;

    select * from range_db_7.t_range_date;

    +----+------+------------+-------------+
    | id | date | date_str | context |
    +----+------+------------+-------------+
    | 1 | NULL | 2015-01-01 | month-1-str |
    +----+------+------------+-------------+
    1 row in set (0.00 sec)

    +----+------+------------+-------------+
    | id | date | date_str | context |
    +----+------+------------+-------------+
    | 2 | NULL | 2015-02-01 | month-2-str |
    +----+------+------------+-------------+
    1 row in set (0.00 sec)

    +----+------+------------+-------------+
    | id | date | date_str | context |
    +----+------+------------+-------------+
    | 3 | NULL | 2015-03-01 | month-3-str |
    +----+------+------------+-------------+
    1 row in set (0.00 sec)

    +----+------+------------+-------------+
    | id | date | date_str | context |
    +----+------+------------+-------------+
    | 4 | NULL | 2015-04-01 | month-4-str |
    +----+------+------------+-------------+
    1 row in set (0.00 sec)

    5 优缺点分析

    1.可以做简单的按月分片,如果真要商起来,可以将一个季度的数据配置到相同的datanode 里去。

    2.不能按年进行循环配置,如果数据结点不足时需要提前加入,并手动清理历史数据。

    本篇完

  • 相关阅读:
    Python学习笔记2——数据类型
    Python学习笔记1——环境配置
    Dagger2 探索记3——两大进阶组件(二)
    NLP 计算机视觉 cv 机器学习 ,入们基础
    Android studio 常用快捷键
    uiautomator 1使用简介
    在Android源码中如何吧so库打包编译进入apk, 集成第三方库(jar和so库)
    android框架Java API接口总注释/**@hide*/和internal API
    Google android开发者 中国官方文档开放了呀
    android aidl通信 RemoteCallbackList客户端注册回调
  • 原文地址:https://www.cnblogs.com/kaye0110/p/5160826.html
Copyright © 2011-2022 走看看