一、问题发生
说一个现象，当收到服务器报警之后，数据库服务器CPU使用超过90%，通过 show processlist 一看，满屏都是 Waiting for table metadata lock 状态的连接。第一反应就是kill掉这些连接，奈何连接实在太多，实在kill不过来，于是重启服务，重启果真能解决90%的问题，但如果不找到问题原因，问题也肯定会再次出现。
类似于下图（没找到Waiting for table metadata lock的示例，将就看下）
Id User Host db Command Time State Info Memory_used Memory_used_by_query Logical_read Physical_sync_read Physical_async_read Temp_user_table_size Temp_sort_table_size Temp_sort_file_size 1277733 data01 172.17.22.204:26173 birds Query 50 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2275632 12912 1012951 1 0 0 0 0 1277751 data01 172.17.22.204:26745 birds Query 49 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2245288 12912 936956 4 0 0 0 0 1277802 data01 172.17.22.204:29587 birds Query 58 Creating sort index SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2584048 12912 761473 2 0 0 0 0 1277825 data01 172.17.22.204:30637 birds Query 57 Creating sort index SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 1414176 12912 751966 0 0 0 0 0 1277827 data01 172.17.22.204:30639 birds Query 56 Creating sort index SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2497392 12912 757188 0 0 0 0 0 1277828 data01 172.17.22.204:30641 birds Query 57 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 1134712 12912 739028 0 0 0 0 0 1277829 data01 172.17.22.204:30723 birds Query 56 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 1134712 12912 725881 0 0 0 0 0 1277830 data01 172.17.22.204:30721 birds Query 55 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2238288 12912 732720 0 0 0 0 0 1277831 data01 172.17.22.204:30765 birds Query 56 Creating sort index SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2497520 12912 757099 0 0 0 0 0 1277832 data01 172.17.22.204:30801 birds Query 55 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 693261 0 0 0 0 0 1277879 data01 172.17.22.204:32905 birds Query 34 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '300780' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536 8544 462365 0 0 0 0 0 1277880 data01 172.17.22.204:32907 birds Query 33 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 425506 3 0 0 0 0 1277881 data01 172.17.22.204:32993 birds Query 29 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '185711' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1138896 8544 483586 1 0 0 0 0 1277914 data01 172.17.22.204:34571 birds Query 25 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2217736 12912 315580 0 0 0 0 0 1277915 data01 172.17.22.204:34573 birds Query 0 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '557415' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1138640 8544 283988 0 0 0 0 0 1277916 data01 172.17.22.204:34677 birds Query 25 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 309707 0 0 0 0 0 1277918 data01 172.17.22.204:34747 birds Query 25 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 299223 0 0 0 0 0 1277919 data01 172.17.22.204:34805 birds Sleep 0 NULL 1123104 8208 283855 3 0 0 0 0 1277920 data01 172.17.22.204:34829 birds Query 6 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '512151' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536 8544 305759 5 0 0 0 0 1277922 data01 172.17.22.204:34909 birds Query 20 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 253827 0 0 0 0 0 1277925 data01 172.17.22.204:35133 birds Query 21 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2218056 12912 276768 0 0 0 0 0 1277929 data01 172.17.22.204:35319 birds Query 20 Sending data SELECT a.tid,a.subject,a.fid,b.name FROM pre_forum_thread as a inner join pre_forum_forum as b on a.fid=b.fid where a.displayorder=0 and a.fid<>302 and a.fid<> 187 and a.fid<> 140 and a.fid<> 139 ORDER BY a.tid DESC LIMIT 0,500 2217928 12912 251582 0 0 0 0 0 1277971 data01 172.17.22.204:36879 birds Query 3 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '489209' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536 8544 45250 0 0 0 0 0 1277972 data01 172.17.22.204:36881 birds Query 3 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '187583' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1150288 8544 40179 0 0 0 0 0 1277981 data01 172.17.22.204:37213 birds Query 0 Sending data SELECT tid,subject FROM pre_forum_thread WHERE tid < '194152' AND fid = '302' AND displayorder >= 0 ORDER BY tid DESC limit 1 1139536 8544 7289 0 0 0 0 0 1277982 data01 172.17.22.204:37231 birds Query 0 Writing to net SELECT * FROM pre_forum_post WHERE tid=585078 AND position>=14361 AND position<14371 ORDER BY position 1164064 10280 374 0 0 0 0 0 1277984 data01 172.17.22.204:37285 NULL Query 0 init show full processlist 73800 8472 0 0 0 0 0 0
原因是什么：
MySQL在进行一些alter table等DDL操作时，如果该表上有未提交的事务则会出现 Waiting for table metadata lock ，而一旦出现metadata lock，该表上的后续操作都会被阻塞，由于当时直接把ALTER给kill了，所以基本没有现场
二、关于Table Metadata Lock
1、为什么会有 Metadata Lock
源起于一个bug，MySQL官方文档链接：http://bugs.mysql.com/bug.php?id=989
Description:
If user1 has an active transaction on a table and then user2 drops this table, then user1 does COMMIT, then in the binlog we have something like:
DROP TABLE t;
BEGIN;
INSERT INTO t ... ;
COMMIT;
which is wrong.
这个bug大致意思是说：当一个会话在主库执行DML操作还没提交时，另一个会话对同一个对象执行了DDL操作如drop table，而由于MySQL的binlog是基于事务提交的先后顺序进行记录的，因此在从库上应用时，就出现了先drop table，然后再向table中insert的情况，导致从库应用出错。
因此，MySQL在5.5.3版本后引入了Metadata lock，只有在事务结束后才会释放Metadata lock，因此在事务提交或回滚前，是无法进行DDL操作的。
2、什么是Table Metadata Lock
首先，看看官方的说法，
MySQL官方文档位置：http://dev.mysql.com/doc/refman/5.6/en/metadata-locking.html
To ensure transaction serializability, the server must not permit one session to perform a data definition language (DDL) statement on a table that is used in an uncompleted explicitly or implicitly started transaction in another session.
The server achieves this by acquiring metadata locks on tables used within a transaction and deferring release of those locks until the transaction ends.
A metadata lock on a table prevents changes to the table's structure.
This locking approach has the implication that a table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends.
从上面的描述可以看到，
-
MDL出现的初衷就是为了保护一个处于事务中的表的结构不被修改。
-
这里提到的事务包括两类，显式事务和AC-NL-RO（auto-commit non-locking read-only）事务。显式事务包括两类：1. 关闭AutoCommit下的操作，2. 以begin或start transaction开始的操作。AC-NL-RO可理解为AutoCommit开启下的select操作。
-
MDL是事务级别的，只有在事务结束后才会释放。在此之前，其实也有类似的保护机制，只不过是语句级别的。
需要注意的是，MDL不仅仅适用于表，同样也适用于其它对象，如下表所示，其中，"等待状态"对应的是"show processlist"中的State。
为了提高数据库的并发度，MDL被细分为了11种类型。
-
MDL_INTENTION_EXCLUSIVE
-
MDL_SHARED
-
MDL_SHARED_HIGH_PRIO
-
MDL_SHARED_READ
-
MDL_SHARED_WRITE
-
MDL_SHARED_WRITE_LOW_PRIO
-
MDL_SHARED_UPGRADABLE
-
MDL_SHARED_READ_ONLY
-
MDL_SHARED_NO_WRITE
-
MDL_SHARED_NO_READ_WRITE
-
MDL_EXCLUSIVE
常用的有MDL_SHARED_READ,MDL_SHARE D_WRITE及MDL_EXCLUSIVE，其分别用于SELECT操作，DML操作及DDL操作。其它类型的对应操作可参考源码sql/mdl.h。
对于MDL_EXCLUSIVE，官方的解释是，
An exclusive metadata lock.
A connection holding this lock can modify both table's metadata and data.
No other type of metadata lock can be granted while this lock is held.
To be used for CREATE/DROP/RENAME TABLE statements and for execution of certain phases of other DDL statements.
简而言之，MDL_EXCLUSIVE是独占锁，在其持有期间是不允许其它类型的MDL被授予，自然也包括SELECT和DML操作。
这也就是为什么DDL操作被阻塞时，后续其它操作也会被阻塞。
关于MDL的补充
1. MDL的最大等待时间由lock_wait_timeout参数决定，其默认值为31536000（365天）。在使用工具进行DDL操作时，这个值就不太合理。事实上，pt-online-schema-change和gh-ost对其就进行了相应的调整，其中，前者60s，后者3s。
2. 如果一个SQL语法上有效，但执行时报错，如，列名不存在，其同样会获取MDL锁，直到事务结束才释放。
三、Metadata Lock解决了什么问题
MySQL 从5.5.3版本，对Metadata lock进行了调整，主要是MDL锁持有的周期从语句变成了事务， 其原因主要是解决两个问题：
-
问题1: 破坏事务隔离级别 在repeatable read的隔离级别下，多次的select语句执行过程中，会因为其它session的DDL语句，而导致select语句执行的结果不相同，破坏了RR的隔离级别。
-
问题2: 破坏binlog的顺序 在对表的DML过程中，会因为其它session的DDL语句，导致binlog里的event顺序在备库执行的结果和主库不一致。
从MySQL 5.5.3开始，MDL锁的持有周期变成了事务，解决了上面提到的两个问题，但在autocommit=off的情况下，也大大增加了阻塞的可能性。DBA对于阻塞的case，处理起来又比较麻烦，原因就是MDL锁的阻塞情况没有暴露明确的信息。
从MySQL 5.7.6开始，可以通过performance schema来查询MDL锁的持有情况。
RR事务隔离级别下不可重复读的问题，演示环境，MySQL 5.5.0。
session1> begin; Query OK, 0 rows affected (0.00 sec) session1> select * from t1; +------+------+ | id | name | +------+------+ | 1 | a | | 2 | b | +------+------+ 2 rows in set (0.00 sec) session2> alter table t1 add c1 int; Query OK, 2 rows affected (0.02 sec) Records: 2 Duplicates: 0 Warnings: 0 session1> select * from t1; Empty set (0.00 sec) session1> commit; Query OK, 0 rows affected (0.00 sec) session1> select * from t1; +------+------+------+ | id | name | c1 | +------+------+------+ | 1 | a | NULL | | 2 | b | NULL | +------+------+------+ 2 rows in set (0.00 sec)
可以看到，虽然是RR隔离级别，但在开启事务的情况下，第二次查询却没有结果
主从复制问题
包括主从数据不一致，主从复制中断等。 如下面的主从数据不一致。
session1> create table t1(id int,name varchar(10)) engine=innodb; Query OK, 0 rows affected (0.00 sec) session1> begin; Query OK, 0 rows affected (0.00 sec) session1> insert into t1 values(1,'a'); Query OK, 1 row affected (0.00 sec) session2> truncate table t1; Query OK, 0 rows affected (0.46 sec) session1> commit; Query OK, 0 rows affected (0.35 sec) session1> select * from t1; Empty set (0.00 sec)
再来看看从库的结果
session1> select * from t1; +------+------+------+ | id | name | c1 | +------+------+------+ | 1 | a | NULL | +------+------+------+ 1 row in set (0.00 sec)
看看binlog的内容，可以看到，truncate操作记录在前，insert操作记录在后。
# at 7140 #180714 19:32:14 server id 1 end_log_pos 7261 Query thread_id=31 exec_time=0 error_code=0 SET TIMESTAMP=1531567934/*!*/; create table t1(id int,name varchar(10)) engine=innodb /*!*/; # at 7261 #180714 19:32:30 server id 1 end_log_pos 7333 Query thread_id=32 exec_time=0 error_code=0 SET TIMESTAMP=1531567950/*!*/; BEGIN /*!*/; # at 7333 #180714 19:32:30 server id 1 end_log_pos 7417 Query thread_id=32 exec_time=0 error_code=0 SET TIMESTAMP=1531567950/*!*/; truncate table t1 /*!*/; # at 7417 #180714 19:32:30 server id 1 end_log_pos 7444 Xid = 422 COMMIT/*!*/; # at 7444 #180714 19:32:34 server id 1 end_log_pos 7516 Query thread_id=31 exec_time=0 error_code=0 SET TIMESTAMP=1531567954/*!*/; BEGIN /*!*/; # at 7516 #180714 19:32:24 server id 1 end_log_pos 7611 Query thread_id=31 exec_time=0 error_code=0 SET TIMESTAMP=1531567944/*!*/; insert into t1 values(1,'a') /*!*/; # at 7611 #180714 19:32:34 server id 1 end_log_pos 7638 Xid = 421 COMMIT/*!*/;
如果会话2执行的是drop table操作，还会导致主从中断。
有意思的是，如果会话2执行的是alter table操作，其依旧会被阻塞，阻塞时间受innodb_lock_wait_timeout参数限制。
mysql> show processlist; +----+------+-----------+----------+---------+------+-------------------+---------------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------+----------+---------+------+-------------------+---------------------------+ | 54 | root | localhost | NULL | Query | 0 | NULL | show processlist | | 58 | root | localhost | slowtech | Sleep | 1062 | | NULL | | 60 | root | localhost | slowtech | Query | 11 | copy to tmp table | alter table t1 add c1 int | +----+------+-----------+----------+---------+------+-------------------+---------------------------+ 3 rows in set (0.00 sec)
可以看到没有MDL锁的情况下，会出现以上两种问题，当有MDL锁保护之后，就正常了
InnoDB层已经有了IS、IX这样的意向锁，有同学觉得可以用来实现上述例子的并发控制。但由于MySQL是Server-Engine架构，所以MDL锁是在Server中实现。另外，MDL锁还能实现其他粒度级别的锁，比如全局锁、库级别的锁、表空间级别的锁，这是InnoDB存储引擎层不能直接实现的锁。
但与InnoDB锁的实现一样，MDL锁也是类似对一颗树的各个对象从上至下进行加锁（对树进行加锁具体见：《MySQL技术内幕：InnoDB存储引擎》）。但是MDL锁对象的层次更多，简单来看有如下的层次：
上图中显示了最常见的4种MDL锁的对象，并且注明了常见的SQL语句会触发的锁。与InnoDB层类似的是，某些类型的MDL锁会从上往下一层层进行加锁。比如LOCK TABLE … WRITE这样的SQL语句，其首先会对GLOBAL级别加INTENTION_EXCLUSIVE锁，再对SCHEMA级别加INTENTION_EXCLUSIVE锁，最后对TABLE级别加SHARED_NO_READ_WRITE锁。这里最令人意外的是还有COMMIT对象层次的锁，其实这主要用于XA事务中。比如分布式事务已经PREPARE成功，但是在XA COMMIT之前有其他会话执行了FLUSH TABLES WITH READ LOCK这样的操作，那么分布式事务的提交就需要等待。
除了上图标注的对象，其实还有TABLESPACE、FUNCTION、PROCEDURE、EVENT等其他对象类型，其实都是为了进行并发控制。只是这些在MySQL数据库中都不常用，故不再赘述（当然也是为了偷懒）。
GLOBAL=0,TABLESPACE,SCHEMA,TABLE,FUNCTION,PROCEDURE,TRIGGER,EVENT,COMMIT,USER_LEVEL_LOCK,LOCKING_SERVICE,NAMESPACE_END
目前MDL有如下锁模式，锁之间的兼容性可见源码mdl.cc：
| 锁模式 | 对应SQL |
| MDL_INTENTION_EXCLUSIVE | GLOBAL对象、SCHEMA对象操作会加此锁 |
| MDL_SHARED | FLUSH TABLES with READ LOCK |
| MDL_SHARED_HIGH_PRIO | 仅对MyISAM存储引擎有效 |
| MDL_SHARED_READ | SELECT查询 |
| MDL_SHARED_WRITE | DML语句 |
| MDL_SHARED_WRITE_LOW_PRIO | 仅对MyISAM存储引擎有效 |
| MDL_SHARED_UPGRADABLE | ALTER TABLE |
| MDL_SHARED_READ_ONLY | LOCK xxx READ |
| MDL_SHARED_NO_WRITE | FLUSH TABLES xxx,yyy,zzz READ |
| MDL_SHARED_NO_READ_WRITE | FLUSH TABLE xxx WRITE |
| MDL_EXCLUSIVE | ALTER TABLE xxx PARTITION BY … |
MDL锁的性能与并发改进
讲到这同学们会发现MDL锁的开销并不比InnoDB层的行锁要小，而且这可能是一个更为密集的并发瓶颈。MySQL 5.6和5.5版本通常通过调整如下两个参数来进行并发调优:
-
metadata_locks_cache_size: MDL锁的缓存大小
-
metadata_locks_hash_instances：通过分片来提高并发度，与InnoDB AHI类似
MySQL 5.7 MDL锁的最大改进之处在于将MDL锁的机制通过lock free算法来实现，从而提高了在多核并发下数据库的整体性能提升。
MDL锁的诊断
MySQL 5.7版本之前并没有提供一个方便的途径来查看MDL锁，github上有一名为mysql-plugin-mdl-info的项目，通过插件的方式来查看，非常有想法的实现，大赞。好在官方也意识到了这个问题，于是在MySQL 5.7中的performance_schea库下新增了一张表metadata_locks，用其来查看MDL锁那是相当的方便:
不过默认PS并没有打开此功能，需要手工将wait/lock/metadata/sql/mdl监控给打开
SELECT * FROM performance_schema.setup_instruments; UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME ='global_instrumentation'; UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME ='wait/lock/metadata/sql/mdl'; select * from performance_schema.metadata_locksG
参考文章
http://mysql.taobao.org/monthly/2015/10/02/
http://www.yunweipai.com/archives/19818.html
https://www.jianshu.com/p/683bd7fcb7a0
https://www.cnblogs.com/zengkefu/p/5690385.html
https://www.cnblogs.com/cchust/p/3826398.html
https://www.cnblogs.com/xpchild/p/3789068.html
http://www.ywnds.com/?p=7209&viewuser=42
https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483729&idx=1&sn=03711c826fcef68ea530a9cfa4decb25&chksm=fd72657bca05ec6d356040139c0f649800b5d5c454a4d675241fdd6a8809bb1657778a4aa032&scene=21#wechat_redirect
https://mp.weixin.qq.com/s?__biz=MzU3ODU4MTgyOQ==&mid=2247483735&idx=1&sn=a3de9f2632581b4643781b44cdbfe819&chksm=fd72657dca05ec6b1a82338e8416b58b58278daee749620393ec268a98833e6d2b0b25fe20ff&mpshare=1&scene=1&srcid=0913ujxgOYxYZzV6fuLfAMhn#rd
http://www.ywnds.com/?p=7209&viewuser=42
https://www.cnblogs.com/zengkefu/p/5690385.html
https://m.aliyun.com/yunqi/articles/51031
https://www.cnblogs.com/xpchild/p/3790139.html