公司内部有一套mysql环境，使用的是percona server分支（和其他几十套环境的版本、参数完全相同），就这套环境每隔两三天就会hang一次，关键hang的时候服务器cpu也就是百分之三四十，swap也没用多少，磁盘空间这些都足够，mysql.err里也没有任何的异常信息。出现问题后，root登录也没有响应一直卡在那里。。。太奇怪了。。。
因为已经发生三四次了，对项目组影响较大，于是觉得找到根本原因。随后在/var/log/message和dmesg中发现了EXT4-fs (dm-0): Unaligned AIO/DIO on inode 1323274 by mysqld; performance will be poor.
因为ext4默认的页面大小为4096，mysql log的默认大小为512。因为我们设置了参数innodb_flush_method=ALL_O_DIRECT，所以导致了该问题（参见https://bugs.launchpad.net/percona-server/+bug/1033051，http://www.oschina.net/translate/more-about-the-innodb_log_block_size-variable，https://www.percona.com/blog/2011/01/03/effect-from-innodb-log-block-size-4096-bytes/）。
所以决定设置innodb_log_block_size=4096，重启mysql，启动的时候发现出错了。。
2016-08-31 10:18:01 9711 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
InnoDB: The log block size is set to 4096.
2016-08-31 10:18:01 9711 [Note] InnoDB: Using atomics to ref count buffer pool pages
2016-08-31 10:18:01 9711 [Note] InnoDB: The InnoDB memory heap is disabled
2016-08-31 10:18:01 9711 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-08-31 10:18:01 9711 [Note] InnoDB: Memory barrier is not used
2016-08-31 10:18:01 9711 [Note] InnoDB: Compressed tables use zlib 1.2.3
2016-08-31 10:18:01 9711 [Note] InnoDB: Using Linux native AIO
2016-08-31 10:18:01 9711 [Note] InnoDB: Using CPU crc32 instructions
2016-08-31 10:18:01 9711 [Note] InnoDB: Initializing buffer pool, size = 512.0M
2016-08-31 10:18:01 9711 [Note] InnoDB: Completed initialization of buffer pool
2016-08-31 10:18:01 9711 [Note] InnoDB: Highest supported file format is Barracuda.
InnoDB: No valid checkpoint found.
InnoDB: If you are attempting downgrade from MySQL 5.7.9 or later,
InnoDB: please refer to http://dev.mysql.com/doc/refman/5.6/en/upgrading-downgrading.html
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/error-creating-innodb.html
2016-08-31 10:18:01 9711 [ERROR] Plugin 'InnoDB' init function returned error.
2016-08-31 10:18:01 9711 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-08-31 10:18:01 9711 [ERROR] Unknown/unsupported storage engine: InnoDB
2016-08-31 10:18:01 9711 [ERROR] Aborting
先把旧的ib_log*人工删除了，再重启，成功。
待观察几天看看，是否只有这原因所致。