mysql sql语句统计查询

环境是mysql8.0

之前调研过mysql audit 插件，均没有发现能适合mysql8.0环境的，除了企业版mysql审计插件，于是想了一个办法，把mysql慢日志时间改为0，这样所有sql都会被记录到慢日志里面,然后在用pt工具解析，过滤，然后写入到数据库。

my.cnf慢日志配置：

slow_query_log_file = /database/mysql/mysql-slow.log
slow_query_log = 1
log_slow_extra=1
long_query_time=0
log_slow_admin_statements=1
log_slow_slave_statements=1

pt-query-digest脚本：

#!/bin/bash
## ==================================================================================
##    让读书成为一种生活方式。就像吃喝拉撒每天必须要干的事，
## 终有一天你的举止、言谈、气质会不一样。 
##                                        —- async 
##
## Created Date: Sunday, 2020-05-16, 3:48:04 pm
## copyright (c):    SZWW Tech. LTD. 
## Engineer:   async 
## Module Name:   
## Revision:   v0.01 
## Description:
##   
## Revision History : 
## Revision     editor         date         Description         
## v0.01        async          2020-05-16   File Created
## ==================================================================================
DIR="$( cd "$( dirname "$0"  )" && pwd  )"
cd $DIR
dt=`date --date='5 min ago' "+%Y-%m-%d %H:%M:%S"`
#配置数据库的连接地址
monitor_db_host="172.18.1.xx"
monitor_db_port=33601
monitor_db_user="xxxxx"
monitor_db_password="xxxxx"
monitor_db_database="xxxx"
dba="xxxxx"
#实例慢日志位置
slowquery_file=/database/mysql/mysql-slow.log
pt_query_digest="/usr/bin/pt-query-digest"
#实例连接信息
hostname="`hostname`" # 和实例配置内容保持一致，用于做筛选

#获取上次分析时间，初始化时请删除last_analysis_time_$hostname文件，可分析全部日志数据
if [ -s ddl_last_analysis_time_$hostname ]; then
    last_analysis_time=`cat ddl_last_analysis_time_$hostname`
else
    last_analysis_time='1000-01-01 00:00:00'
fi

#收集日志
/usr/bin/cat $slowquery_file|grep -vE 'gtid_executed_v3|java' >/database/slow_log/mysql_slow.log && 
$pt_query_digest 
--user=$monitor_db_user --password=$monitor_db_password --port=$monitor_db_port 
--history h=$monitor_db_host,D=$monitor_db_database,t=review_history  
--no-report --limit=100% --charset=utf8mb4 
--filter="$event->{add_column}=length($event->{arg}) and $event->{fingerprint}=~ m/^(create|alter|drop|rename|truncate|comment)/i 
and $event->{client}=$event->{ip}" /database/slow_log/mysql_slow.log > /tmp/analysis_slow_query.log
if [ $? -ne 0 ]; then
echo "failed"|/usr/bin/mailx -s "$0 failed on $(hostname) at $(date) " ${dba}
else
echo `date +"%Y-%m-%d %H:%M:%S"`> ddl_last_analysis_time_$hostname
fi

我的日志里面有一些错误格式的日志，所以我把gtid_executed_v3这种过滤掉了，然后过滤出ddl ~ m/^(create|alter|drop|rename|truncate|comment)/i ，过滤dml为 ~ m/^(update|insert|delete)/i ，我主要过滤我关注的几类，然后写入表 review_history 

review_history建表语句:

CREATE TABLE review_history (
  id int(11) NOT NULL AUTO_INCREMENT,
  hostname_max varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs DEFAULT NULL,
  client_max varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs DEFAULT NULL,
  user_max varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs DEFAULT NULL,
  db_max varchar(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs DEFAULT NULL,
  checksum char(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs DEFAULT NULL,
  sample longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs,
  ts_min timestamp NULL DEFAULT NULL,
  ts_max timestamp NULL DEFAULT NULL,
  ts_cnt float DEFAULT NULL,
  Query_time_sum float DEFAULT NULL,
  Query_time_min float DEFAULT NULL,
  Query_time_max float DEFAULT NULL,
  Query_time_pct_95 float DEFAULT NULL,
  Query_time_stddev float DEFAULT NULL,
  Query_time_median float DEFAULT NULL,
  Lock_time_sum float DEFAULT NULL,
  Lock_time_min float DEFAULT NULL,
  Lock_time_max float DEFAULT NULL,
  Lock_time_pct_95 float DEFAULT NULL,
  Lock_time_stddev float DEFAULT NULL,
  Lock_time_median float DEFAULT NULL,
  Rows_sent_sum float DEFAULT NULL,
  Rows_sent_min float DEFAULT NULL,
  Rows_sent_max float DEFAULT NULL,
  Rows_sent_pct_95 float DEFAULT NULL,
  Rows_sent_stddev float DEFAULT NULL,
  Rows_sent_median float DEFAULT NULL,
  Rows_examined_sum float DEFAULT NULL,
  Rows_examined_min float DEFAULT NULL,
  Rows_examined_max float DEFAULT NULL,
  Rows_examined_pct_95 float DEFAULT NULL,
  Rows_examined_stddev float DEFAULT NULL,
  Rows_examined_median float DEFAULT NULL,
  Rows_affected_sum float DEFAULT NULL,
  Rows_affected_min float DEFAULT NULL,
  Rows_affected_max float DEFAULT NULL,
  Rows_affected_pct_95 float DEFAULT NULL,
  Rows_affected_stddev float DEFAULT NULL,
  Rows_affected_median float DEFAULT NULL,
  Rows_read_sum float DEFAULT NULL,
  Rows_read_min float DEFAULT NULL,
  Rows_read_max float DEFAULT NULL,
  Rows_read_pct_95 float DEFAULT NULL,
  Rows_read_stddev float DEFAULT NULL,
  Rows_read_median float DEFAULT NULL,
  Merge_passes_sum float DEFAULT NULL,
  Merge_passes_min float DEFAULT NULL,
  Merge_passes_max float DEFAULT NULL,
  Merge_passes_pct_95 float DEFAULT NULL,
  Merge_passes_stddev float DEFAULT NULL,
  Merge_passes_median float DEFAULT NULL,
  InnoDB_IO_r_ops_min float DEFAULT NULL,
  InnoDB_IO_r_ops_max float DEFAULT NULL,
  InnoDB_IO_r_ops_pct_95 float DEFAULT NULL,
  InnoDB_IO_r_ops_stddev float DEFAULT NULL,
  InnoDB_IO_r_ops_median float DEFAULT NULL,
  InnoDB_IO_r_bytes_min float DEFAULT NULL,
  InnoDB_IO_r_bytes_max float DEFAULT NULL,
  InnoDB_IO_r_bytes_pct_95 float DEFAULT NULL,
  InnoDB_IO_r_bytes_stddev float DEFAULT NULL,
  InnoDB_IO_r_bytes_median float DEFAULT NULL,
  InnoDB_IO_r_wait_min float DEFAULT NULL,
  InnoDB_IO_r_wait_max float DEFAULT NULL,
  InnoDB_IO_r_wait_pct_95 float DEFAULT NULL,
  InnoDB_IO_r_wait_stddev float DEFAULT NULL,
  InnoDB_IO_r_wait_median float DEFAULT NULL,
  InnoDB_rec_lock_wait_min float DEFAULT NULL,
  InnoDB_rec_lock_wait_max float DEFAULT NULL,
  InnoDB_rec_lock_wait_pct_95 float DEFAULT NULL,
  InnoDB_rec_lock_wait_stddev float DEFAULT NULL,
  InnoDB_rec_lock_wait_median float DEFAULT NULL,
  InnoDB_queue_wait_min float DEFAULT NULL,
  InnoDB_queue_wait_max float DEFAULT NULL,
  InnoDB_queue_wait_pct_95 float DEFAULT NULL,
  InnoDB_queue_wait_stddev float DEFAULT NULL,
  InnoDB_queue_wait_median float DEFAULT NULL,
  InnoDB_pages_distinct_min float DEFAULT NULL,
  InnoDB_pages_distinct_max float DEFAULT NULL,
  InnoDB_pages_distinct_pct_95 float DEFAULT NULL,
  InnoDB_pages_distinct_stddev float DEFAULT NULL,
  InnoDB_pages_distinct_median float DEFAULT NULL,
  QC_Hit_cnt float DEFAULT NULL,
  QC_Hit_sum float DEFAULT NULL,
  Full_scan_cnt float DEFAULT NULL,
  Full_scan_sum float DEFAULT NULL,
  Full_join_cnt float DEFAULT NULL,
  Full_join_sum float DEFAULT NULL,
  Tmp_table_cnt float DEFAULT NULL,
  Tmp_table_sum float DEFAULT NULL,
  Tmp_table_on_disk_cnt float DEFAULT NULL,
  Tmp_table_on_disk_sum float DEFAULT NULL,
  Filesort_cnt float DEFAULT NULL,
  Filesort_sum float DEFAULT NULL,
  Filesort_on_disk_cnt float DEFAULT NULL,
  Filesort_on_disk_sum float DEFAULT NULL,
  Bytes_sum float DEFAULT NULL,
  Bytes_min float DEFAULT NULL,
  Bytes_max float DEFAULT NULL,
  Bytes_pct_95 float DEFAULT NULL,
  Bytes_stddev float DEFAULT NULL,
  Bytes_median float DEFAULT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY checksum (checksum,ts_min,ts_max),
  KEY idx_hostname_max_ts_min (hostname_max,ts_min)
) ENGINE=InnoDB AUTO_INCREMENT=797 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_as_cs ROW_FORMAT=COMPRESSED

参考官方链接https://www.percona.com/doc/percona-toolkit/3.0/pt-query-digest.html

然后加入crontab，每个几分钟执行一次，后来发现这样会漏记录一些sql，频繁去解析slow_log对操作系统io也有一定消耗，思考以后，改成了每隔6小时全量执行一次，这样就会有一点延时，不过这个主要用于审计sql,延时也能接受，如果要实时，在想想办法。不增量以后，没再发现漏记sql。

主要就是pt的filter参数：

 --filter="$event->{add_column}=length($event->{arg}) and $event->{hostname}="$hostname" and $event->{client}=$event->{ip}"  

一开始是这样写的，记录的太多，查表筛选的时候慢，于是只过滤了自己需要的类型ddl与dml：

--filter="$event->{add_column}=length($event->{arg}) and $event->{fingerprint}=~ m/^(create|alter|drop|rename|truncate|comment|delete|insert|update)/i 
and $event->{client}=$event->{ip}"

然后修改下脚本就实现了mysql的sql审计，然后再配合init-connect，可以实现比较精准快速的排查问题。