Prometheus搜集mysql和nginx log指标

Prometheus搜集mysql和nginx log指标

• Prometheus+Grafana+Mysql-exporter+Nginx_logs-exporter
• 给予docker的方式部署运行
• 以下配置涉及的192.168.1.11为服务器IP，请根据实际而修改

一、部署 Prometheus+Grafana

Docker Compose For Prometheus+Grafana YML File

$ cat docker-compose-monitoring.yml

version: "2.3"
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: prometheus
    volumes:
      - ./prometheus/config:/etc/prometheus
      - ./prometheus/data:/prometheus
      - /etc/localtime:/etc/localtime
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--log.level=info'
      - '--web.listen-address=0.0.0.0:9090'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention=30d'
      - '--query.max-concurrency=50'
      - '--web.enable-lifecycle'
    ports:
      - "11090:9090"
    logging:
      driver: "json-file"
    restart: always
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana
    restart: always
    volumes:
      - /data/work/monitoring_data/grafana/logs:/var/log/grafana
      - /data/work/monitoring_data/grafana/data:/var/lib/grafana
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
      - /etc/localtime:/etc/localtime
    ports:
      - "11091:3000"
    user: "104"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Grafana 配置文件

• 由于grafana.ini配置文件内容太多就不一一贴出来了，本文仅体现修改。
• grafana.ini文件可以从grafana容器内copy出来(先运行grafana的容器，然后拷贝配置文件：docker cp grafana:/etc/grafana/grafana.ini .)

$ mkdir grafana
$ vim grafana/grafana.ini
[paths]
data = /var/lib/grafana
logs = /var/log/grafana

Prometheus 配置文件

$ mkdir -p prometheus/config
$ cat prometheus/config/prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    monitor: 'Monitoring'

scrape_configs:

启动&停止

docker-compose -f docker-compose-monitoring.yml up -d
docker-compose -f docker-compose-monitoring.yml down

• 浏览器打开Prometheus地址验证http://Prometheus_IP:11090/
  

• 浏览器打开Grafana地址验证http://Prometheus_IP:11091/，默认用户名密码为admin/admin
  

二、MySQL Exporter

$ cat docker-compose-mysql_exporter.yml
version: "2.3"
services:
  mysql_exporter:
    image: prom/mysqld-exporter:latest
    container_name: mysql_exporter
    hostname: mysql_exporter
    environment:
      DATA_SOURCE_NAME: "exporter:PWD123@(192.168.1.11:11009)/"
    ports:
      - "11092:9104"
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

登录需要监控的mysql服务，创建监控用户

$ mysql -uroot -p123456 -h 192.168.1.11 -P 11009

// 此处创建的监控用户名为exporter 密码为PWD123 根据实际情况进行修改.
> create user 'exporter'@'%';
> alter user 'exporter'@'%' identified with mysql_native_password by 'PWD123';
> GRANT REPLICATION CLIENT, PROCESS ON *.* TO 'exporter'@'%';
> GRANT SELECT ON performance_schema.* TO 'exporter'@'%';
> flush privileges;
> exit

根据MySQL的连接信息，修改compose相关的变量

      DATA_SOURCE_NAME: "exporter:PWD123@(192.168.1.11:11009)/"
#说明      DATA_SOURCE_NAME: "mysql用户:mysql密码@(mysql IP地址:mysql端口)/"

启动mysql exporter

docker-compose -f docker-compose-mysql_exporter.yml up -d
docker-compose -f docker-compose-mysql_exporter.yml down

访问

浏览器访问mysql exporter的接口即可获取相应的mysql metrics
http://服务器IP:11092/metrics

将MySQL Exporter添加到prometheus.yml文件

$ vim prometheus/config/prometheus.yml
global:
  scrape_interval: 15s
  external_labels:
    monitor: 'Monitoring'

scrape_configs:
# MySQL
  - job_name: 'MySQL'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.1.11:11092']
        labels:
          instance: '192.168.1.11'

重新加载配置

curl -X POST http://192.168.1.11:11090/-/reload

登录prometheus输入mysql关键词可以看的mysql相关指标

三、Nginx Logs Exporter

$ cat docker-compose-nginxlogs_exporter.yml
version: "2.3"
services:
  nginx_exporter:
    image: quay.io/martinhelmich/prometheus-nginxlog-exporter
    command: -config-file /etc/prometheus-nginxlog-exporter.hcl
    container_name: nginx_exporter
    hostname: nginx_exporter
    volumes:
      - /usr/local/var/log/nginx:/mnt/nginxlogs # 本机实际的nginx日志问价位置：/usr/local/var/log/nginx
      - ./config.hcl:/etc/prometheus-nginxlog-exporter.hcl
    ports:
      - "11093:4040"
    restart: always
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

配置文件

listen {
  port = 4040
  address = "0.0.0.0"
}

namespace "nginx" {
  format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time"

  source_files = ["/mnt/nginxlogs/access.log"]
  labels {
    app = "nginx-log"
  }

  relabel "request_url" {
    from = "request"
    split = 2

#    match "^/api/*" {
#      replacement = "api/"
#    }

    match "^/api/([a-z]+|[a-z]+_+[a-z]+)/([a-z]+|[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+|[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+_+[a-z]+)(/.*)" {
      replacement = "api/$1/$2"
    }

  }

}

Nginx logs exports config file

// config.hcl

定义的 format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time" # nginx日志保存格式，需要与nginx.conf配置文件相同
source_files: # nginx日志文件位置
match "^/api/*" {  : # 仅匹配以api关键词开头的接口信息，支持正则表达式过滤。

修改nginx的log输出格式：需要修改nginx的 nginx.conf 配置文件如下，重启后记得reload nginx

  log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         '$request_time $upstream_response_time';

  access_log /usr/local/var/log/nginx/access.log upstream_time;

启动&停止

docker-compose -f docker-compose-nginxlogs_exporter.yml up -d
docker-compose -f docker-compose-nginxlogs_exporter.yml down

prometheus.yml增加

# Nginx
  - job_name: 'Nginx'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.1.11:11093']
        labels:
          instance: '192.168.1.11'

重新加载prometheus

curl -X POST http://192.168.1.11:11090/-/reload

登录prometheus输入nginx关键词可以看的nginx相关指标

四、配置Grafana通过图表显示

• 登录Grafana，默认用户名密码为admin/admin

添加数据源（prometheus），名称以APP为例，URL为prometheus的地址

点击最后的Save&Test即可

导入Mysql 监控模版

• 推荐模版
• 更多模版可以在grafana官网查找

导入Nginx 监控模版

• 推荐模版
• 更多模版可以在grafana官网查找
• Import导入模版即可