prometheus系列监控：jvm,mongodb,mysql,redis,consul

jvm：

maven添加dependence

<!-- https://mvnrepository.com/artifact/io.micrometer/micrometer-registry-prometheus -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.3.5</version>
</dependency>

编辑springboot项目的yml文件

yml配置参考https://blog.csdn.net/u014401141/article/details/84784422

server:
  port: 8085


spring:
  #for monitor
  application:
    name: mall_prometheus






management:
  #TODO endpoint 解释查询
  endpoints:
    web:
      exposure:
        #include: "*"
        include: info, health, beans, env, metrics, mappings, scheduledtasks, sessions, threaddump, docs, logfile, jolokia, prometheus
      base-path: /actuator #默认该路径，不更改可不用配置
      #cors跨域支持
      cors:
        allowed-origins: http://example.com
        allowed-methods: GET,PUT,POST,DELETE
    prometheus:
      id: springmetrics
  endpoint:
    beans:
      cache:
        time-to-live: 10s #端点缓存响应的时间量
    health:
      show-details: always #详细信息显示给所有用户
  server:
    port: 8001 #默认8888
    #address: 127.0.0.1 #配置此项表示不允许远程连接
  #monitor
  metrics:
    export:
      datadog:
        application-key: ${spring.application.name}
    web:
      server:
        auto-time-requests: true

　　

配置prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
  - job_name: linux
    static_configs:
      - targets: ['47.112.188.174:9100']
        labels:
          instance: node
  - job_name: 'spring'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['47.112.188.174:8001']

　　

docker 启动服务时，开放8001端口

# 9100是exporter的端口
docker run -p 8001:8001 -p 8085:8085 --name mall-portal 
 --link mall-mysql:db 
 --link mall-redis:redis 
 --link mongo:mongo 
 --link rabbitmq:rabbit 
 -v /etc/localtime:/etc/localtime 
 -v/usr/local/dockerdata/mall-project/mall-port/logs:/var/logs 
 -d mall/mall-portal:1.0-SNAPSHOT

　　

启动prometheus时加载配置

docker run --name prometheus -d -p 9090:9090 --privileged=true -v /usr/local/dockerdata/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

　　

grafana添加对应dashboard:4701

https://grafana.com/grafana/dashboards/4701

 

 mysql

启动mysql的时候

数据库内执行：(添加exporter的权限)

use mall;
GRANT REPLICATION CLIENT, PROCESS ON *.* to 'exporter'@'%' identified by 'exporter';
GRANT SELECT ON performance_schema.* TO 'exporter'@'%';
flush privileges;

　　

docker 启动 mysqld_exporter（使用9104端口向prometheus传递数据）

docker run -d --restart=always --name mysqld-exporter -p 9104:9104 -e DATA_SOURCE_NAME='exporter:exporter@(47.112.188.174:3306)/' prom/mysqld-exporter

　　

　

curl 验证一下

curl localhost:9104/metrics
process_virtual_memory_bytes 1.16281344e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

　　 

然后修改一下prometheus的yaml。启动prometheus。。。。。具体编写和运行不再赘述。。。

--------------------------------------------这里是服务发现---------------------------------------------------------------------

上面这种方式太过繁琐，几乎每次加一个exporter，都要修改prometheus.yml，并重启。

优化方案：可以使用consul来自动搜寻服务，每当要加一个exporter ，就通过consul的自动发现来加到prometheus，不用重新配置yml。

consul官网：https://www.consul.io/downloads.html

步骤1 ：docker 启动consul，启用8500端口

consul 服务发现
docker run --restart=always --name consul -d -p 8500:8500 consul

　

服务启动后访问url看一下： curl localhost:8500/ui　

步骤2：修改prometheus.yml　　，添加consul的job，这里配置成根据tag里面有“mall"字样来添加。

　

  - job_name: consul
    consul_sd_configs:
      - server: '47.112.188.174:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*mall.*
        action: keep

启动prometheus时，传入该配置：

docker run --name prometheus -d -p 9090:9090 --privileged=true -v /usr/local/dockerdata/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml
　　

　　

curl localhost:9090/target 看一下是否启动成功，这里根据配置会有两个job：prometheus，node

步骤3：注册服务(即传入exporter的开放端口，这里传入mysqld_exporter的9104)

curl -X PUT -d '{"id": "mysql01","name": "mysql01","address": "47.112.188.174","port": 9104,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174:9104/","interval": "5s"}]}' http://localhost:8500/v1/agent/service/register

-----------------------------------------------添加redis,mongodb 只需要启动对应的docker exporter，然后步骤3传入服务就可以了--------------------------------------------------------------

添加一下mongodb redis的exporter

mongodb / redis
./redis_exporter -redis.addr 47.93.32.161:6379 & -web.listenaddress 0.0.0.0:9122
./mongodb_exporter-linux-amd64 -mongodb.uri mongodb://47.93.32.161:27017/admin http://localhost:8500/v1/agent/service/register

mongodb_exporter我用的docker

docker run -d --name=mongodb_exporter -p 9001:9001  mongodb_exporter

　

添加mongodb的服务

curl -X PUT -d '{"id": "mongo01","name": "mongo01","address": "47.112.188.174","port": 9001,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174/","interval": "5s"}]}' http://localhost:8500/v1/agent/service/register

　

对应的dashboard 参考github给的吧。。。懒得搞了

https://github.com/dcu/mongodb_exporter　

添加spring的服务

curl -X PUT -d '{"id": "spring01","name": "spring01","address": "47.112.188.174","port": 8001,"tags": ["mall"],"checks": [{"http": "http://47.112.188.174:8001/actuator/prometheus","interval": "5s"}]}'  http://localhost:8500/v1/agent/service/register　　

　　

　

这里因为spring的exporter 路径不是/metrics，在curl后 服务从 localhost:8001/metrics读不到有效数据，会有报错。这里就需要将默认的/metrics转换成/actuator/prometheus，更改一下yml文件：

  - job_name: consul
    consul_sd_configs:
      - server: '47.112.188.174:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*mall.*
        action: keep
      - source_labels: [__meta_consul_service]
        regex: spring01
        target_label: __metrics_path__
        replacement: /actuator/prometheus
        action: replace

　　

------------------------------------其他---------------------------------

几个exporter的git地址，这里要特别注意，搜索exporter的时候要注意是否有自带的dashboard，否则直接从grafana添加，可能dashboard并不匹配

https://github.com/prometheus/mysqld_exporter/releases
https://github.com/oliver006/redis_exporter/releases
https://github.com/percona/mongodb_exporter/releases

最后：

1.curl后 consul的展示

2. consul自动接入的服务，在prometheus上是否存储成功

mysql的grafana视图选6239