prometheus实战：

一：安装部分：

https://prometheus.io/download/ ###下载源码解压即可

https://grafana.com/grafana/dashboards ###搜索数据源为prometheus的

这里下载了：prometheus、node_exporter、alertmanager、pushgateway

同时机器需要安装docker

yum install docker -y

systemctl start docker.service

安装gragana：

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.3.4-1.x86_64.rpm

yum localinstall grafana-5.3.4-1.x86_64.rpm 

systemctl start grafana-server

二 ：配置:

1、prometheus配置：

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  scrape_timeout: 10s     

alerting:

  alertmanagers:

  - static_configs:

    - targets:

       - 127.0.0.1:9093

rule_files:

  - "./rules/rule_*.yml"

  # - "second_rules.yml"

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'  ###这个必须配置，这个地址抓取的所有数据会自动加上`job=prometheus`的标签

    # metrics_path defaults to '/metrics' #抓取监控目标的路径，默认是/metrics 可以根据自己业务的需要进行修改

    # scheme defaults to 'http'.

    static_configs:   #这是通过静态文件的配置方法：这种方法直接指定要抓去目标的ip和端口

    - targets: ['localhost:9090']

  - job_name: gateway

    static_configs: 

    - targets: ['127.0.0.1:9091']

      labels:   ## 打上标签，instance会被指定为‘gataway’

        instance: gataway

  - job_name: node_export

    file_sd_configs: 

      #refresh_interval: 1m #刷新发现文件的时间间隔

      - files:

        - /data/prometheus-2.12.0.linux-amd64/node_discovery.json

  - job_name: mysql_discovery

    file_sd_configs: 

      #refresh_interval: 1m #刷新发现文件的时间间隔

      - files:

        - /data/prometheus-2.12.0.linux-amd64/mysql_discovery.json

  - job_name: redis_discovery

    file_sd_configs: 

      #refresh_interval: 1m #刷新发现文件的时间间隔

      - files:

        - /data/prometheus-2.12.0.linux-amd64/redis_discovery.json

各种discovery的模版为：

[{"targets": ["127.0.0.1:9100"],"labels": {"instance": "test","idc": "beijing"}},{"targets": ["127.0.0.1:9101"],"labels": {"instance": "test2","idc": "beijing"}}]

2、alertmanager配置：

global:

  resolve_timeout: 5m

#templates:

#  - 'demo.tmpl'

route:

  receiver: webhook

  group_wait: 30s

  group_interval: 5m

  repeat_interval: 10m

  group_by: ['alertname']

  routes:

  - receiver: webhook

    group_wait: 10s

    match:

      job_name: mysql|kubernetes

  - receiver: 'webhook-kafka'

    group_by: [instance, alertname]

    match_re:

      instance: ^kafka-(.*)

receivers:

- name: webhook

  webhook_configs:

  - url: http://localhost:8060/dingtalk/ops_dingding/send 

    send_resolved: true

- name: webhook-kafka

  webhook_configs:

  - url: http://localhost:8062/dingtalk/ops_59/send

    send_resolved: true

####备注：尽量使用动态发现的配置，以免配置文件过长

- job_name: mysql_discovery

file_sd_configs:

#refresh_interval: 1m #刷新发现文件的时间间隔

- files:

- /data/prometheus-2.12.0.linux-amd64/redis_discovery.json

三：启动服务：

1、promethues的server启动：

nohup sh start.sh 2>&1 > prometheus.log &

--start.sh内容为

./prometheus --storage.tsdb.path=./data --storage.tsdb.retention.time=168h --web.enable-lifecycle --storage.tsdb.no-lockfil

2、alertmanager的启动：

nohup sh start.sh 2>&1 > alertmanager.log &

--start.sh内容为

./alertmanager --config.file="alertmanager.yml" 

其余服务的启动

nohup ./node_exporter &

nohup ./pushgateway &

--------------------------

启动云数据库的采集服务：

docker run -d

  -p 9104:9104

  -e DATA_SOURCE_NAME="user:password@(url)/"

  prom/mysqld-exporter

docker run -d

  -p 9121:9121

  -e REDIS_ADDR="redis://url:port"

  -e REDIS_PASSWORD="password"

  oliver006/redis_exporter

如果有多组 只要修改外部暴漏端口和连接信息等等就可以了

-----报警规则配置：

rule_node.yml

groups:

    - name: 主机状态-监控告警

      rules:

      - alert: 主机状态

        expr: up == 0

        for: 1m

        labels:

          status: 非常严重

        annotations:

          summary: "{{$labels.instance}}:服务器宕机"

          description: "{{$labels.instance}}:服务器延时超过5分钟"

      

      - alert: CPU使用情况

        expr: 100-(avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by(instance)* 100) > 60

        for: 1m

        labels:

          status: 一般告警

        annotations:

          summary: "{{$labels.mountpoint}} CPU使用率过高！"

          description: "{{$labels.mountpoint }} CPU使用大于60%(目前使用:{{$value}}%)"

  

      - alert: 内存使用

        expr: 100 -(node_memory_MemTotal_bytes -node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes ) / node_memory_MemTotal_bytes * 100> 80

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} 内存使用率过高！"

          description: "{{$labels.mountpoint }} 内存使用大于80%(目前使用:{{$value}}%)"

      - alert: IO性能

        expr: 100-(avg(irate(node_disk_io_time_seconds_total[1m])) by(instance)* 100) < 60

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} 流入磁盘IO使用率过高！"

          description: "{{$labels.mountpoint }} 流入磁盘IO大于60%(目前使用:{{$value}})"

 

      - alert: 网络

        expr: ((sum(rate (node_network_receive_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} 流入网络带宽过高！"

          description: "{{$labels.mountpoint }}流入网络带宽持续2分钟高于100M. RX带宽使用率{{$value}}"

 

      - alert: 网络

        expr: ((sum(rate (node_network_transmit_bytes_total{device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[5m])) by (instance)) / 100) > 102400

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} 流出网络带宽过高！"

          description: "{{$labels.mountpoint }}流出网络带宽持续2分钟高于100M. RX带宽使用率{{$value}}"

      

      - alert: TCP会话

        expr: node_netstat_Tcp_CurrEstab > 1000

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} TCP_ESTABLISHED过高！"

          description: "{{$labels.mountpoint }} TCP_ESTABLISHED大于1000%(目前使用:{{$value}}%)"

 

      - alert: 磁盘容量

        expr: 100-(node_filesystem_free_bytes{fstype=~"ext4|xfs"}/node_filesystem_size_bytes {fstype=~"ext4|xfs"}*100) > 80

        for: 1m

        labels:

          status: 严重告警

        annotations:

          summary: "{{$labels.mountpoint}} 磁盘分区使用率过高！"

          description: "{{$labels.mountpoint }} 磁盘分区使用大于80%(目前使用:{{$value}}%)"

rule_mysql.yml

groups:

- name: MySQLStatsAlert

  rules:

  - alert: MySQL is down

    expr: up{job="mysql-discorvery"} == 0

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} MySQL is down"

      description: "MySQL database is down. This requires immediate action!"

  - alert: Read buffer size is bigger than max. allowed packet size

    expr: mysql_global_variables_read_buffer_size > mysql_global_variables_slave_max_allowed_packet 

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Read buffer size is bigger than max. allowed packet size"

      description: "Read buffer size (read_buffer_size) is bigger than max. allowed packet size (max_allowed_packet).This can break your replication."

  - alert: Sort buffer possibly missconfigured

    expr: mysql_global_variables_innodb_sort_buffer_size <256*1024 or mysql_global_variables_read_buffer_size > 4*1024*1024 

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Sort buffer possibly missconfigured"

      description: "Sort buffer size is either too big or too small. A good value for sort_buffer_size is between 256k and 4M."

  - alert: Thread stack size is too small

    expr: mysql_global_variables_thread_stack <196608

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Thread stack size is too small"

      description: "Thread stack size is too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size."

  - alert: Used more than 90% of max connections limited 

    expr: mysql_global_status_threads_connected > mysql_global_variables_max_connections * 0.8

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Used more than 80% of max connections limited"

      description: "Used more than 80% of max connections limited"

  - alert: InnoDB Force Recovery is enabled

    expr: mysql_global_variables_innodb_force_recovery != 0 

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} InnoDB Force Recovery is enabled"

      description: "InnoDB Force Recovery is enabled. This mode should be used for data recovery purposes only. It prohibits writing to the data."

  - alert: InnoDB Log File size is too small

    expr: mysql_global_variables_innodb_log_file_size < 16777216 

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} InnoDB Log File size is too small"

      description: "The InnoDB Log File size is possibly too small. Choosing a small InnoDB Log File size can have significant performance impacts."

  - alert: Table definition cache too small

    expr: mysql_global_status_open_table_definitions > mysql_global_variables_table_definition_cache

    for: 1m

    labels:

      severity: page

    annotations:

      summary: "Instance {{ $labels.instance }} Table definition cache too small"

      description: "Your Table Definition Cache is possibly too small. If it is much too small this can have significant performance impacts!"

  - alert: Thread stack size is possibly too small

    expr: mysql_global_variables_thread_stack < 262144

    for: 1m

    labels:

      severity: page

    annotations:

      summary: "Instance {{ $labels.instance }} Thread stack size is possibly too small"

      description: "Thread stack size is possibly too small. This can cause problems when you use Stored Language constructs for example. A typical is 256k for thread_stack_size."

  - alert: InnoDB Plugin is enabled

    expr: mysql_global_variables_ignore_builtin_innodb == 1

    for: 1m

    labels:

      severity: page

    annotations:

      summary: "Instance {{ $labels.instance }} InnoDB Plugin is enabled"

      description: "InnoDB Plugin is enabled"

  - alert: Binary Log is disabled

    expr: mysql_global_variables_log_bin != 1

    for: 1m

    labels:

      severity: warning

    annotations:

      summary: "Instance {{ $labels.instance }} Binary Log is disabled"

      description: "Binary Log is disabled. This prohibits you to do Point in Time Recovery (PiTR)."

  - alert: IO thread stopped

    expr: mysql_slave_status_slave_io_running != 1

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} IO thread stopped"

      description: "IO thread has stopped. This is usually because it cannot connect to the Master any more."

  - alert: SQL thread stopped 

    expr: mysql_slave_status_slave_sql_running == 0

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} SQL thread stopped"

      description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master."

  - alert: SQL thread stopped

    expr: mysql_slave_status_slave_sql_running != 1

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} Sync Binlog is enabled"

      description: "SQL thread has stopped. This is usually because it cannot apply a SQL statement received from the master."

  - alert: Slave lagging behind Master

    expr: rate(mysql_slave_status_seconds_behind_master[1m]) >30 

    for: 1m

    labels:

      severity: warning 

    annotations:

      summary: "Instance {{ $labels.instance }} Slave lagging behind Master"

      description: "Slave is lagging behind Master. Please check if Slave threads are running and if there are some performance issues!"

  - alert: Instance has slow logs

    expr: irate(mysql_global_status_slow_queries[5m]) > 10

    for: 1m

    labels:

      severity: worning

    annotations:

      summary: "Instance {{ $labels.instance }} has slow log"

      description: "slow log"

rule_redis.yml:

groups:

- name: MySQLStatsAlert

  rules:

  - alert: redis is down

    expr: up{job="redis-discorvery"} == 0

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} MySQL is down"

      description: "MySQL database is down. This requires immediate action!"

  - alert: redis memory alert

    expr: 100 * (redis_memory_used_bytes{instance !~ "pro-sas|pro-redis-dun"}  / redis_config_maxmemory{instance !~ "pro-sas|pro-redis-dun"} ) > 90

    for: 1m

    labels:

      severity: critical

    annotations:

      summary: "Instance {{ $labels.instance }} memory userd over 90"

      description: "redis memory short"

写在最后：

prometheus监控k8s:

http://note.youdao.com/noteshare?id=dbbc868d32835cab5eac5a455df243ed

prometheus钉钉报警:

https://github.com/timonwong/prometheus-webhook-dingtalk