基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus监控

基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus及Exporter安装

目录

• 基于Prometheus+Grafana搭建可视化监控服务 (一) Prometheus及Exporter安装
  • 一、概述
  • 二、安装Prometheus
    • 2.1.安装Prometheus
    • 2.2.将Promethues配置成系统服务
    • 2.3.通过systemctl启动prometheus
    • 2.4.prometheus界面
  • 三、监控远程Linux主机
    • 3.1.安装node_exporter
    • 3.2.将node_exporter配置成系统服务
    • 3.3.通过systemctl启动node_exporter
    • 3.4.验证node_exporter
    • 3.5.防火墙配置
    • 3.6.Prometheus侧配置
      • 3.6.1.prometheus.yml配置调整
      • 3.6.2.校验配置正确性
      • 3.6.3.重新加载配置
      • 3.6.4.验证结果
  • 四、监控MySQL
    • 4.1.创建监控用账号
    • 4.2.安装mysqld_exporter
    • 4.4.将mysqld_exporter配置成系统服务
    • 4.5.通过systemctl启动mysqld_exporter
    • 4.6.验证mysqld_exporter
    • 4.7.防火墙配置
    • 4.8.Prometheus侧配置
      • 4.8.1.prometheus.yml配置调整
      • 4.8.2.校验配置正确性
      • 4.8.3.重新加载配置
      • 4.8.4.验证结果
  • 五、监控Redis
    • 5.1.安装redis_exporter
    • 5.2.将redis_exporter配置成系统服务
    • 5.3.通过systemctl启动redis_exporter
    • 5.4.验证redis_exporter
    • 5.5.防火墙配置
    • 5.6.Prometheus侧配置
      • 5.6.1.prometheus.yml配置调整
      • 5.6.2.校验配置正确性
      • 5.6.3.重新加载配置
      • 5.6.4.验证结果
      • 5.6.5 Redis集群监控配置示例
  • 六、监控RabbitMQ
    • 6.1.安装rabbitmq_prometheus
    • 6.2.验证rabbitmq_prometheus
    • 6.3.防火墙配置
    • 6.4.Prometheus侧配置
      • 6.4.1.prometheus.yml配置调整
      • 6.4.2.校验配置正确性
      • 6.4.3.重新加载配置
      • 6.4.4.验证结果
  • 七、监控Elasticsearch
    • 7.1.安装elasticsearch_exporter
    • 7.2.将elasticsearch_exporter配置成系统服务
    • 7.3.通过systemctl启动elasticsearch_exporter
    • 7.4.验证elasticsearch_exporter
    • 7.5.防火墙配置
    • 7.6.Prometheus侧配置
      • 7.6.1.prometheus.yml配置调整
      • 7.6.2.校验配置正确性
      • 7.6.3.重新加载配置
      • 7.6.4.验证结果
  • 八、监控Kafka
    • 8.1.安装kafka_exporter
    • 8.2.将kafka_exporter配置成系统服务
    • 8.3.通过systemctl启动kafka_exporter
    • 8.4.验证kafka_exporter
    • 8.5.防火墙配置
    • 8.6.Prometheus侧配置
      • 8.6.1.prometheus.yml配置调整
      • 8.6.2.校验配置正确性
      • 8.6.3.重新加载配置
      • 8.6.4.验证结果
  • 九、监控Doris
    • 9.1.验证doris_prometheus
    • 9.2.Prometheus侧配置
      • 9.2.1.prometheus.yml配置调整
      • 9.2.2.校验配置正确性
      • 9.2.3.重新加载配置
      • 9.2.4.验证结果
  • 十、监控端口(HTTP/TCP等)
    • 10.1.安装blackbox_exporter
    • 10.2.将blackbox_exporter配置成系统服务
    • 10.3.通过systemctl启动blackbox_exporter
    • 10.4.Prometheus侧配置
      • 10.4.1.prometheus.yml配置调整
      • 10.4.2.校验配置正确性
      • 10.4.3.重新加载配置
      • 10.4.4.验证结果
  • 十一、监控MongoDB
    • 11.1.安装mongodb_exporter
    • 11.2.将mongodb_exporter配置成系统服务
    • 11.3.通过systemctl启动mongodb_exporter
    • 11.4.验证mongodb_exporter
    • 11.5.防火墙配置
    • 11.6.Prometheus侧配置
      • 11.6.1.prometheus.yml配置调整
  • 附:其他参考资料

一、概述

Prometheus官网文档: https://prometheus.io/docs/introduction/overview/

各组件默认端口:
prometheus ：9090
node 默认端口：9091(原默认端口为9100,与elasticsearch端口冲突，故这里改为9091)
mysql_exporter默认端口：9104
redis_exporter 默认端口：9121
rabbitmq_prometheus 默认端口： 15692
elasticsearch_exporter 默认端口： 9114
kafka_exporter 默认端口： 9308
blackbox_exporter默认端口: 9115
mongodb_exporter 端口: 9216
alertmanager默认端口：9093

注意：
1.本文使用的各程序版本是当下最新版本，实际需要到官网去看是否有更新的版本，推荐使用最新版本。
2.本文Prometheus等程序安装及监控目标都为Linux环境(CentOS7)
3.Promethues支持pull和push两种方式，本文描述的是pull方式
4.exporter一般都部署在监控目标服务器上，blackbox_exporter则和prometheus部署在一起

二、安装Prometheus

2.1.安装Prometheus

[root@server ~]# cd /usr/local/src
[root@server src]# wget https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz
[root@server src]# mkdir -p /usr/local/prometheus/
[root@server src]# tar xvf prometheus-2.28.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@server src]# mv /usr/local/prometheus/prometheus-2.28.1.linux-amd64/ /usr/local/prometheus/prometheus

2.2.将Promethues配置成系统服务

[root@server src]# vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target
  
[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/prometheus
ExecStart=/usr/local/prometheus/prometheus/prometheus 
  --config.file=/usr/local/prometheus/prometheus/prometheus.yml 
  --storage.tsdb.path=/usr/local/prometheus/prometheus/data 
  --storage.tsdb.retention.time=30d
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID
[Install]
WantedBy=multi-user.target

config.file指定配置文件地址
storage.tsdb.path指定数据文件目录
storage.tsdb.retention.time指定数据文件保留时间

Promethues提供了很多的命令行配置选项，可通过./prometheus -h进行查看

2.3.通过systemctl启动prometheus

[root@server src]# systemctl daemon-reload
[root@server src]# systemctl start prometheus
[root@server src]# systemctl status prometheus
[root@server src]# systemctl enable prometheus
[root@server src]# systemctl reload prometheus

2.4.prometheus界面

通过浏览器访问http://服务器IP:9090就可以访问到prometheus的主界面
http://172.16.111.48:9090/targets

三、监控远程Linux主机

https://github.com/prometheus/node_exporter/

3.1.安装node_exporter

[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/prometheus/node_exporter/releases/download/v1.2.0/node_exporter-1.2.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf node_exporter-1.2.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/node_exporter-1.2.0.linux-amd64/ /usr/local/prometheus/node_exporter

3.2.将node_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/node_exporter.service

[Unit]
Description=node_exporter Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target
  
[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/node_exporter
ExecStart=/usr/local/prometheus/node_exporter/node_exporter 
  --web.listen-address=0.0.0.0:9101
ExecStop=/bin/kill -SIGINT $MAINPID

[Install]
WantedBy=multi-user.target

3.3.通过systemctl启动node_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start node_exporter
[root@dbserver src]# systemctl status node_exporter
[root@dbserver src]# systemctl enable node_exporter

3.4.验证node_exporter

[root@dbserver src]# curl "http://127.0.0.1:9101/metrics"

3.5.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9101/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9101" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

3.6.Prometheus侧配置

3.6.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'node_exporter'
    static_configs:
    - targets: ['192.168.1.100:9101','192.168.1.101:9101']
      labels:
        service: 服务A
    - targets: ['192.168.1.102:9101','192.168.1.103:9101']
      labels:
        service: 服务B

3.6.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

3.6.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

3.6.4.验证结果

访问http://ip:9090/targets查看

四、监控MySQL

https://github.com/prometheus/mysqld_exporter/

4.1.创建监控用账号

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'Exporter#123';
GRANT PROCESS,RELOAD,REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
FLUSH PRIVILEGES;

4.2.安装mysqld_exporter

[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.13.0/mysqld_exporter-0.13.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf mysqld_exporter-0.13.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/mysqld_exporter-0.13.0.linux-amd64/ /usr/local/prometheus/mysqld_exporter
```

### 4.3.增加配置文件
```shell
[root@dbserver src]# vi /usr/local/prometheus/mysqld_exporter/.my.cnf
[client]
user=exporter
password=Exporter#123
socket=/var/lib/mysql/mysql.sock

4.4.将mysqld_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/mysqld_exporter.service

[Unit]
Description=mysqld_exporter Server
Documentation=https://github.com/prometheus/mysqld_exporter/
After=network.target

[Service]
Environment=DATA_SOURCE_NAME=exporter:Exporter#123@(localhost:3306)/
ExecStart=/usr/local/prometheus/mysqld_exporter/mysqld_exporter 
  --web.listen-address=0.0.0.0:9104 
  --config.my-cnf=/usr/local/prometheus/mysqld_exporter/.my.cnf 
  --collect.binlog_size 
  --collect.slave_status 
  --collect.slave_hosts 
  --collect.engine_innodb_status 
  --collect.info_schema.processlist 
  --collect.info_schema.innodb_metrics 
  --collect.info_schema.innodb_tablespaces 
  --collect.info_schema.clientstats 
  --collect.perf_schema.tableiowaits 
  --collect.perf_schema.indexiowaits 
  --collect.perf_schema.tablelocks 
  --collect.perf_schema.file_events 
  --collect.perf_schema.eventswaits
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target

4.5.通过systemctl启动mysqld_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start mysqld_exporter
[root@dbserver src]# systemctl status mysqld_exporter
[root@dbserver src]# systemctl enable mysqld_exporter

4.6.验证mysqld_exporter

[root@dbserver src]# curl "http://127.0.0.1:9104/metrics"

4.7.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9104/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9104" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

4.8.Prometheus侧配置

4.8.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'mysql_exporter'
    static_configs:
    - targets: ['192.168.1.100:9104']
      labels:
        service: 服务A-MySQL
    - targets: ['192.168.1.101:9104']
      labels:
        service: 服务B-MySQL

4.8.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

4.8.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

4.8.4.验证结果

访问http://ip:9090/targets查看

五、监控Redis

https://github.com/oliver006/redis_exporter/

5.1.安装redis_exporter

[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/oliver006/redis_exporter/releases/download/v1.24.0/redis_exporter-v1.24.0.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf redis_exporter-v1.24.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/redis_exporter-v1.24.0.linux-amd64/ /usr/local/prometheus/redis_exporter

5.2.将redis_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/redis_exporter.service

[Unit]
Description=redis_exporter Server
Documentation=https://github.com/oliver006/redis_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/redis_exporter/redis_exporter 
  --web.listen-address=0.0.0.0:9121 
  --redis.addr=127.0.0.1:6379 
  --redis.password=Test#123
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target

5.3.通过systemctl启动redis_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start redis_exporter
[root@dbserver src]# systemctl status redis_exporter
[root@dbserver src]# systemctl enable redis_exporter

5.4.验证redis_exporter

[root@dbserver src]# curl "http://127.0.0.1:9121/metrics"

5.5.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9121/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9121" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

5.6.Prometheus侧配置

5.6.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'redis_exporter'
    static_configs:
    - targets: ['192.168.1.100:9121']
      labels:
        service: 服务A-Redis
    - targets: ['192.168.1.101:9121']
      labels:
        service: 服务B-Redis

5.6.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

5.6.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

5.6.4.验证结果

访问http://ip:9090/targets查看

5.6.5 Redis集群监控配置示例

(本人未验证)

  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://192.168.1.101:7000
        - redis://192.168.1.102:7001
        - redis://192.168.1.103:7002
        - redis://192.168.1.104:7003
        - redis://192.168.1.105:7004
        - redis://192.168.1.106:7005
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 192.168.1.100:9121
  ## config for scraping the exporter itself
  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - 192.168.1.100:9121

六、监控RabbitMQ

6.1.安装rabbitmq_prometheus

RabbitMQ 3.8版本以后已内置支持Prometheus插件
https://www.rabbitmq.com/prometheus.html

启用方式：

[root@dbserver src]# rabbitmq-plugins enable rabbitmq_prometheus

默认端口号为15692
启动后可访问 http://ip:15692/metrics

如要修改默认配置，对应配置文件为 /etc/rabbitmq/rabbitmq.conf
prometheus.path = /metrics
prometheus.tcp.port = 15692

也可以使用单独的rabbitmq_exporter来监控，可适应rabbitmq3.8以前版本
（注意 3.6.x与更早版本监控有差别，需要注意)
https://github.com/kbudde/rabbitmq_exporter/

6.2.验证rabbitmq_prometheus

[root@dbserver src]# curl "http://127.0.0.1:15692/metrics"

6.3.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=15692/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="15692" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

6.4.Prometheus侧配置

6.4.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'rabbitmq_exporter'
    static_configs:
    - targets: ['192.168.1.100:15692']
      labels:
        service: 服务A-RabbitMQ
    - targets: ['192.168.1.101:15692']
      labels:
        service: 服务B-RabbitMQ

6.4.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

6.4.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

6.4.4.验证结果

访问http://ip:9090/targets查看

七、监控Elasticsearch

Elasticsearch Exporter
https://grafana.com/oss/prometheus/exporters/elasticsearch-exporter/
https://github.com/prometheus-community/elasticsearch_exporter

7.1.安装elasticsearch_exporter

[root@dbserver ~]# cd /usr/local/src
[root@dbserver src]# wget https://github.com/justwatchcom/elasticsearch_exporter/releases/download/v1.2.1/elasticsearch_exporter-1.2.1.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf elasticsearch_exporter-1.2.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/elasticsearch_exporter-1.2.1.linux-amd64/ /usr/local/prometheus/elasticsearch_exporter

7.2.将elasticsearch_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/elasticsearch_exporter.service

[Unit]
Description=elasticsearch_exporter Server
Documentation=https://github.com/prometheus-community/elasticsearch_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/elasticsearch_exporter/elasticsearch_exporter 
  --web.listen-address=0.0.0.0:9114 
  --es.uri=http://127.0.0.1:9200
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target

如果有启用账号密码，则需要携带上 http://admin:pass@localhost:9200

7.3.通过systemctl启动elasticsearch_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start elasticsearch_exporter
[root@dbserver src]# systemctl status elasticsearch_exporter
[root@dbserver src]# systemctl enable elasticsearch_exporter

7.4.验证elasticsearch_exporter

[root@dbserver src]# curl "http://127.0.0.1:9114/metrics"

7.5.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9114/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9114" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

7.6.Prometheus侧配置

7.6.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'elasticsearch_exporter'
    static_configs:
    - targets: ['192.168.1.100:9114']
      labels:
        service: 服务A-Elasticsearch
    - targets: ['192.168.1.101:9114']
      labels:
        service: 服务B-Elasticsearch

7.6.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

7.6.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

7.6.4.验证结果

访问http://ip:9090/targets查看

八、监控Kafka

https://github.com/danielqsj/kafka_exporter/

8.1.安装kafka_exporter

[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.3.1/kafka_exporter-1.3.1.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf kafka_exporter-1.3.1.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/kafka_exporter-1.3.1.linux-amd64/ /usr/local/prometheus/kafka_exporter

8.2.将kafka_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/kafka_exporter.service

[Unit]
Description=kafka_exporter Server
Documentation=https://github.com/danielqsj/kafka_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/kafka_exporter/kafka_exporter 
  --web.listen-address=0.0.0.0:9308 
  --kafka.server=127.0.0.1:9092
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target

8.3.通过systemctl启动kafka_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start kafka_exporter
[root@dbserver src]# systemctl status kafka_exporter
[root@dbserver src]# systemctl enable kafka_exporter

8.4.验证kafka_exporter

[root@dbserver src]# curl "http://127.0.0.1:9308/metrics"

8.5.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9308/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9308" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

8.6.Prometheus侧配置

8.6.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'elasticsearch_exporter'
    static_configs:
    - targets: ['192.168.1.100:9308']
      labels:
        service: 服务A-Kafka
    - targets: ['192.168.1.101:9308']
      labels:
        service: 服务B-Kafka

8.6.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

8.6.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

8.6.4.验证结果

访问http://ip:9090/targets查看

九、监控Doris

Doris已内置支持对FE,BE的Promotheus监控支持
无需额外配置，防护墙允许Prometheus访问 8030,8040端口即可

9.1.验证doris_prometheus

#FE
[root@dbserver src]# curl "http://127.0.0.1:8030/metrics"

#BE
[root@dbserver src]# curl "http://127.0.0.1:8040/metrics"

9.2.Prometheus侧配置

9.2.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'doris_cluster'
    static_configs:
    - targets: ['192.168.1.100:8030','192.168.1.101:8030']
      labels:
	    service: 服务A-Doris-FE
        group: fe

    - targets: ['192.168.1.102:8040','192.168.1.103:8040','192.168.1.104:8040']
      labels:
	    service: 服务A-Doris-BE
        group: be

9.2.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

9.2.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

9.2.4.验证结果

访问http://ip:9090/targets查看

十、监控端口(HTTP/TCP等)

10.1.安装blackbox_exporter

这里使用 blackbox_exporter来监控HTTP/TCP

[root@appserver ~]# cd /usr/local/src/
[root@appserver src]# wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
[root@appserver src]# tar xvf blackbox_exporter-0.19.0.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@appserver src]# mv /usr/local/prometheus/blackbox_exporter-0.19.0.linux-amd64/ /usr/local/prometheus/blackbox_exporter

10.2.将blackbox_exporter配置成系统服务

编辑启动文件

[root@appserver src]# vi /etc/systemd/system/blackbox_exporter.service

[Unit]
Description=blackbox_exporter Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target
  
[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheus/blackbox_exporter
ExecStart=/usr/local/prometheus/blackbox_exporter/blackbox_exporter 
  --web.listen-address ":9115" 
  --config.file=/usr/local/prometheus/blackbox_exporter/blackbox.yml
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -SIGINT $MAINPID

[Install]
WantedBy=multi-user.target

10.3.通过systemctl启动blackbox_exporter

[root@appserver src]# systemctl daemon-reload
[root@appserver src]# systemctl start blackbox_exporter
[root@appserver src]# systemctl status blackbox_exporter
[root@appserver src]# systemctl enable blackbox_exporter

10.4.Prometheus侧配置

10.4.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

# 网站监控
  - job_name: 'http_status'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['http://a1.domain.com:81']
        labels:
          env: PROD
          domain: "a1.domain.com"
          port: "81"
          service: 服务A
          instance: "http://a1.domain.com:81"
          group: web
      - targets: ['http://a1.domain.com:82/abc']
        labels:
          env: PROD
          domain: "a1.domain.com"
          port: "82"
          service: XX API接口
          instance: "http://a1.domain.com:82/abc"
          group: api

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 
		
# ping 检测
  - job_name: 'ping_status'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['192.168.1.100']
        labels:
          env: PROD
          service: 服务B IP
          instance: "192.168.1.100"
          group: ping

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 
		
# 端口监控
  - job_name: 'port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['192.168.1.100:84']
        labels:
          env: PROD
          port: "84"
          service: 服务B XX端口
          instance: "192.168.1.100:84"
          group: port

    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.1.100:9115 

blackbox_exporter涉及的module配置文件 /usr/local/prometheus/blackbox_exporter/blackbox.yml

网上示例参考：https://github.com/tools-env/blackbox_exporter/blob/master/example.yml

10.4.2.校验配置正确性

[root@dbserver src]# /usr/local/prometheus/prometheus/promtool check config /usr/local/prometheus/prometheus/prometheus.yml 

10.4.3.重新加载配置

[root@dbserver src]# systemctl reload prometheus

10.4.4.验证结果

访问http://ip:9090/targets查看

十一、监控MongoDB

https://github.com/percona/mongodb_exporter

11.1.安装mongodb_exporter

[root@dbserver ~]# cd /usr/local/src/
[root@dbserver src]# wget https://github.com/percona/mongodb_exporter/releases/download/v0.20.7/mongodb_exporter-0.20.7.linux-amd64.tar.gz
[root@dbserver src]# mkdir -p /usr/local/prometheus/
[root@dbserver src]# tar xvf mongodb_exporter-0.20.7.linux-amd64.tar.gz -C /usr/local/prometheus/
[root@dbserver src]# mv /usr/local/prometheus/mongodb_exporter-0.20.7.linux-amd64 /usr/local/prometheus/mongodb_exporter

11.2.将mongodb_exporter配置成系统服务

[root@dbserver src]# vi /etc/systemd/system/mongodb_exporter.service

[Unit]
Description=mongodb_exporter Server
Documentation=https://github.com/percona/mongodb_exporter
After=network.target

[Service]
ExecStart=/usr/local/prometheus/mongodb_exporter/mongodb_exporter 
  --web.listen-address=0.0.0.0:9216 
  --mongodb.uri=mongodb://username:password@127.0.0.1:27017/admin?replicaSet=rs0&&authSource=admin
ExecStop=/bin/kill -SIGINT $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target

11.3.通过systemctl启动mongodb_exporter

[root@dbserver src]# systemctl daemon-reload
[root@dbserver src]# systemctl start mongodb_exporter
[root@dbserver src]# systemctl status mongodb_exporter
[root@dbserver src]# systemctl enable mongodb_exporter

11.4.验证mongodb_exporter

[root@dbserver src]# curl "http://127.0.0.1:9216/metrics"

11.5.防火墙配置

[root@dbserver src]# firewall-cmd --permanent --add-port=9216/tcp
[root@dbserver src]# firewall-cmd --reload

#也可以限定仅能指定的IP访问
[root@dbserver src]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.100" port protocol="tcp" port="9216" accept"

#或可直接关闭防火墙
[root@dbserver src]# systemctl stop firewalld
[root@dbserver src]# systemctl disable firewalld

11.6.Prometheus侧配置

11.6.1.prometheus.yml配置调整

[root@dbserver src]# vi /usr/local/prometheus/prometheus/prometheus.yml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'mongodb_exporter'
    static_configs:
    - targets: ['192.168.1.100:9216']
      labels:
        service: 服务A-MongoDB

附:其他参考资料

• https://www.cnblogs.com/cyleon/category/1756369.html
• https://www.cnblogs.com/gaofeng-henu/category/1675861.html
• https://www.cnblogs.com/winstom/p/11926140.html
• https://www.cnblogs.com/caoweixiong/category/1622389.html
• https://www.bookstack.cn/read/prometheus-book/README.md