mtail + Grafana + node_exporter + Prometheus + jmx_exporter + alertManager + 钉钉 日志监控

参考资料: https://blog.csdn.net/bluuusea/article/details/104341054

下载搭建node_exporter(采集机器的负载情况)

tar zxvf node_exporter-1.3.1.linux-386.tar.gz

创建启动命令

cd node_exporter-1.3.1.linux-386/
touch start_node_exporter.sh
vim start_node_exporter.sh
nohup ./node_exporter> nodeout.file 2>&1 &

授权

chmod +x start_node_exporter.sh

验证

http://:9100/

mtail下载搭建(应用日志监控)

tar zxvf mtail_3.0.0-rc47_Linux_x86_64.tar.gz

此版本是可执行二进制文件，将其重命名为mtail，移动至/usr/local/sbin下即可使用

mv mtail /usr/local/sbin

验证

mtail -version

创建 配置文件路径

mkdir mtail_files
cd mtail_files/
touch qianlan-server.mtail
vim qianlan-server.mtail

counter login_success
/登录成功/ {
login_success++
}

赋权

chmod +x *

启动命令脚本

touch start_mtail.sh
vim start_mtail.sh
nohup mtail -port 3903 -logtostderr -progs /opt/mtail_files/qianlan-server.mtail -logs '/home/kuma/logs/prod/qianlan-/-127.0.0.1.log' 2>&1 &

验证

http://:3903/

查看进程是否启动成功

ps -ef | grep mtail

下载安装prometheus

tar zxvf prometheus-2.32.0-rc.0.linux-amd64.tar.gz

修改配置文件prometheus.yml添加node_exporter和matail 的采集路径

cd prometheus-2.32.0-rc.0.linux-amd64/
vim prometheus.yml

scrape_configs:

The job name is added as a label job=<job_name> to any timeseries scraped from this config.

• job_name: 'prometheus'
  static_configs:

  • targets: ['localhost:9090']

• job_name: 'node_exporter'
  static_configs:

  • targets: [':9100']

• job_name: '73_node_exporter'
  static_configs:

  • targets: [':9100']

• job_name: 'mtail'
  static_configs:

  • targets: [':3903']

• job_name: 'grafana'
  static_configs:

  • targets: ['localhost:3000']

以下内容为jmx-exporter应用配置

• job_name: 'student-jmx-exporter'
  scrape_interval: 10s
  metrics_path: '/metrics'
  static_configs:

  • targets: [ ':1000' ]
    labels:
    appname: 'student-jmx-exporter'

• job_name: 'student-jmx-exporter'
  static_configs:
  - targets: [':1000']

• job_name: 'mng-jmx-exporter'
  static_configs:
  - targets: [':1001']

touch start_promtool.sh

vim start_promtool.sh
nohup ./prometheus --config.file=prometheus.yml > prometheus.file 2>&1 &

http://:9090/graph

ps -ef |grep prometheus

grafana(图像展示)

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.1.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-8.3.1.linux-amd64.tar.gz
cd grafana-8.3.1/
touch start_grafana.sh
vim start_grafana.sh
nohup ./bin/grafana-server start > grafana.file 2>&1 &
chmod +x start_grafana.sh
ps -ef |grep grafana

http://:3000

配置数据源 prometheus

grafana添加 node_exporter 的Dashboard
导入id 8919 的桌面

jmx_exporter + Prometheus 监控jmx
简单说一下启动，官方给出的 jmx_exporter启动命令是：

java -javaagent:./jmx_prometheus_javaagent-0.13.0.jar=8080:config.yaml -jar yourJar.jar

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar

进入java服务中创建配置文件

cd /opt/qianlan-student/config
touch jmx_exporter.yml
chmod +x jmx_exporter.yml
vim jmx_exporter.yml

模板:
https://github.com/prometheus/jmx_exporter/blob/master/example_configs/tomcat.yml

lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem"]
blacklistObjectNames: []
rules:

• pattern: 'Tomcat<type=GlobalRequestProcessor, name="(\w+-\w+)-(\d+)"><>(\w+):'
  name: tomcat_$3_total
  labels:
  port: "$2"
  protocol: "$1"
  help: Tomcat global $3
  type: COUNTER
• pattern: 'Tomcat<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=_{_|!:.,;]*[-a-zA-Z0-9+&@#/%=}|]), name=([-a-zA-Z0-9+/$%~-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
  name: tomcat_servlet_$3_total
  labels:
  module: "$1"
  servlet: "$2"
  help: Tomcat servlet $3 total
  type: COUNTER
• pattern: 'Tomcat<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
  name: tomcat_threadpool_$3
  labels:
  port: "$2"
  protocol: "$1"
  help: Tomcat threadpool $3
  type: GAUGE
• pattern: 'Tomcat<type=Manager, host=([-a-zA-Z0-9+&@#/%?=_{_|!:.,;]*[-a-zA-Z0-9+&@#/%=}|]), context=([-a-zA-Z0-9+/$%~-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
  name: tomcat_session_$3_total
  labels:
  context: "$2"
  host: "$1"
  help: Tomcat session $3 total
  type: COUNTER
• pattern: 'java.lang<type=OperatingSystem><>(committed_virtual_memory|free_physical_memory|free_swap_space|total_physical_memory|total_swap_space)size:'
  name: os$1_bytes
  type: GAUGE
  attrNameSnakeCase: true
• pattern: 'java.lang<type=OperatingSystem><>((?!process_cpu_time)\w+):'
  name: os_$1
  type: GAUGE
  attrNameSnakeCase: true
• pattern: ".*"

jar 的方式启动

java -javaagent:/opt/lib/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar=1000:/opt/lib/jmx_exporter/jmx_exporter.yml -Dspring.config.location=def.properties -Dserver.port=1000 -jar

nohup ${JAVA_EXE} -Denv=${PROJECT_ENV} -Dapp.id=${APP_ID} -javaagent:/opt/lib/jmx_exporter/jmx_prometheus_javaagent-0.12.0.jar=1000:/opt/lib/jmx_exporter/jmx_exporter.yml -Dspring.config.location=def.properties -Dserver.port=10001
-jar ${PROJECT_HOME}/${PROJECT_JAR}
--spring.config.location=${PROJECT_CONFIG}
--logging.config=${PROJECT_CONFIG}/logback-spring.xml > /dev/null 2>&1 &

进入prometheus 检查配置是否生效

使用id为8563进行 grafana 配置。
修改 job 和 prometheus保持一致

打通钉钉机器人

mkdir /webhook
cd webhook/

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz
tar -zxvf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz

cd /opt/webhook/prometheus-webhook-dingtalk-1.4.0.linux-amd64

touch start_prometheus-webhook-dingtalk.sh
chmod +x start_prometheus-webhook-dingtalk.sh
vim start_prometheus-webhook-dingtalk.sh

内容

nohup ./prometheus-webhook-dingtalk --ding.profile=ops_dingding= > prometheus-webhook-dingtalk.log 2>&1 &

测试钉钉
curl https://oapi.dingtalk.com/robot/send?access_token= -X POST -H "Content-Type: application/json" -d '{"msgtype":"text","text":{"content":"prometheus - 钉钉告警测试"}}'

wget https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz

tar -zxvf alertmanager-0.23.0.linux-amd64.tar.gz
cd alertmanager-0.23.0.linux-amd64
vim alertmanager.yml
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'ops_dingding'
receivers:

• name: 'ops_dingding'
  webhook_configs:
  • url: 'http://localhost:8060/dingtalk/ops_dingding/send'
    send_resolved: true
    inhibit_rules:
  • source_match:
    severity: 'critical'
    target_match:
    severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

touch start_alert_menager.sh
chmod +x start_alert_menager.sh
vim start_alert_menager.sh

nohup ./alertmanager --config.file=alertmanager.yml > alertmanager.log 2>&1 &

配置prometheus
vim prometheus.yml

Alertmanager configuration

alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093

Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

• "rules/*_rules.yml"

- "second_rules.yml"