微服务监控之三：Prometheus + Grafana Spring Boot 应用可视化监控

一、Springboot增加Prometheus

1、Spring Boot 应用暴露监控指标，添加如下依赖

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
 
<dependency>
    <groupId>io.prometheus</groupId>
    <artifactId>simpleclient_spring_boot</artifactId>
    <version>0.0.26</version>
</dependency>

gradle配置：

    compile 'org.springframework.boot:spring-boot-starter-actuator'
    compile 'io.prometheus:simpleclient_spring_boot:0.0.26'

2、然后，在启动类 Application.java 添加如下注解：

@SpringBootApplication
@EnablePrometheusEndpoint
@EnableSpringBootMetricsCollector
public class Application {
 
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
 
}

3、配置文件设置

在application.xml里设置属性：spring.metrics.servo.enabled=false，  
去掉重复的metrics，不然在prometheus的控制台的targets页签里，会一直显示此endpoint为down状态。  

#应用可视化监控  
management.security.enabled=false  
spring.metrics.servo.enabled=false  

4、访问：http://192.168.10.213:6010/prometheus，可以看到 Prometheus 格式的指标数据

二、自定义prometheus注解

2.1、自定义prometheus注解

import java.lang.annotation.*;

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface PrometheusMetrics {
    /**
     * 默认为空,程序使用method signature作为Metric name 如果name有设置值,使用name作为Metric name
     * 
     * @return
     */
    String name() default "";
}

2.2、自定义prometheus切面

import io.prometheus.client.Counter;
import io.prometheus.client.Histogram;
import org.apache.commons.lang3.StringUtils;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.stereotype.Component;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.ServletRequestAttributes;
import javax.servlet.http.HttpServletRequest;

@Aspect
@Component
public class PrometheusMetricsAspect {
    private static final Counter requestTotal = Counter.build().name("couter_all").labelNames("api")
            .help("total request couter of api").register();
    private static final Counter requestError = Counter.build().name("couter_error").labelNames("api")
            .help("response Error couter of api").register();
    private static final Histogram histogram = Histogram.build().name("histogram_consuming").labelNames("api")
            .help("response consuming of api").register();

    // 自定义Prometheus注解的全路径
    @Pointcut("@annotation(com....annotation.PrometheusMetrics)")
    public void pcMethod() {
    }

    @Around(value = "pcMethod() && @annotation(annotation)")
    public Object MetricsCollector(ProceedingJoinPoint joinPoint, PrometheusMetrics annotation) throws Throwable {
        MethodSignature methodSignature = (MethodSignature) joinPoint.getSignature();
        PrometheusMetrics prometheusMetrics = methodSignature.getMethod().getAnnotation(PrometheusMetrics.class);
        if (prometheusMetrics != null) {
            String name;
            if (StringUtils.isNotEmpty(prometheusMetrics.name())) {
                name = prometheusMetrics.name();
            } else {
                HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes())
                        .getRequest();
                name = request.getRequestURI();
            }
            requestTotal.labels(name).inc();
            Histogram.Timer requestTimer = histogram.labels(name).startTimer();
            Object object;
            try {
                object = joinPoint.proceed();
            } catch (Exception e) {
                requestError.labels(name).inc();
                throw e;
            } finally {
                requestTimer.observeDuration();
            }
            return object;
        } else {
            return joinPoint.proceed();
        }
    }
}

2.3、被监控的方法上添加--自定义prometheus注解

    @PrometheusMetrics
    @PostMapping(value = "isBacklist")
    @ApiOperation(value = "黑名单判断", notes = "是否在黑名单中，如果存在并且记录状态为2，则为黑名单，返回true，否则返回：false")
    @Log
    public RespResult<Boolean> isBacklist(@RequestBody BacklistReqDTO reqDTO) {

三、Prometheus 采集 Spring Boot 指标数据

首先，获取 Prometheus 的 Docker 镜像：

docker pull prom/prometheus

 

3.1、然后，编写配置文件 prometheus.yml ：

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

 # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

static_configs:
- targets: ['10.200.110.100:8080']   #此处填写 Spring Boot 应用的 IP + 端口号

3.2、接着，启动 Prometheus :

docker run -d -p 9090:9090 
-u root 
-v /opt/prometheus/tsdb:/etc/prometheus/tsdb 
-v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml 
--privileged=true prom/prometheus 
--storage.tsdb.path=/etc/prometheus/tsdb 
--storage.tsdb.retention=7d 
--config.file=/etc/prometheus/prometheus.yml

非docker环境的启动方式：

./prometheus --config.file=prometheus2.yml

 结果：

duanxz@ubuntu:~/Downloads/prometheus-2.0.0.linux-amd64$ ./prometheus --config.file=prometheus2.yml
level=info ts=2018-06-19T08:27:47.222527495Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2018-06-19T08:27:47.222895906Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
level=info ts=2018-06-19T08:27:47.223110655Z caller=main.go:217 host_details="(Linux 4.4.0-128-generic #154~14.04.1-Ubuntu SMP Fri May 25 14:58:51 UTC 2018 x86_64 ubuntu (none))"
level=info ts=2018-06-19T08:27:47.227443134Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-19T08:27:47.234616341Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2018-06-19T08:27:47.244932582Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
level=info ts=2018-06-19T08:27:47.24608357Z caller=main.go:326 msg="TSDB started"
level=info ts=2018-06-19T08:27:47.246514727Z caller=main.go:394 msg="Loading configuration file" filename=prometheus2.yml
level=info ts=2018-06-19T08:27:47.247799187Z caller=main.go:371 msg="Server is ready to receive requests."

最后，访问 http://127.0.0.1:9090/targets , 检查 Spring Boot 采集状态是否正常。

 

四、Grafana 可视化监控数据

首先，获取 Grafana 的 Docker 镜像：

docker pull grafana/grafana

 

然后，启动 Grafana：

docker run -d -p 3000:3000

-v /opt/grafana:/var/lib/grafana

-e "GF_SMTP_ENABLED=true"

-e "GF_SMTP_HOST=smtp.139.com:25"

-e "GF_SMTP_USER=13616052510@139.com"

-e "GF_SMTP_PASSWORD=like110120"

-e "GF_SMTP_FROM_ADDRESS=13616052510@139.com"

--privileged=true grafana/grafana

 

接着，访问 http://localhost:3000/ 配置 Prometheus 数据源：

Grafana 登录账号 admin 密码 admin

 

五、常用Prometheus 表达式

QPS[5分钟]

rate(lz_http_requests_total{job="02_lzmh_microservice_base_service_docker"}[5m]) > 0

QPS[5分钟]，根据handler分组

sum(rate(lz_http_requests_total{job="lzmh_microservice_weixin_applet_api"}[5m])) by (handler) > 0

 

平均响应时间[5分钟]

(

rate(lz_http_response_time_milliseconds_sum{job="02_lzmh_microservice_base_service_docker"}[5m]) 

/

rate(lz_http_response_time_milliseconds_count{job="02_lzmh_microservice_base_service_docker"}[5m])

) > 0

平均响应时间[5分钟]，根据handler分组

sum(

rate(lz_http_response_time_milliseconds_sum{job="lzmh_microservice_weixin_applet_api"}[5m]) 

/

rate(lz_http_response_time_milliseconds_count{job="lzmh_microservice_weixin_applet_api"}[5m])

) by (handler) > 0

参考：http://www.spring4all.com/article/265