Prometheus监控k8s中使用exporter的服务

一、监控使用exporter暴露metrics信息的服务

　　对于一些应用如：kafka、redis或者mysql等，其需要使用exporter来暴露本身的metrics信息。这些服务的监控，可以使用prometheus operator中的servicemonitor来匹配该服务的exporter的service，达到采集metrics的目的。

　　现对kafka进行数据采集和监控

　　kafka_exporter地址：https://github.com/danielqsj/kafka_exporter

　　安装kafka_exporter

　　kafka-exporter.yaml　

apiVersion: v1
kind: Service
metadata:
  name: kafka-exporter
  namespace: monitoring
  labels:
    k8s-app: kafka-exporter
spec:
  selector:
    k8s-app: kafka-exporter
  ports:
  - name: kafka-exporter
    port: 9308
    protocol: TCP
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-exporter
  namespace: monitoring
  labels:
    k8s-app: kafka-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kafka-exporter
  template:
    metadata:
      labels:
        k8s-app: kafka-exporter
    spec:
      containers:
      - name: kafka-exporter
        image: danielqsj/kafka-exporter:latest
        imagePullPolicy: IfNotPresent
        args: ["--kafka.server=kafka-0.kafka-headless.public-service:9092"]
        resources:
          requests:
            memory: 100M
        volumeMounts:
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
      nodeSelector:
        monitor: "true"
      volumes:
        - name: tz-config
          hostPath:
            path: /usr/share/zoneinfo/Asia/Shanghai

• 创建了kafka_exporter的service和deployment，service用于和Prometheus servicemonitor进行匹配，采集数据
• args：Kafka_exporter启动参数，为k8s集群中kafka的地址

查看

# kubectl get svc,deploy -n monitoring -l k8s-app=kafka-exporter
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kafka-exporter   ClusterIP   10.98.228.115   <none>        9308/TCP   50m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kafka-exporter   1/1     1            1           49m

测试

# curl 10.98.228.115:9308/metrics | more


# TYPE go_gc_duration_seconds summary
1086k  go_gc_duration_seconds{quantile="0"} 7.7719e-05
 go_gc_duration_seconds{quantile="0.25"} 0.00010556
  go_gc_duration_seconds{quantile="0.5"} 0.0001406
 0 go_gc_duration_seconds{quantile="0.75"} 0.00019177
--go_gc_duration_seconds{quantile="1"} 0.001047939
:-go_gc_duration_seconds_sum 0.004194563
-:go_gc_duration_seconds_count 21
--# HELP go_goroutines Number of goroutines that currently exist.
 -# TYPE go_goroutines gauge
-go_goroutines 11
...

• kafka中有topic时，exporter才能采集更多的数据

二、配置servicemonitor

　　prometheus-serviceMonitorKafka.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: kafka-exporter
  name: kafka-exporter
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: kafka-exporter
    scheme: http
    tlsConfig:
      insecureSkipVerify: false
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
      k8s-app: kafka-exporter

• 该servicemonitor匹配的是 monitoring这个namespace下，label为k8s-app=kafka-exporter这个service

查看Prometheus