我们用的是阿里云托管的K8S集群1.21版本，用的 kube-prometheus 0.9 版本，如果你也是用的阿里云托管的ACK，提前提工单打开授权管理，不然安装的时候会找不到RoleBinding。
参考文档：
http://www.servicemesher.com/blog/prometheus-operator-manual/ https://github.com/coreos/prometheus-operator https://github.com/coreos/kube-prometheus
https://www.cnblogs.com/twobrother/p/11165417.html
1、概述
1.1在k8s中部署Prometheus监控的方法
通常在k8s中部署prometheus监控可以采取的方法有以下三种
- 通过yaml手动部署
- operator部署
- 通过helm chart部署
1.2 什么是Prometheus Operator
Prometheus Operator的本职就是一组用户自定义的CRD资源以及Controller的实现，Prometheus Operator负责监听这些自定义资源的变化，并且根据这些资源的定义自动化的完成如Prometheus Server自身以及配置的自动化管理工作。以下是Prometheus Operator的架构图：
在配置prometheus-operator 监控jvm之前，我们必须要了解prometheus-operator的4个crd组件，这四个CRD作用如下：
Prometheus: 由 Operator 依据一个自定义资源kind: Prometheus类型中，所描述的内容而部署的 Prometheus Server 集群，可以将这个自定义资源看作是一种特别用来管理Prometheus Server的StatefulSets资源。
ServiceMonitor: 一个Kubernetes自定义资源(和kind: Prometheus一样是CRD)，该资源描述了Prometheus Server的Target列表，Operator 会监听这个资源的变化来动态的更新Prometheus Server的Scrape targets并让prometheus server去reload配置(prometheus有对应reload的http接口/-/reload)。而该资源主要通过Selector来依据 Labels 选取对应的Service的endpoints，并让 Prometheus Server 通过 Service 进行拉取（拉）指标资料(也就是metrics信息),metrics信息要在http的url输出符合metrics格式的信息,ServiceMonitor也可以定义目标的metrics的url.
Alertmanager：Prometheus Operator 不只是提供 Prometheus Server 管理与部署，也包含了 AlertManager，并且一样通过一个 kind: Alertmanager 自定义资源来描述信息，再由 Operator 依据描述内容部署 Alertmanager 集群。
PrometheusRule:对于Prometheus而言，在原生的管理方式上，我们需要手动创建Prometheus的告警文件，并且通过在Prometheus配置中声明式的加载。而在Prometheus Operator模式中，告警规则也编程一个通过Kubernetes API 声明式创建的一个资源.告警规则创建成功后，通过在Prometheus中使用想servicemonitor那样用ruleSelector通过label匹配选择需要关联的PrometheusRule即可。
2.安装部署
1.下载部署包
wget -c https://github.com/prometheus-operator/kube-prometheus/archive/v0.7.0.zip
2.修改文件
其中kubelet的metrics采集端口，10250是https的，10255是http的
kube-scheduler的metrics采集端，10259是https的，10251是http的
Kube-controller的metrics采集端，10257是https的，10252是http的
测试：在主机上curl相关端口/metrics，即可获取相关metrics，如获取kubelet相关指标只需curl http://127.0.0.1:10255/metrics即可
-
kubernetes-serviceMonitorKubeScheduler.yaml
-
kubernetes-serviceMonitorKubeControllerManager.yaml
-
kubernetes-serviceMonitorKubelet.yaml
Yaml文件中相关信息采集默认采用https的端口，即10250端口，这样我们需要将port的端口改为http-metrics,同样的scheme改为http
参考:https://www.cnblogs.com/xinbat/p/15116903.html
3.部署
# cd kube-prometheus\manifests\setup
# kubectl apply .
# cd kube-prometheus\manifests\
# kubectl apply .
为prometheus、grafana、alertmanager 创建 ingress:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: prometheus-alertmangaer-grafana-ingress namespace: monitoring annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/ssl-redirect: 'true' nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" nginx.ingress.kubernetes.io/proxy-read-timeout: "600" nginx.ingress.kubernetes.io/proxy-send-timeout: "600" nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive" nginx.ingress.kubernetes.io/proxy-http-version: "1.1" nginx.ingress.kubernetes.io/proxy-body-size: 80m spec: tls: - hosts: - 'prometheus.xxx.com' secretName: xxx-com-secret - hosts: - 'grafana.xxx.com' secretName: xxx-com-secret - hosts: - 'alertmanager.xxx.com' secretName: xxx-com-secret rules: - host: prometheus.xxx.com http: paths: - path: / backend: serviceName: prometheus-k8s servicePort: 9090 - host: grafana.xxx.com http: paths: - path: / backend: serviceName: grafana servicePort: 3000 - host: alertmanager.xxx.com http: paths: - path: / backend: serviceName: alertmanager-main servicePort: 9093
解决Watchdog、ControllerManager、Scheduler监控问题
Watchdog是一个正常的报警，这个告警的作用是：如果alermanger或者prometheus本身挂掉了就发不出告警了，因此一般会采用另一个监控来监控prometheus，或者自定义一个持续不断的告警通知，哪一天这个告警通知不发了，说明监控出现问题了。prometheus operator已经考虑了这一点，本身携带一个watchdog，作为对自身的监控。
如果需要关闭，删除或注释掉Watchdog部分
prometheus-rules.yaml ... - name: general.rules rules: - alert: TargetDown annotations: message: 'xxx' expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job, namespace, service)) > 10 for: 10m labels: severity: warning # - alert: Watchdog # annotations: # message: | # This is an alert meant to ensure that the entire alerting pipeline is functional. # This alert is always firing, therefore it should always be firing in Alertmanager # and always fire against a receiver. There are integrations with various notification # mechanisms that send a notification when this alert is not firing. For example the # "DeadMansSnitch" integration in PagerDuty. # expr: vector(1) # labels: # severity: none
对应的Watchdog的ServiceMonitor也可以删除。
KubeControllerManagerDown、KubeSchedulerDown的解决
原因是因为在prometheus-serviceMonitorKubeControllerManager.yaml中有如下内容，但默认安装的集群并没有给系统kube-controller-manager组件创建svc
selector:
matchLabels:
k8s-app: kube-controller-manager
修改kube-controller-manager的监听地址：
# vim /etc/kubernetes/manifests/kube-controller-manager.yaml ... spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=0.0.0.0
# netstat -lntup|grep kube-contro tcp6 0 0 :::10257 :::* LISTEN 38818/kube-controll
创建
prometheus-kube-controller-manager-service.yaml
prometheus-kube-scheduler-service.yaml，以便serviceMonitor监听
# cat prometheus-kube-controller-manager-service.yaml apiVersion: v1 kind: Service metadata: namespace: kube-system name: kube-controller-manager labels: k8s-app: kube-controller-manager spec: selector: component: kube-controller-manager ports: - name: http-metrics port: 10252 targetPort: 10252 protocol: TCP
# cat prometheus-kube-scheduler-service.yaml apiVersion: v1 kind: Service metadata: namespace: kube-system name: kube-scheduler labels: k8s-app: kube-scheduler spec: selector: component: kube-scheduler ports: - name: http-metrics port: 10251 targetPort: 10251 protocol: TCP
#10251是kube-scheduler组件 metrics 数据所在的端口，10252是kube-controller-manager组件的监控数据所在端口。
上面 labels 和 selector 部分，labels 区域的配置必须和我们上面的 ServiceMonitor 对象中的 selector 保持一致，selector下面配置的是component=kube-scheduler，为什么会是这个 label 标签呢，我们可以去 describe 下 kube-scheduelr 这个 Pod
# kubectl describe pod kube-scheduler-k8s-master -n kube-system Name: kube-scheduler-k8s-master Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: k8s-master/10.6.76.25 Start Time: Thu, 29 Aug 2019 09:21:01 +0800 Labels: component=kube-scheduler tier=control-plane # kubectl describe pod kube-controller-manager-k8s-master -n kube-system Name: kube-controller-manager-k8s-master Namespace: kube-system Priority: 2000000000 Priority Class Name: system-cluster-critical Node: k8s-master/10.6.76.25 Start Time: Thu, 29 Aug 2019 09:21:01 +0800 Labels: component=kube-controller-manager tier=control-plane
浏览器ingress方式访问
https://prometheus.xxx.com/
https://alertmanager.xxx.com/
https://grafana.xxx.com/
grafana默认账号密码admin admin需要重置密码进入
参考:
https://www.cnblogs.com/huss2016/p/14865316.html
http://t.zoukankan.com/ssgeek-p-14441149.html
https://www.cnblogs.com/xinbat/p/15116903.html
https://www.cnblogs.com/zhangrui153169/p/13609172.html
https://blog.csdn.net/twingao/article/details/105261641
https://www.cnblogs.com/twobrother/p/11165417.html
https://blog.csdn.net/qq_43164571/article/details/119990724
https://www.kococ.cn/20210302/cid=697.html