zoukankan      html  css  js  c++  java
  • Kubernetes集群资源监控

    监控指标

    一个好的系统,主要监控以下内容

    • 集群监控
      • 节点资源利用率
      • 节点数
      • 运行Pods
    • Pod监控
      • 容器指标
      • 应用程序【程序占用多少CPU、内存】

    监控平台

    使用普罗米修斯【prometheus】 + Grafana 搭建监控平台

    • prometheus【定时搜索被监控服务的状态】

      • 开源的
      • 监控、报警、数据库
      • 以HTTP协议周期性抓取被监控组件状态
      • 不需要复杂的集成过程,使用http接口接入即可
    • Grafana

      • 开源的数据分析和可视化工具
      • 支持多种数据源

    部署prometheus

    首先需要部署一个守护进程node-export.yaml

    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: node-exporter
      namespace: kube-system
      labels:
        k8s-app: node-exporter
    spec:
      selector:
        matchLabels:
          k8s-app: node-exporter
      template:
        metadata:
          labels:
            k8s-app: node-exporter
        spec:
          containers:
          - image: prom/node-exporter
            name: node-exporter
            ports:
            - containerPort: 9100
              protocol: TCP
              name: http
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        k8s-app: node-exporter
      name: node-exporter
      namespace: kube-system
    spec:
      ports:
      - name: http
        port: 9100
        nodePort: 31672
        protocol: TCP
      type: NodePort
      selector:
        k8s-app: node-exporter

    执行下面命令

    kubectl apply -f node-export.yaml

    然后通过yaml的方式部署prometheus

    [root@test prometheus]# ll
    total 20
    -rw-r--r-- 1 root root 5632 Dec 23 16:03 configmap.yaml
    -rw-r--r-- 1 root root 1115 Dec 23 16:04 prometheus.deploy.yml
    -rw-r--r-- 1 root root  238 Dec 23 16:05 prometheus.svc.yml
    -rw-r--r-- 1 root root  716 Dec 23 16:02 rbsc-setup.yaml
    • configmap:定义一个configmap:存储一些配置文件【不加密】
    • prometheus.deploy.yaml:部署一个deployment【包括端口号,资源限制】
    • prometheus.svc.yaml:对外暴露的端口
    • rbac-setup.yaml:分配一些角色的权限

    下面我们进入目录下,首先部署 rbac-setup.yaml

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: prometheus
    rules:
    - apiGroups: [""]
      resources:
      - nodes
      - nodes/proxy
      - services
      - endpoints
      - pods
      verbs: ["get", "list", "watch"]
    - apiGroups:
      - extensions
      resources:
      - ingresses
      verbs: ["get", "list", "watch"]
    - nonResourceURLs: ["/metrics"]
      verbs: ["get"]
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: prometheus
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: prometheus
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: prometheus
    subjects:
    - kind: ServiceAccount
      name: prometheus
      namespace: kube-system

    执行下面的命令

    kubectl apply -f rbac-setup.yaml

    然后分别部署

    configmap.yaml

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: prometheus-config
      namespace: kube-system
    data:
      prometheus.yml: |
        global:
          scrape_interval:     15s
          evaluation_interval: 15s
        scrape_configs:
    
        - job_name: 'kubernetes-apiservers'
          kubernetes_sd_configs:
          - role: endpoints
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
            action: keep
            regex: default;kubernetes;https
    
        - job_name: 'kubernetes-nodes'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics
    
        - job_name: 'kubernetes-cadvisor'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    
        - job_name: 'kubernetes-service-endpoints'
          kubernetes_sd_configs:
          - role: endpoints
          relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::d+)?;(d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
    
        - job_name: 'kubernetes-services'
          kubernetes_sd_configs:
          - role: service
          metrics_path: /probe
          params:
            module: [http_2xx]
          relabel_configs:
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox-exporter.example.com:9115
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name
    
        - job_name: 'kubernetes-ingresses'
          kubernetes_sd_configs:
          - role: ingress
          relabel_configs:
          - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
            regex: (.+);(.+);(.+)
            replacement: ${1}://${2}${3}
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox-exporter.example.com:9115
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_ingress_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_ingress_name]
            target_label: kubernetes_name
    
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
          - role: pod
          relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::d+)?;(d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name
    prometheus.deploy.yml
    ---
    apiVersion: apps/v1beta2
    kind: Deployment
    metadata:
      labels:
        name: prometheus-deployment
      name: prometheus
      namespace: kube-system
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: prometheus
      template:
        metadata:
          labels:
            app: prometheus
        spec:
          containers:
          - image: prom/prometheus:v2.0.0
            name: prometheus
            command:
            - "/bin/prometheus"
            args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus"
            - "--storage.tsdb.retention=24h"
            ports:
            - containerPort: 9090
              protocol: TCP
            volumeMounts:
            - mountPath: "/prometheus"
              name: data
            - mountPath: "/etc/prometheus"
              name: config-volume
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
              limits:
                cpu: 500m
                memory: 2500Mi
          serviceAccountName: prometheus    
          volumes:
          - name: data
            emptyDir: {}
          - name: config-volume
            configMap:
              name: prometheus-config   
    prometheus.svc.yml
    ---
    kind: Service
    apiVersion: v1
    metadata:
      labels:
        app: prometheus
      name: prometheus
      namespace: kube-system
    spec:
      type: NodePort
      ports:
      - port: 9090
        targetPort: 9090
        nodePort: 30003
      selector:
        app: prometheus

    执行命令

    # 部署configmap
    kubectl create -f configmap.yaml
    # 部署deployment
    kubectl create -f prometheus.deploy.yml
    # 部署svc
    kubectl create -f prometheus.svc.yml

    部署完成后用查看

    kubectl get pods -n kube-system

    在我们部署完成后,即可看到 prometheus 的 pod了,然后通过下面命令,能够看到对应的端口

    kubectl get svc -n kube-system

    通过这个,我们可以看到 prometheus 对外暴露的端口为 30003,访问页面即可对应的图形化界面

    http://10.120.10.3:30003        #ip 为本机的宿主机地址

    在上面我们部署完prometheus后,我们还需要来部署grafana

    grafana-deploy.yaml
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: grafana-core
      namespace: kube-system
      labels:
        app: grafana
        component: core
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: grafana
            component: core
        spec:
          containers:
          - image: grafana/grafana:4.2.0
            name: grafana-core
            imagePullPolicy: IfNotPresent
            # env:
            resources:
              # keep request = limit to keep this container in guaranteed class
              limits:
                cpu: 100m
                memory: 100Mi
              requests:
                cpu: 100m
                memory: 100Mi
            env:
              # The following env variables set up basic auth twith the default admin user and admin password.
              - name: GF_AUTH_BASIC_ENABLED
                value: "true"
              - name: GF_AUTH_ANONYMOUS_ENABLED
                value: "false"
              # - name: GF_AUTH_ANONYMOUS_ORG_ROLE
              #   value: Admin
              # does not really work, because of template variables in exported dashboards:
              # - name: GF_DASHBOARDS_JSON_ENABLED
              #   value: "true"
            readinessProbe:
              httpGet:
                path: /login
                port: 3000
              # initialDelaySeconds: 30
              # timeoutSeconds: 1
            volumeMounts:
            - name: grafana-persistent-storage
              mountPath: /var
          volumes:
          - name: grafana-persistent-storage
            emptyDir: {}
    grafana-svc.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: grafana
      namespace: kube-system
      labels:
        app: grafana
        component: core
    spec:
      type: NodePort
      ports:
        - port: 3000
      selector:
        app: grafana
        component: core
    grafana-ing.yaml
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
       name: grafana
       namespace: kube-system
    spec:
       rules:
       - host: k8s.grafana
         http:
           paths:
           - path: /
             backend:
              serviceName: grafana
              servicePort: 3000
    分别执行下面命令
    # 创建deployment
    kubectl create -f grafana-deploy.yaml
    # 创建svc
    kubectl create -f grafana-svc.yaml
    # 创建 ing
    kubectl create -f grafana-ing.yaml

    配置数据源

    下面我们需要开始打开 Grafana,然后配置数据源,导入数据显示模板

    我们可以通过 ip + 30431 访问我们的 grafana 图形化页面

      然后输入账号和密码:admin admin

    进入后,我们就需要配置 prometheus 的数据源

      和 对应的IP【这里IP是我们的ClusterIP】

      设置显示数据的模板

    选择Dashboard,导入我们的模板

      然后输入 315 号模板

      然后选择 prometheus数据源 mydb,导入即可

      导入后的效果如下所示

  • 相关阅读:
    Add Two Numbers
    Reverse Linked List II
    Reverse Linked List
    Remove Duplicates from Sorted List
    Remove Duplicates from Sorted List II
    Partition List
    Intersection of Two Linked Lists
    4Sum
    3Sum
    2Sum
  • 原文地址:https://www.cnblogs.com/fat-girl-spring/p/14183308.html
Copyright © 2011-2022 走看看