zoukankan      html  css  js  c++  java
  • k8s集群监控并实现HPA自动扩缩容

    部署metrics

    kubernetes早期版本依靠Heapster来实现完整的性能数据采集和监控功能,k8s在1.8版本开始,性能数据开始以Metrics API的方式提供标准化接口,并且从1.10版本开始讲Heapster替换为Metrics Server,在新版本的Metrics当中可以对Node,Pod的cpu,内存的使用指标进行监控

    我们可以先在k8s集群当中用kubectl top命令去尝试查看资源使用率,如下:

    可以看到并不能查看资源,这是因为没有安装Metrics

    [root@master redis]# kubectl top nodes
    Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

    Metrics部署文件在github上可以找到,如下:

     我们点进去然后把它的代码下载下来

    [root@master test]# git clone https://github.com/kubernetes-sigs/metrics-server.git
    正克隆到 'metrics-server'...
    remote: Enumerating objects: 53, done.
    remote: Counting objects: 100% (53/53), done.
    remote: Compressing objects: 100% (43/43), done.
    remote: Total 11755 (delta 12), reused 27 (delta 3), pack-reused 11702
    接收对象中: 100% (11755/11755), 12.35 MiB | 134.00 KiB/s, done.
    处理 delta 中: 100% (6113/6113), done.

    进入部署代码目录

    [root@master test]# cd metrics-server/deploy/kubernetes/

    在metrics-server-deployment.yaml 我们需要添加如下4行内容

     开始部署文件

    [root@master kubernetes]# kubectl apply -f .
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
    serviceaccount/metrics-server created
    deployment.apps/metrics-server created
    service/metrics-server created
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

    查看metrics pod的状态

     查看api

     我们可以用kubectl proxy来尝试访问这个新的api

     没有任何问题,接下来我们在去尝试用kubectl top命令,可以看到已经可以正常使用了

     部署grafana+prometheus集群性能监控平台

    Prometheus是有SoundCloud公司开发的开源监控系统,是继kubernetes之后CNCF第二个毕业项目,在容器和微服务领域得到了广泛应用,Prometheus有以下特点:

    • 使用指标名称及键值对标识的多维度数据模型
    • 采用灵活的查询语言PromQL
    • 不依赖分式存储,为自治的单节点服务
    • 使用HTTP完成对监控数据的拉取
    • 支持通过网关推送时序数据
    • 支持多种图形和dashboard的展示,例如Grafana

    prometheus架构图

     开始部署第一步

     1.下载文件到本地

    [root@master test]# git clone https://github.com/iKubernetes/k8s-prom.git
    正克隆到 'k8s-prom'...
    remote: Enumerating objects: 49, done.
    remote: Total 49 (delta 0), reused 0 (delta 0), pack-reused 49
    Unpacking objects: 100% (49/49), done.

    2.创建名称空间

    [root@master k8s-prom]# kubectl apply -f namespace.yaml 
    namespace/prom created

    3.部署k8s-prom中的node_exporter/中的yaml文件来让prometheus获取数据

    [root@master k8s-prom]# kubectl apply -f node_exporter/
    daemonset.apps/prometheus-node-exporter created
    service/prometheus-node-exporter created

    4.部署prometheus/中的yaml文件

    [root@master k8s-prom]# kubectl apply -f prometheus/
    configmap/prometheus-config created
    deployment.apps/prometheus-server created
    clusterrole.rbac.authorization.k8s.io/prometheus created
    serviceaccount/prometheus created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus created
    service/prometheus created

    5.部署k8s-prom中的k8s-prometheus-adapter/中的文件,但是由于这个文件中用的是http协议,但是我们k8s当中用的是https协议,所以在部署前需要创建秘钥

    [root@master k8s-prom]# cd /etc/kubernetes/pki/
    [root@master pki]# (umask 077;openssl genrsa -out serving.key 2048)
    Generating RSA private key, 2048 bit long modulus
    .....+++
    .....................................+++
    e is 65537 (0x10001)
    [root@master pki]# openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
    [root@master pki]# openssl x509 -req -in serving.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out serving.crt -days 3650
    Signature ok
    subject=/CN=serving
    Getting CA Private Key
    [root@master pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key -n prom 
    secret/cm-adapter-serving-certs created

    6.部署k8s-prom中的k8s-prometheus-adapter/中的文件

    [root@master k8s-prom]# kubectl apply -f k8s-prometheus-adapter/
    clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
    rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
    deployment.apps/custom-metrics-apiserver created
    clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
    serviceaccount/custom-metrics-apiserver created
    service/custom-metrics-apiserver created
    apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
    clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
    configmap/adapter-config created
    clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
    clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created

    7.部署kube-state-metrics中的yaml文件

    [root@master k8s-prom]# kubectl apply -f kube-state-metrics/

    8.部署grafana.yaml文件

    [root@master k8s-prom]# kubectl apply -f grafana.yaml 
    deployment.apps/monitoring-grafana created
    service/monitoring-grafana created

    9.验证各个pod运行无误

    [root@master k8s-prom]# kubectl get ns
    NAME              STATUS   AGE
    default           Active   19h
    kube-node-lease   Active   19h
    kube-public       Active   19h
    kube-system       Active   19h
    prom              Active   10m
    [root@master k8s-prom]# kubectl get pods -n prom
    NAME                                        READY   STATUS    RESTARTS   AGE
    custom-metrics-apiserver-7666fc78cc-xlnzn   1/1     Running   0          3m25s
    monitoring-grafana-846dd49bdb-8gpkw         1/1     Running   0          61s
    prometheus-node-exporter-45qxt              1/1     Running   0          8m28s
    prometheus-node-exporter-6mhwn              1/1     Running   0          8m28s
    prometheus-node-exporter-k6d7m              1/1     Running   0          8m28s
    prometheus-server-69b544ff5b-9mk9x          1/1     Running   0          107s
    [root@master k8s-prom]# kubectl get svc -n prom
    NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
    custom-metrics-apiserver   ClusterIP   10.98.67.254    <none>        443/TCP          4m2s
    monitoring-grafana         NodePort    10.102.49.116   <none>        80:30080/TCP     97s
    prometheus                 NodePort    10.107.21.128   <none>        9090:30090/TCP   2m24s
    prometheus-node-exporter   ClusterIP   None            <none>        9100/TCP         9m5s

    10.打开浏览器

     配置为

     导入模板

     

    创建HPA

    动态扩缩容

    创建1个pod

    kubectl run myapp --image=liwang7314/myapp:v1 --replicas=1 --requests='cpu=50m,memory=50Mi' --limits='cpu=50m,memory=50Mi' --labels='app=myapp' --expose --port=80

    创建hpa

    kubectl autoscale deployment myapp --min=1 --max=8 --cpu-percent=60

    使用ab工具压力测试,模拟100人访问50w次

    ab -c 100 -n 500000 http://192.168.254.12:30958/index.html

    观察pod数量

    [root@master k8s-prom]# kubectl get pods -o wide -w
    NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES
    myapp-54fbc848b4-7lf9m   1/1     Running   0          3m45s   10.244.1.32   node1   <none>           <none>
    myapp-54fbc848b4-7v6fq   0/1     Pending   0          0s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-7v6fq   0/1     Pending   0          1s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-7v6fq   0/1     ContainerCreating   0          1s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-7v6fq   1/1     Running             0          7s      10.244.2.42   node2    <none>           <none>
    myapp-54fbc848b4-frrp2   0/1     Pending             0          0s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-frrp2   0/1     Pending             0          1s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-6vttw   0/1     Pending             0          0s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-6vttw   0/1     Pending             0          0s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-frrp2   0/1     ContainerCreating   0          1s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-6vttw   0/1     ContainerCreating   0          0s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-frrp2   1/1     Running             0          5s      10.244.1.33   node1    <none>           <none>
    myapp-54fbc848b4-6vttw   1/1     Running             0          13s     10.244.2.43   node2    <none>           <none>
    myapp-54fbc848b4-t5pgq   0/1     Pending             0          1s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-t5pgq   0/1     Pending             0          4s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-lnmns   0/1     Pending             0          3s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-jw8kp   0/1     Pending             0          4s      <none>        <none>   <none>           <none>
    myapp-54fbc848b4-lnmns   0/1     Pending             0          4s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-jw8kp   0/1     Pending             0          5s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-t5pgq   0/1     ContainerCreating   0          9s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-lnmns   0/1     ContainerCreating   0          7s      <none>        node2    <none>           <none>
    myapp-54fbc848b4-jw8kp   0/1     ContainerCreating   0          7s      <none>        node1    <none>           <none>
    myapp-54fbc848b4-t5pgq   1/1     Running             0          11s     10.244.1.34   node1    <none>           <none>

    查看HPA

    [root@master ~]# kubectl describe hpa myapp 
    Name:                                                  myapp
    Namespace:                                             default
    Labels:                                                <none>
    Annotations:                                           <none>
    CreationTimestamp:                                     Sat, 14 Mar 2020 22:42:49 +0800
    Reference:                                             Deployment/myapp
    Metrics:                                               ( current / target )
      resource cpu on pods  (as a percentage of request):  100% (50m) / 60%
    Min replicas:                                          1
    Max replicas:                                          8
    Deployment pods:                                       7 current / 7 desired
    Conditions:
      Type            Status  Reason              Message
      ----            ------  ------              -------
      AbleToScale     True    ReadyForNewScale    recommended size matches current size
      ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
      ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    Events:
      Type    Reason             Age    From                       Message
      ----    ------             ----   ----                       -------
      Normal  SuccessfulRescale  3m49s  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
      Normal  SuccessfulRescale  2m47s  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
      Normal  SuccessfulRescale  78s    horizontal-pod-autoscaler  New size: 7; reason: cpu resource utilization (percentage of request) above target
  • 相关阅读:
    Synchronized和Lock的实现原理和锁升级
    如何禁止CPU指令重排
    MESI缓存一致性
    归并排序
    强软弱虚四种引用和ThreadLocal内存泄露
    VINS-Mono代码分析与总结(完整版)
    IMU误差模型与校准
    小感
    K8S conul部署
    Centos Consul集群及Acl配置
  • 原文地址:https://www.cnblogs.com/fengzi7314/p/12494298.html
Copyright © 2011-2022 走看看