HPA 自动水平伸缩(基于CPU)

自动水平伸缩，是指运行在k8s上的应用负载(POD)，可以根据资源使用率进行自动扩容、缩容，它依赖metrics-server服务pod使用资源指标收集；我们知道应用的资源使用率通常都有高峰和低谷，所以k8s的HPA特性应运而生；它也是最能体现区别于传统运维的优势之一，不仅能够弹性伸缩，而且完全自动化！

我们在生产中通常用得最多的就是基于服务pod的cpu使用率metrics来自动扩容pod数量，下面来以生产的标准来实战测试下（注意：使用HPA前我们要确保K8s集群的dns服务和metrics服务是正常运行的，并且我们所创建的服务需要配置指标分配）

Github地址:

https://github.com/kubernetes/kubernetes/blob/release-1.19/cluster/addons/metrics-server/

解决不能下载google 镜像仓库的问题：

由于无法下载google的k8s.gcr.io镜像，选择国内的镜像代理站点下载同步镜像：

例如：
docker pull k8s.gcr.io/metrics-server-amd64:v0.3.6
替换成阿里云镜像
docker pull registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6

*** 如果生产使用建议放在harbor上.

1 安装前修改kube-apiserver.yaml配置文件:

vim /etc/kubernetes/manifests/kube-apiserver.yaml 

。。。。。。省略
    - --enable-bootstrap-token-auth=true
    - --enable-aggregator-routing=true                      # 添加此行
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
。。。。。。省略

***  一定要添加上: "- --enable-aggregator-routing=true"  保证有这行.

2 安装metrics-server v0.3.6

mkdir metrics-server
cd metrics-server

#2.1  下载yaml文件：
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

#2.2  修改components.yaml文件
vim components.yaml

......其他不做变动省略即可......
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        #image: k8s.gcr.io/metrics-server-amd64:v0.3.6                                  #  注释原行
        image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6      #  替换成阿里云的代理镜像
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - /metrics-server                                                              #新增
          - --kubelet-preferred-address-types=InternalIP                                 #新增
          - --kubelet-insecure-tls                                                       #新增
        ports:
        - name: main-port
......其他不做变动省略即可......


## 2.3 安装metrics-server ：
kubectl apply -f components.yaml

## 2.4 查看metrics-server服务状态：
[root@k8s-master1 data]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-6ddbc8ff55-zwqdm                   1/1     Running   0          38m

## 2.5 检查接口是否有异常：
[root@k8s-master1 data]# kubectl api-versions
......
events.k8s.io/v1beta1
extensions/v1beta1
metrics.k8s.io/v1beta1          # 需要有此行
networking.k8s.io/v1
networking.k8s.io/v1beta1
......

[root@k8s-master1 data]# kubectl describe apiservice v1beta1.metrics.k8s.io   # 看到下面的信息API接口即正常
Name:         v1beta1.metrics.k8s.io
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2021-11-11T09:22:34Z
  Managed Fields:
    API Version:  apiregistration.k8s.io/v1
 ......省略部分

3 配置一个deploment，并添加资源限制条件：

## pod内资源分配的配置格式如下：
## 默认可以只配置requests，但根据生产中的经验，建议把limits资源限制也加上，因为对K8s来说，只有这两个都配置了且配置的值都要一样，这个pod资源的优先级才是最高的，在node资源不够的情况下，首先是把没有任何资源分配配置的pod资源给干掉，其次是只配置了requests的，最后才是两个都配置的情况，仔细品品

vim deployment-nginx.yaml

kind: Deployment
metadata:
  labels:
    app: ng-web
  name: ng-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ng-web
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: ng-web
    spec:
      containers:
      - image: 192.168.10.19/devops/nginx:1.14.0
        name: nginx
        resources:                                    # 资源使用
          limits:                                     # (最高)限制单个POD 最多能使用1核(1000m 毫核)CPU以及2G内存
            cpu: '1'
            memory: 20Gi
          requests:                                   # (最低)保证此pod 初始获取这么多资源
            cpu: '1'
            memory: 20Mi
        ports:
        - containerPort: 80
        volumeMounts:
        - name: my-nfs
          mountPath: /usr/share/nginx/html
        - name: k8snfs-db
          mountPath: /data/nginx/html
      volumes:
       - name: my-nfs
         nfs:
           server: 192.168.10.19
           path: /data/k8sdata
       - name: k8snfs-db
         nfs:
           server: 192.168.10.19
           path: /data/k8sdb
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ng-web
  name: ng-web
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: ng-web
  type: ClusterIP

****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** ****** 

# 安装部署测试的deployment：
kubectl apply -f deployment-nginx.yaml

# 查看pod 的资源使用:
[root@k8s-master1 data]# kubectl top pod
NAME                      CPU(cores)   MEMORY(bytes)   
ng-web-59f56fdfb6-58ch6   0m           1Mi             
ng-web-59f56fdfb6-9ds9x   0m           1Mi             
ng-web-59f56fdfb6-xmnks   0m           2Mi             

# 查看Node 使用情况:
[root@k8s-master1 data]# kubectl top node
NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master1.example.com   242m         12%    1172Mi          61%       
node1.example.com         66m          3%     505Mi           56%       
node2.example.com         56m          2%     520Mi           58%       
node3.example.com         80m          4%     534Mi           60% 

4 为测试deployment，创建一个HPA:

# 为deployment资源web创建HPA，pod数量上限3个，最低1个，在pod平均CPU达到50%后开始扩容
 kubectl  autoscale deployment ng-web --max=3 --min=1 --cpu-percent=50

# 下面提示说到，HPA缺少最小资源分配的request参数:
[root@k8s-master1 linux39_ng]# kubectl describe hpa web1

  Type     Reason                        Age                    From                       Message
  ----     ------                        ----                   ----                       -------
  Warning  FailedGetScale                3m30s (x3 over 142m)   horizontal-pod-autoscaler  deployments/scale.apps "ng-web" not found
  Warning  FailedGetResourceMetric       3m (x2 over 3m15s)     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  3m (x2 over 3m15s)     horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       2m15s (x3 over 2m45s)  horizontal-pod-autoscaler  missing request for cpu
  Warning  FailedComputeMetricsReplicas  2m15s (x3 over 2m45s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu

*****  所以在deployment-nginx.yaml里面一定要加入resources limits request 这几个资源限制参数.

5 查看HPA的资源信息：

## 添加资源限制后重新应用：
kubectl apply -f deployment-nginx.yaml

[root@k8s-master1 linux39_ng]# kubectl autoscale deployment web1 --max=3  --min=1 --cpu-percent=1 
horizontalpodautoscaler.autoscaling/web1 autoscaled

# --cpu-percent=1    CPU百分比，测试建议放低点

## 等待一会，可以看到相关的hpa信息（K8s上metrics服务收集所有pod资源的时间间隔大概在60s的时间）
[root@k8s-master1 web1]# kubectl get hpa -w
NAME     REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web1     Deployment/web1     0%/1%     1         3         1          48m

## 再次查看HPA：
[root@k8s-master1 linux39_ng]# kubectl describe  hpa  web1
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range

6 模拟业务流量增长，看看hpa自动伸缩的效果：

# 获取ng-web 的CluseterIP：
[root@k8s-master2 ng]# kubectl get service 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
web1        ClusterIP   172.16.1.108   <none>        80/TCP    16d

# 其他node模拟流量,随便写了个html页面用于测试:
[root@k8s-master2 ~]# while :;do curl 172.16.1.108 ;done 
<h1>this is NFS PV PVC  pod show</h1>
<img src='123.png' >
<h1>this is NFS PV PVC  pod show</h1>

一台不行就2个节点上量.....

# 过一会在看看HPA 信息，不会上来就自动扩容, 需要一定的时间全达到阈值范围，就开始自动扩容.

[root@k8s-master1 web1]# kubectl get hpa -w
web1     Deployment/web1     10%/1%    1         3         1          48m
web1     Deployment/web1     10%/1%    1         3         3          48m
web1     Deployment/web1     0%/1%     1         3         3          49m

# 停止Node 并发访问，在查看HPA是不是慢慢的进行了缩容：
[root@k8s-master1 web1]# kubectl get hpa -w
NAME     REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web1     Deployment/web1     10%/1%    1         3         1          48m
web1     Deployment/web1     0%/1%     1         3         3          54m
web1     Deployment/web1     0%/1%     1         3         1          54m

7 文件方式创建HPA:

vim hpa_web1.yaml
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: web1
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web1
  minReplicas: 1
  maxReplicas: 3
  targetCPUUtilizationPercentage: 50


## 执行文件：
kubectl apply -f hpa_web1.yaml

错误：

1. kubectl get hpa -w 不显示资源使用率：

问题 1：
  [root@k8s-master1 web1]# kubectl delete ingress web1
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (NotFound): ingresses.extensions "web1" not found
[root@k8s-master1 web1]# kubectl describe hpa web1
Name:                                                  web1
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 23 Nov 2021 16:13:41 +0800
Reference:                                             Deployment/web1
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 1%
Min replicas:                                          1
Max replicas:                                          3
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                         From                       Message
  ----     ------                        ----                        ----                       -------
  Warning  FailedComputeMetricsReplicas  10s (x12 over 2m58s)        horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
  Warning  FailedGetResourceMetric       <invalid> (x20 over 2m58s)  horizontal-pod-autoscaler  missing request for cpu

问题 2 ：
[root@k8s-master1 web1]# kubectl get hpa -w
NAME     REFERENCE           TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
web1     Deployment/web1     <unknown>/1%   1         3         0          12s
web1     Deployment/web1     <unknown>/1%   1         3         1          15s

*************************************************************************************************************************************
解决：
    以上的两个问题都是因为创建deployment 时的resources 这部分的限制超标或者资源限制的配置格式不对. 仔细检查deployment 清单里的资源限制书写格式部分.

正确的限制资源格式如下：
......
        # resources: {}                   # 这行很重要,命令行导出自带的, 1 删除 2 把资源限制写在里面. 我就直接注释，大家可以选择删除.
        resources:                        # 资源使用   
          limits:                         # (最高)限制单个POD 最多能使用1核(1000m 毫核)CPU以及2G内存
            cpu: "20m"
            memory: "50Mi"
          requests:                       # (最低)保证此pod 初始获取这么多资源
            cpu: "10m"
            memory: "20Mi"
......