zoukankan      html  css  js  c++  java
  • Kubernetes HPA AutoScale引起异常告警现象

    事件回顾


    告警事件,在这个时间段,已知的信息,无版本发布,及人为操作

     事件排查


    1. 查看Kubernetes事件信息,事件信息给出的引导信息非常少,只是一些简单的调度操作,并没有说明问题的本质,其中从事件信息获取到二个Kill的操作,但是是正常操作(type)
      <root@PROD-K8S-CP1 ~># kubectl get events -A
      NAMESPACE   LAST SEEN   TYPE      REASON              OBJECT                                              MESSAGE
      prod        57m         Normal    Killing             pod/prod-common-trust-service-6b6b85d679-d5tdn      Stopping container prod-common-trust-service
      prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Readiness probe failed: HTTP probe failed with statuscode: 503
      prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Liveness probe failed: HTTP probe failed with statuscode: 503
      prod        57m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-d5tdn      Readiness probe failed: Get http://172.21.4.49:9141/actuator/health: dial tcp 172.21.4.49:9141: connect: no route to host
      prod        <unknown>   Normal    Scheduled           pod/prod-common-trust-service-6b6b85d679-gnvtr      Successfully assigned prod/prod-common-trust-service-6b6b85d679-gnvtr to prod-be-k8s-wn12
      prod        55m         Normal    Pulling             pod/prod-common-trust-service-6b6b85d679-gnvtr      Pulling image "imagehub.qiangyun.com/common/prod-common-trust-service:develop-6"
      prod        55m         Normal    Pulled              pod/prod-common-trust-service-6b6b85d679-gnvtr      Successfully pulled image "imagehub.qiangyun.com/common/prod-common-trust-service:develop-6"
      prod        55m         Normal    Created             pod/prod-common-trust-service-6b6b85d679-gnvtr      Created container prod-common-trust-service
      prod        55m         Normal    Started             pod/prod-common-trust-service-6b6b85d679-gnvtr      Started container prod-common-trust-service
      prod        49m         Normal    Killing             pod/prod-common-trust-service-6b6b85d679-gnvtr      Stopping container prod-common-trust-service
      prod        49m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-gnvtr      Readiness probe failed: HTTP probe failed with statuscode: 503
      prod        49m         Warning   Unhealthy           pod/prod-common-trust-service-6b6b85d679-gnvtr      Liveness probe failed: Get http://172.21.4.47:9141/actuator/health: dial tcp 172.21.4.47:9141: connect: no route to host
      prod        57m         Normal    SuccessfulDelete    replicaset/prod-common-trust-service-6b6b85d679     Deleted pod: prod-common-trust-service-6b6b85d679-d5tdn
      prod        55m         Normal    SuccessfulCreate    replicaset/prod-common-trust-service-6b6b85d679     Created pod: prod-common-trust-service-6b6b85d679-gnvtr
      prod        49m         Normal    SuccessfulDelete    replicaset/prod-common-trust-service-6b6b85d679     Deleted pod: prod-common-trust-service-6b6b85d679-gnvtr
      prod        55m         Normal    SuccessfulRescale   horizontalpodautoscaler/prod-common-trust-service   New size: 6; reason: cpu resource utilization (percentage of request) above target
      prod        55m         Normal    ScalingReplicaSet   deployment/prod-common-trust-service                Scaled up replica set prod-common-trust-service-6b6b85d679 to 6
      prod        49m         Normal    SuccessfulRescale   horizontalpodautoscaler/prod-common-trust-service   New size: 5; reason: All metrics below target
      prod        49m         Normal    ScalingReplicaSet   deployment/prod-common-trust-service                Scaled down replica set prod-common-trust-service-6b6b85d679 to 5
    2.  查看阿里云Kubernetes事件中心
    3. 查看Pod历史副本数

       
  • 相关阅读:
    【漏洞分析】dedecms有前提前台任意用户密码修改
    关于t00ls的挂机脚本
    关于pocsuite的使用
    [代码审计]青云客Cms前台有条件注入至getshell,后台xss至getshell、至弹你一脸计算器
    警惕phpstudy等开发神器使用默认配置可能带来的危险
    [代码审计]DM企业建站系统v201710 sql注入漏洞分析 | 新版v201712依旧存在sql注入
    [代码审计]XiaoCms(后台任意文件上传至getshell,任意目录删除,会话固定漏洞)
    对长短按设置不同反应 安卓
    如何数冲突域(collision domains)个数
    Computer Network Homework2’s hard question
  • 原文地址:https://www.cnblogs.com/apink/p/15543447.html
Copyright © 2011-2022 走看看