偶然查看服务时发现有个单副本的 pod 一直处于Terminating状态，已达 8 天之久，且手动删除删除不掉
[appuser@lyj 2019-09-20 14:35:36 ~]$ kubectl get pods -n sit |grep loan-after-query-service-sit
loan-after-query-service-sit-868758d5c8-7sfcr 1/1 Running 0 14m
loan-after-query-service-sit-868758d5c8-scmfx 1/1 Terminating 0 8d
使用命令kubectl describe pod loan-after-query-service-sit-868758d5c8-scmfx -n NAMESPACE 查看该 pod的描述信息, 得到如下
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Killing 13s (x21 over 36m) kubelet, cn-beijing.i-2ze0dg85eb6vxi1t0vcr Killing container with id docker://loan-after-query-service:Need to kill Pod
这个问题是Kubernetes偶发的BUG，使用以下命令强制删除：
kubectl delete pod xxxxxx --grace-period=0 --force
执行命令后返回一条警告信息---> 警告:立即删除并不等待正在运行的资源已被终止的确认。资源可以无限期地继续在集群上运行。
[appuser@lyj 2019-09-20 14:24:34 ~]$ kubectl delete pods loan-after-query-service-sit-868758d5c8-scmfx -n NAMESPACE --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "loan-after-query-service-sit-868758d5c8-scmfx" deleted
去到容器所在 Node机器上通过 docker ps 仍看得到这个已经运行 8 天的容器
[root@iz2ze0dg85eb6vxi1t0vcrz ~]# docker ps -a|grep loan-after-query-service-sit
007eb7071429 registry-vpc.cn-beijing.aliyuncs.com/9fbank/loan-after-query-service "/bin/sh -c 'sh /u..." 7 minutes ago Up 7 minutes k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-7sfcr_sit_3eb28c46-db6f-11e9-aee8-00163e082a66_0
7b36d58a26d5 registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 7 minutes ago Up 7 minutes k8s_POD_loan-after-query-service-sit-868758d5c8-7sfcr_sit_3eb28c46-db6f-11e9-aee8-00163e082a66_0
21ad6e62a0f2 registry-vpc.cn-beijing.aliyuncs.com/9fbank/loan-after-query-service "/bin/sh -c 'sh /u..." 8 days ago Up 8 days k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0
7691a8019419 registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 8 days ago Exited (0) 43 minutes ago k8s_POD_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0
手动删除该容器
[root@iz2ze0dg85eb6vxi1t0vcrz ~]# docker rm -f k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0
k8s_loan-after-query-service_loan-after-query-service-sit-868758d5c8-scmfx_sit_efeb3399-d462-11e9-aee8-00163e082a66_0