报错如下
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.381558 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.581422 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.781432 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:18 node1 kubelet[1722]: E0619 17:15:18.981401 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.181612 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.381434 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.581538 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.781372 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:19 node1 kubelet[1722]: E0619 17:15:19.981466 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.182079 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
Jun 19 17:15:20 node1 kubelet[1722]: E0619 17:15:20.381529 1722 desired_state_of_world_populator.go:312] Error processing volume "manual165" for pod "gr8333e7-0_672f06d5992f4b4580ae04289e33dde4(311dd5e7-1ce9-484d-9862-c7e60eeba6e5)": error processing PVC 672f06d5992f4b4580ae04289e33dde4/manual165-gr8333e7-0: PVC is being deleted
处理办法
查找到该pod详情，将控制器删除即可
[root@master ~]# kubectl get po -A|grep gr8333e7
672f06d5992f4b4580ae04289e33dde4 gr8333e7-0
[root@master ~]# kubectl describe po gr8333e7-0 -n 672f06d5992f4b4580ae04289e33dde4
Name: gr8333e7-0
Namespace: 672f06d5992f4b4580ae04289e33dde4
Priority: 0
Node: node1/172.31.200.68
Start Time: Fri, 19 Jun 2020 14:54:34 +0800
Labels: controller-revision-hash=gr8333e7-595ff46986
creater_id=1592549674520418745
Annotations: rainbond.com/tolerate-unready-endpoints: true
Status: Running
IP: 10.244.3.196
IPs:
IP: 10.244.3.196
Controlled By: StatefulSet/gr8333e7
Containers:
f6719b6d0f2adace1d930dc5f48333e7:
Container ID: docker://2258ee0b766f2ce261563de4bda331f8bcb172ec474f0c78a9e0627eb6dbe708
Image: goodrain.me/f6719b6d0f2adace1d930dc5f48333e7:20200619145017
Image ID: docker-pullable://goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab@sha256:8e4eca55761ebadacc6503acced877fa69689389b27c56a15ba165810e563e31
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 19 Jun 2020 14:54:36 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 1280m
memory: 1Gi
Requests:
cpu: 240m
memory: 1Gi
Readiness: tcp-socket :3306 delay=4s timeout=5s period=3s #success=1 #failure=3
Environment:
LOGGER_DRIVER_NAME: streamlog
REVERSE_DEPEND_SERVICE: gr512123:28d93ce6688d13325dc7986169512123,gr58ee27:599b46254ee0690d3ee750b5ab58ee27,gr5efe93:9c69edab427540f0aecc9bd0bb5efe93
DB_HOST: 127.0.0.1
DB_PORT: 3306
TENANT_ID: 672f06d5992f4b4580ae04289e33dde4
SERVICE_ID: f6719b6d0f2adace1d930dc5f48333e7
MEMORY_SIZE: large
SERVICE_NAME: gr8333e7
SERVICE_POD_NUM: 1
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
Mounts:
/var/lib/mysql from manual165 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lctjg (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
manual165:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: manual165-gr8333e7-0
ReadOnly: false
default-token-lctjg:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lctjg
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
[root@master ~]# grctl service get gr8333e7 -t npwrtv4l
Namespace: 672f06d5992f4b4580ae04289e33dde4
ServiceID: f6719b6d0f2adace1d930dc5f48333e7
ReplicationType: statefulset
ReplicationID: gr8333e7
Status: running
------------Service------------
+---------------------+----------------+------------+
| Name | IP | Port |
+---------------------+----------------+------------+
| gr8333e7 | None | (TCP:3306) |
| service-392-3306 | 10.108.223.74 | (TCP:3306) |
| service-392-3306out | 10.105.125.132 | (TCP:3306) |
+---------------------+----------------+------------+
------------Ingress------------
+------+------+
| Name | Host |
+------+------+
+------+------+
-------------------Pod_1-----------------------
PodName: gr8333e7-0
PodStatus: Initialized : True Ready : True ContainersReady : True PodScheduled : True
PodIP: 10.244.3.196
PodHostIP: 172.31.200.68
PodHostName: node1
PodVolumePath:
PodStratTime: 2020-06-19T14:54:34+08:00
Containers:
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
| ID | Name | Image | State |
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
| 2258ee0b766f | f6719b6d0f2adace1d930dc5f48333e7 | goodrain.me/0b8f5af437254bb55b4d8907a0bbb3ab:20200424215058 | Running(2020-06-19T14:54:36+08:00) |
+--------------+----------------------------------+-------------------------------------------------------------+------------------------------------+
[root@master ~]# kubectl delete sts gr8333e7 -n 672f06d5992f4b4580ae04289e33dde4
statefulset.apps "gr8333e7" deleted
查看日志即正常