记一次k8s部署问题

问题描述

在k8s master节点按步骤创建成功后，执行从节点的加入操作，发现一直超时无法加入

问题日志

[root@node02 kubernetes-1.14]# kubeadm join 10.0.0.121:6443 --token oxqn4k.2olj80b59gndjmdj --discovery-token-unsafe-skip-ca-verification
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'
error execution phase kubelet-start: timed out waiting for the condition

问题解决

1.按网上查了并不是token过期的原因

2.反复操作过程中，发现master非常卡，后来执行命令都无法返回结果了

kubeadm token list
failed to list bootstrap tokens: Get https://10.0.0.121:6443/api/v1/namespaces/kube-system/secrets?fieldSelector=type%3Dbootstrap.kubernetes.io%2Ftoken: net/http: TLS handshake timeout

3.判断可能是内存不足

[root@node01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           972M        524M         61M         50M        386M         76M
Swap:            0B          0B          0B

• 只有1G内存

4.于是打开虚拟机设置，增加1G内存

[root@node01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        552M        595M         50M        848M        1.2G
Swap:            0B          0B          0B

发现master机流畅多了

5.重新执行节点加入，成功加入

kubeadm join 10.0.0.121:6443 --token befijo.vtrzxgw2ds7m5bak     --discovery-token-ca-cert-hash sha256:5ea9439afdd166995f92d9d42a6fcb73aa8c92e5af5570850a709273f361b5b0
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details

[root@node01 ~]# kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
node01   Ready    master   20h   v1.14.1
node02   Ready    <none>   30m   v1.14.1

问题总结

• 主节点由于内存不足无法正常服务导致从节点无法加入
• 又是一起简单的原因导致难查的BUG
• 由此可知k8s部署主节点至少需要分配2G内存