本文记录的是将 kubernetes 集群从 1.17.0 升级至最新版 1.20.2 的实际操作步骤，由于 1.17.0 无法直接升级到 1.20.2，需要进行2次过滤升级，1.17.0 -> 1.18.0 -> 1.19.0 -> 1.20.2。
先在 control plane node 上进行升级操作，通过下面的命令查看最新版 kubeadm 的版本号
apt update
apt-cache madison kubeadm
最新版是 1.20.2-00，用下面的命令安装最新版 kubeadm
apt-get update &&
apt-get install -y --allow-change-held-packages kubeadm=1.20.2-00
然后查看升级计划
kubeadm upgrade plan
提示无法直接升级1.20.2
[upgrade/config] FATAL: this version of kubeadm only supports deploying clusters with the control plane version >= 1.19.0. Current version: v1.17.0
于是尝试先升级到 1.19.0
apt-get update &&
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.19.0-00
查看升级计划1.19.0
[upgrade/config] FATAL: this version of kubeadm only supports deploying clusters with the control plane version >= 1.18.0. Current version: v1.17.0
只能先升级到1.18.0
apt-get update &&
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.18.0-00
查看升级计划
kubeadm upgrade plan
启动升级操作
kubeadm upgrade apply v1.18.0
集群成功升级至 1.18.0
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.18.0". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
分别在 control plane node 与 worker node 上 将 kubelet 和 kubectl 升级至 1.18.0（如果是生产环境，在升级之前先 uncordon 要升级的节点）
apt-get update &&
apt-get install -y --allow-change-held-packages kubelet=1.18.0-00 kubectl=1.18.0-00
systemctl daemon-reload && systemctl restart kubelet
继续，向 1.19.0 升级！
升级 kubeadm 至 1.19.0
apt-get install -y --allow-change-held-packages --allow-downgrades kubeadm=1.19.0-00
升级集群至 1.19.0
kubeadm upgrade plan
kubeadm upgrade apply v1.19.0
升级 kubelet 和 kubectl 至 1.19.0
apt-get install -y --allow-change-held-packages kubelet=1.19.0-00 kubectl=1.19.0-00
systemctl daemon-reload && systemctl restart kubelet
冲刺，向最新版升级！
升级 kubeadm 至 1.20.2
apt-get install -y --allow-change-held-packages kubeadm=1.20.2-00
升级集群至 1.20.2
kubeadm upgrade plan
kubeadm upgrade apply v1.20.2
各个 node 升级 kubelet、kubectl、kubeadm 至 1.20.2
apt-get install -y --allow-change-held-packages kubelet=1.20.2-00 kubectl=1.20.2-00 kubeadm=1.20.2-00
systemctl daemon-reload && systemctl restart kubelet
胜利，升级成功！
NAME STATUS ROLES AGE VERSION
k8s-master0 Ready control-plane,master 371d v1.20.2
kube-node1 Ready <none> 2d15h v1.20.2
kube-node2 Ready <none> 38h v1.20.2
kube-node3 Ready <none> 17h v1.20.2
升级过程中遇到的问题
我们在 coredns 配置中使用了 rewrite answer ，升级时出现下面的错误提示
[preflight] Some fatal errors occurred:
[ERROR CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Option "answer" in plugin "rewrite" is unsupported by this migration tool in 1.6.5. Option "answer"
通过 plan --ignore-preflight-errors 参数忽略了这个错误
kubeadm upgrade plan --ignore-preflight-errors=CoreDNSUnsupportedPlugins
所以升级后 coredns 的配置没有迁移过来，需要在升级前导出 coredns 的 ConfigMap
kubectl get cm coredns -n kube-system -o yaml > coredns_cm.yaml
然后在升级后替换 coredns 的配置
kubectl replace -n kube-system -f coredns_cm.yaml