kubeadm安装的k8s集群证书过期处理

# kubeadm安装的k8s证书过期处理

## 一、背景说明

1. kubeadm默认证书为一年，一年过期后，会导致api service不可用，使用过程中会出现：`x509: certificate has expired or is not yet valid`
2. kubelet证书分为`server`和`client`两种， k8s1.9开始默认启用了client证书的自动轮换，但server证书自动轮换需要用户配置开启

## 二、开启server证书自动轮换

> 此方案适用于证书还未过期

### 1. 增加kubelet参数

``` bash
#在/etc/sysconfig/kubelet增加，若多master，都需要配置：
KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true --rotate-server-certificates=true
```

### 2. 配置kube-controller-manager

``` yaml
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --experimental-cluster-signing-duration=87600h0m0s #增加证书颁发时间参数
- --feature-gates=RotateKubeletServerCertificate=true #开启server证书签发
- --allocate-node-cidrs=true
```

### 3. 创建rbac对象，允许节点轮换kubelet server证书

``` bash
cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests/selfnodeserver
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubeadm:node-autoapprove-certificate-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
EOF
kubectl apply -f ca-update.yaml
```

### 4. 重启kubelet

``` bash
systemctl restart kubelet
#查看csr,状态会由Pending to Approved
kubectl get csr
```

### 5. 多master其他节点一直处于Pending

``` bash
#出于安全原因,处于pending状态的master节点需要手动审批
kubectl certificate approve <name>
```

## 三、替换server证书

> 此方案适用于证书已过期，处理完成后，再执行`开启server证书自动轮换`

### 1. 报错信息

``` bash
kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid
```

### 2. 证书备份

``` bash
cp -Ra /etc/kubernetes /opt/kubernetes-backup-time
```

### 3. 删除过期证书

``` bash
#apiserver证书
rm -f /etc/kubernetes/pki/apiserver*
#front-proxy-client证书
rm -f /etc/kubernetes/pki/front-proxy-client.*
#etcd证书，若etcd是部署在集群外自签证书，不执行以下命令
rm -rf /etc/kubernetes/pki/etcd/healthcheck-client.*
rm -rf /etc/kubernetes/pki/etcd/server.*
rm -rf /etc/kubernetes/pki/etcd/peer.*
```

### 4. 重新生成证书

``` bash
#下载对应版本的kubeadm
wget https://dl.k8s.io/release/v1.10.1/bin/linux/amd64/kubeadm
chmod a+x kubeadm
#生成证书,若使用HA需要配置成vip地址
./kubeadm alpha phase certs all --apiserver-advertise-address <IP address of your master server>
```

### 5. 重新生成配置文件

``` bash
#备份配置文件
mv /etc/kubernetes/*.conf /tmp
#生成配置文件
./kubeadm alpha phase kubeconfig all --apiserver-advertise-address <IP address of your master server>
```

### 6. 重启kubelet

``` bash
systemctl restart kubelet
```

### 7. 验证集群

``` bash
#查看证书过期时间
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
#集群节点状态
kubectl get no
```

## 四、参考

* [kubelet-tls-bootstrapping](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping)
* [certificate-rotation](https://kubernetes.io/docs/tasks/tls/certificate-rotation/)