kubernetes 集群搭建 -- 二进制方式

主机信息

作为练习，只准备一台master，一台node，操作系统为centos7.9，多台node的添加只是重复工作。

参照上一篇，和kubeadm方式安装时的初始化一样，进行关闭防火墙、关闭 selinux、关闭swap、修改主机名、在 master 添加 hosts、将桥接的IPv4流量传递到 iptables 的链以及时间同步等初始化工作。

部署Etcd 集群

Etcd 是一个分布式键值存储系统，Kubernetes 使用 Etcd 进行数据存储，所以先准备一个 Etcd 数据库，为解决 Etcd 单点故障，应采用集群方式部署，这里只使用 2 台组建集群。

3台组件的集群，可容忍 1 台机器故障；5 台组建的集群，可容忍 2 台机器故障。

注：为了节省机器，这里与 K8s 节点机器复用；当然也可以独立于 k8s 集群之外部署，只要apiserver 能连接到就行。

1.安装 cfssl 证书生成工具

cfssl 是一个开源的证书管理工具，使用 json 文件生成证书，相比 openssl 更方便使用。

找任意一台服务器操作，这里用 Master 节点，进行安装：

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

就可以使用这三个命令了：

[root@binary-master bin]# pwd
/usr/local/bin
[root@binary-master bin]# ll
total 18808
-rwxr-xr-x 1 root root 10376657 Mar 30  2016 cfssl
-rwxr-xr-x 1 root root  6595195 Mar 30  2016 cfssl-certinfo
-rwxr-xr-x 1 root root  2277873 Mar 30  2016 cfssljson

2.生成 Etcd 证书

（1）自签证书颁发机构（CA）

创建工作目录：

mkdir -p ~/TLS/{etcd,k8s}
cd TLS/etcd

自签CA，两个json文件：

cat > ca-config.json<< EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

cat > ca-csr.json<< EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF

生成证书（初始化）：cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

[root@binary-master etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/03/22 17:36:34 [INFO] generating a new CA key and certificate from CSR
2021/03/22 17:36:34 [INFO] generate received request
2021/03/22 17:36:34 [INFO] received CSR
2021/03/22 17:36:34 [INFO] generating key: rsa-2048
2021/03/22 17:36:35 [INFO] encoded CSR
2021/03/22 17:36:35 [INFO] signed certificate with serial number 558179141153695072425640815117027228526462947382
[root@binary-master etcd]# ls *pem
ca-key.pem  ca.pem

（2）使用自签 CA 签发 Etcd HTTPS 证书

创建证书申请文件：

cat > server-csr.json<< EOF
{
"CN": "etcd",
"hosts": [
"172.31.93.210",
"172.31.93.211"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF

注：上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP，一个都不能少！为了方便后期扩容可以多写几个预留的 IP。

生成证书：cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

[root@binary-master etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/03/22 17:50:56 [INFO] generate received request
2021/03/22 17:50:56 [INFO] received CSR
2021/03/22 17:50:56 [INFO] generating key: rsa-2048
2021/03/22 17:50:56 [INFO] encoded CSR
2021/03/22 17:50:56 [INFO] signed certificate with serial number 487201065380785180427832079165743752767773807902
2021/03/22 17:50:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@binary-master etcd]# ls *.pem
ca-key.pem  ca.pem  server-key.pem  server.pem

其中server-key.pem和server.pem就是最后需要的etcd证书。

3.部署etcd集群

Etcd下载地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

以下在master上操作，为简化操作，待会将master 生成的所有文件拷贝到node节点。

（1）创建工作目录并解压二进制包，

mkdir -p /opt/etcd/{bin,cfg,ssl}
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

（2）创建 etcd 配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.93.210:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.93.210:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.93.210:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.93.210:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://172.31.93.210:2380,etcd-2=https://172.31.93.211:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

说明：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群 Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing表示加入已有集群

（3）systemd 管理 etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd 
--cert-file=/opt/etcd/ssl/server.pem 
--key-file=/opt/etcd/ssl/server-key.pem 
--peer-cert-file=/opt/etcd/ssl/server.pem 
--peer-key-file=/opt/etcd/ssl/server-key.pem 
--trusted-ca-file=/opt/etcd/ssl/ca.pem 
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem 
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

（4）拷贝已经生成好的证书

把刚才生成的证书拷贝到etcd.service文件中设置的路径：

cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

即四个.pem文件：

[root@binary-master ssl]# pwd
/opt/etcd/ssl
[root@binary-master ssl]# ll
total 16
-rw------- 1 root root 1679 Mar 22 22:03 ca-key.pem
-rw-r--r-- 1 root root 1265 Mar 22 22:03 ca.pem
-rw------- 1 root root 1679 Mar 22 22:03 server-key.pem
-rw-r--r-- 1 root root 1330 Mar 22 22:03 server.pem

 到此，Master上的etcd部署已完成。

（5）将上面节点master所有生成的文件拷贝到节点node

scp -r /opt/etcd/ root@172.31.93.211:/opt/
scp /usr/lib/systemd/system/etcd.service root@172.31.93.211:/usr/lib/systemd/system/

（6）在node节点上修改 etcd.conf 配置文件中的节点名称和当前服务器 IP

（7）在master和node节点上都启动并设置开机启动

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

（8）使用etcdctl查看集群状态

/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.31.93.210:2379,https://172.31.93.211:2379" endpoint health

[root@binary-master ssl]# /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://172.31.93.210:2379,https://172.31.93.211:2379" endpoint health
https://172.31.93.210:2379 is healthy: successfully committed proposal: took = 11.468435ms
https://172.31.93.211:2379 is healthy: successfully committed proposal: took = 12.058593ms

如果输出上面信息，就说明etcd集群部署成功。如果有问题第一步先看日志：/var/log/message 或 journalctl -u etcd

所有节点二进制安装docker

之前使用kubeadm安装k8s集群时，是用yum安装的docker，这次用二进制安装k8s集群，所以也用二进制安装docker。当然用yum也可以，只是不同的安装方式而已。如果在不联网的情况下，只能上传二进制包来安装。

下载地址：https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

1.解压二进制包

tar zxvf docker-19.03.9.tgz
mv docker/* /usr/bin

2.systemd 管理 docker

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

3.创建配置文件，配置阿里云镜像加速器

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}
EOF

4.启动并设置开机启动

systemctl daemon-reload
systemctl start docker
systemctl enable docker

5.把docker-19.03.9.tgz包scp到其他节点上，按照上述步骤都安装一遍

部署master节点

Master节点包括三个组件：apiserver、controller-manager、scheduler

部署步骤都一致：

• 创建配置文件
• 由systemd管理组件服务
• 启动并设置开机启动

其中apiserver还需要生成并配置上证书（和etcd一样），并启用 TLS Bootstrapping 机制。

1.生成 kube-apiserver 证书

这个过程和上面生成etcd证书的过程一样。在TLS目录下，之前创建了etcd和k8s两个文件夹，apiserver证书生成的操作就在k8s文件夹中执行：

[root@binary-master TLS]# pwd
/root/TLS
[root@binary-master TLS]# ls
etcd  k8s
[root@binary-master TLS]# cd k8s/

（1）自签证书颁发机构（CA）

cat > ca-config.json<< EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

cat > ca-csr.json<< EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF

（2）生成证书（初始化）

cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

[root@binary-master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/03/23 16:48:08 [INFO] generating a new CA key and certificate from CSR
2021/03/23 16:48:08 [INFO] generate received request
2021/03/23 16:48:08 [INFO] received CSR
2021/03/23 16:48:08 [INFO] generating key: rsa-2048
2021/03/23 16:48:08 [INFO] encoded CSR
2021/03/23 16:48:08 [INFO] signed certificate with serial number 674155349167693061708935668878485858330936601612
[root@binary-master k8s]# ls *pem
ca-key.pem  ca.pem

（3）使用自签 CA 签发 kube-apiserver HTTPS 证书

创建证书申请文件：

cat > server-csr.json<< EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"172.31.93.210",
"172.31.93.211",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF

注意，其中一定要hosts中添加集群的所有可信任IP。

（4）生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

[root@binary-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2021/03/23 16:54:41 [INFO] generate received request
2021/03/23 16:54:41 [INFO] received CSR
2021/03/23 16:54:41 [INFO] generating key: rsa-2048
2021/03/23 16:54:41 [INFO] encoded CSR
2021/03/23 16:54:41 [INFO] signed certificate with serial number 321029219572130619553803174282705384482149958458
2021/03/23 16:54:41 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements"). 
[root@binary-master k8s]# ls server*pem
server-key.pem  server.pem

其中server-key.pem和server.pem就是最后需要的apiserver自签证书。

2.从GitHub下载 kubernetes-server-linux-amd64.tar.gz 包

下载地址：https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG

选择想要下载的版本，进入后里面会有很多包，注意要下载服务端的版本包（Server Binaries），其中已经包含了Master和Node二进制文件。

3.解压二进制包，并创建相应工作目录，复制需要的命令工具

tar zxvf kubernetes-server-linux-amd64.tar.gz

目录结构：

[root@binary-master kubernetes]# ll
total 32792
drwxr-xr-x 2 root root        6 Aug 26  2020 addons
-rw-r--r-- 1 root root 33576969 Aug 26  2020 kubernetes-src.tar.gz
drwxr-xr-x 3 root root       49 Aug 26  2020 LICENSES
drwxr-xr-x 3 root root       17 Aug 26  2020 server

然后执行：

#创建工作目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
#复制相关命令
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

4.部署 kube-apiserver

（1）创建配置文件kube-apiserver.conf

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://172.31.93.210:2379,https://172.31.93.211:2379 \
--bind-address=172.31.93.210 \
--secure-port=6443 \
--advertise-address=172.31.93.210 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

说明：
上面两个  第一个是转义符，第二个是换行符，使用转义符是为了使用 EOF 保留换行符。
–logtostderr：启用日志
—v：日志等级
–log-dir：日志目录
–etcd-servers：etcd 集群地址
–bind-address：监听地址
–secure-port：https 安全端口
–advertise-address：集群通告地址
–allow-privileged：启用授权
–service-cluster-ip-range：Service 虚拟 IP 地址段
–enable-admission-plugins：准入控制模块
–authorization-mode：认证授权，启用 RBAC 授权和节点自管理
–enable-bootstrap-token-auth：启用 TLS bootstrap 机制
–token-auth-file：bootstrap token 文件
–service-node-port-range：Service nodeport 类型默认分配端口范围
–kubelet-client-xxx：apiserver 访问 kubelet 客户端证书
–tls-xxx-file：apiserver https 证书
–etcd-xxxfile：连接 Etcd 集群证书
–audit-log-xxx：审计日志

一般只需修改其中的几个ip地址即可。

（2）启用 TLS Bootstrapping 机制

TLS Bootstraping：Master apiserver 启用 TLS 认证后，Node 节点 kubelet 和 kube-proxy 要与 kube-apiserver 进行通信，必须使用 CA 签发的有效证书才可以，当 Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署。所以强烈建议在 Node 上使用这种方式，目前主要用于 kubelet，kube-proxy 还是由我们统一颁发一个证书。

即上述配置文件中的：

--enable-bootstrap-token-auth=true \                  表示开启
--token-auth-file=/opt/kubernetes/cfg/token.csv \    指定bootstrap token文件

所以创建上述配置文件中 token 文件：

cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-
bootstrapper"
EOF

文件格式为：token，用户名，UID，用户组

（3）拷贝刚才生成的apiserver证书到配置文件中指定的路径：

cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

[root@binary-master ssl]# pwd
/opt/kubernetes/ssl
[root@binary-master ssl]# ll
total 16
-rw------- 1 root root 1675 Mar 24 16:02 ca-key.pem
-rw-r--r-- 1 root root 1359 Mar 24 16:02 ca.pem
-rw------- 1 root root 1675 Mar 24 16:02 server-key.pem
-rw-r--r-- 1 root root 1659 Mar 24 16:02 server.pem

（4）systemd 管理 apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

（5）启动并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

（6）授权 kubelet-bootstrap 用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap 
--clusterrole=system:node-bootstrapper 
--user=kubelet-bootstrap


[root@binary-master ssl]# kubectl create clusterrolebinding kubelet-bootstrap 
> --clusterrole=system:node-bootstrapper 
> --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

5.  部署 kube-controller-manager

（1）创建配置文件

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF

说明：
–master：通过本地非安全本地端口 8080 连接 apiserver。
–leader-elect：当该组件启动多个时，自动选举（HA）
–cluster-signing-cert-file/–cluster-signing-key-file：自动为 kubelet 颁发证书的CA，与 apiserver 保持一致

这里的配置基本不用改，都是内部通信。

（2）systemd 管理 controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

（3）启动并设置开机启动

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

6.部署 kube-scheduler

（1）创建配置文件

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false 
--v=2 
--log-dir=/opt/kubernetes/logs 
--leader-elect 
--master=127.0.0.1:8080 
--bind-address=127.0.0.1"
EOF

说明：
–master：通过本地非安全本地端口 8080 连接 apiserver。
–leader-elect：当该组件启动多个时，自动选举（HA）

（2）systemd 管理 scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

（3）启动并设置开机启动

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

7.查看集群状态

此时3个组件都已经启动成功，通过 kubectl 工具查看当前集群组件状态：

[root@binary-master cfg]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   

如上输出说明 Master 节点各组件运行正常，master节点部署完成。

部署Worker Node节点

1.创建工作目录并拷贝二进制文件

创建工作目录：

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

从 master 节点scp拷贝下载安装包里的二进制命令文件：

[root@binary-master bin]# pwd
/root/kubernetes/server/bin
[root@binary-master bin]# ls
apiextensions-apiserver  kube-apiserver             kube-controller-manager             kubectl     kube-proxy.docker_tag  kube-scheduler.docker_tag
kubeadm                  kube-apiserver.docker_tag  kube-controller-manager.docker_tag  kubelet     kube-proxy.tar         kube-scheduler.tar
kube-aggregator          kube-apiserver.tar         kube-controller-manager.tar         kube-proxy  kube-scheduler         mounter
[root@binary-master bin]# scp  kubelet kube-proxy root@172.31.93.211:/opt/kubernetes/bin

worker node上就有kubelet和kube-proxy命令了：

[root@binary-node bin]# pwd
/opt/kubernetes/bin
[root@binary-node bin]# ll
total 145296
-rwxr-xr-x 1 root root 109974472 Mar 30 14:41 kubelet
-rwxr-xr-x 1 root root  38805504 Mar 30 14:41 kube-proxy
[root@binary-node bin]# 

2.部署 kubelet

（1）创建配置文件

cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=binary-node \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

说明：
–hostname-override：显示名称，集群中唯一
–network-plugin：启用 CNI
–kubeconfig：空路径，会自动生成，后面用于连接 apiserver
–bootstrap-kubeconfig：首次启动向 apiserver 申请证书
–config：配置参数文件
–cert-dir：kubelet 证书生成目录
–pod-infra-container-image：管理 Pod 网络容器的镜像

一般只要改一下--hostname-override即可。

（2）配置参数文件

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

（3）在master上生成bootstrap.kubeconfig文件，然后复制到node节点

需要用到kubectl命令，所以在master上操作：

# apiserver IP:PORT
KUBE_APISERVER="https://172.31.93.210:6443"
# 与 token.csv 里保持一致
TOKEN="c47ffb939f5ca36231d9e3121a252940"
# 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes 
--certificate-authority=/opt/kubernetes/ssl/ca.pem 
--embed-certs=true 
--server=${KUBE_APISERVER} 
--kubeconfig=bootstrap.kubeconfig
kubectl config set-credentials "kubelet-bootstrap" 
--token=${TOKEN} 
--kubeconfig=bootstrap.kubeconfig
kubectl config set-context default 
--cluster=kubernetes 
--user="kubelet-bootstrap" 
--kubeconfig=bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

一步步执行，执行后会生成一个bootstrap.kubeconfig文件：

[root@binary-master ~]# KUBE_APISERVER="https://172.31.93.210:6443"
[root@binary-master ~]# TOKEN="c47ffb939f5ca36231d9e3121a252940"
[root@binary-master ~]# kubectl config set-cluster kubernetes 
> --certificate-authority=/opt/kubernetes/ssl/ca.pem 
> --embed-certs=true 
> --server=${KUBE_APISERVER} 
> --kubeconfig=bootstrap.kubeconfig
Cluster "kubernetes" set.
[root@binary-master ~]# kubectl config set-credentials "kubelet-bootstrap" 
> --token=${TOKEN} 
> --kubeconfig=bootstrap.kubeconfig
User "kubelet-bootstrap" set.
[root@binary-master ~]# kubectl config set-context default 
> --cluster=kubernetes 
> --user="kubelet-bootstrap" 
> --kubeconfig=bootstrap.kubeconfig
Context "default" created.
[root@binary-master ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
Switched to context "default".
[root@binary-master ~]# cat bootstrap.kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVZGhZcjMwN2hSRVJQc3lJbGZYMXBPTlFiN0F3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEl4TURNeU16QTRORE13TUZvWERUSTJNRE15TWpBNE5ETXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd2RvYzcyOWNkd2dEODBGUzdZb1EKRjlScEdHb1BqMEJaTzdpL3dNaGRkRi9XVXZSem81ZHF0N1pzM25VS2VPU3dGUzg5SlNMOWZndVhuQVZ5RXZNcQpwQzBxMk8zZlAvbG5ua203bmlzeDJXYStHYVZuYUphRkRQMmkrbG1CWnhVeXVibTlHMFdydXJ3aEcydGxuVHMxClpwaG9JVWJoMngxRHJub2VldFI3Rk11UXpDZzJrUmhQOTFFY0xITVR6YUhwVnNjYnk1ZTh3NXZCbjU4YWRselEKZThKVkFSemFHUVZMREZ0OUpZQlNoaUYwSmF6RHJCMmlBdzE5VklFY3MrSUdjdjlFWHVMTjdkKytGbWwyZWJoego1WmNoWEIxZHhvUUhadFZvcDRTVzUwQ005YzRuQkdCbFVJVTN4VTBvVTlnVkNSQnI2MDlGTkZzUmJ1RjZMc2VZCmVRSURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVK3BKL0dPczRrL3hub2N2L3FGajZyaW5Uc244d0h3WURWUjBqQkJnd0ZvQVUrcEovR09zNAprL3hub2N2L3FGajZyaW5Uc244d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFCTW5LV1lUeXQxQmJ1Q0h0SDRFCjNhVDB4S0RNNUZKcVR6bU1vV1ZWWExtZjlTV0V5UU13QlErK0dvOUo3NlBqOVYwV0tSSk0rNFdOam9WWEViZnMKaUdDSlhubE5kZk5aSFplSzN6U1dhNFl2M1kzTmlaUVpxaGtHUHFTdXJuempjd0FKVlU4dzlZUWxXaTBFaFdjVApXanBBSitWRVlWWjh6aWF2RWZ3OXhjaTZ0UGhSYlpxazU2MlBBUkNvWUx5RC9DQmpnSFljejQ3YVcvSWNINHNECmFJODBuUWpzbldOSEFvR0JBQmJ2TnJRNzRSanF0M3pQcnlRYWhxVGp6OUczVmRSS2REYnJKVjN2VG8zN3dZQ3AKRkpVTTgvdlc0dGhxUTBwd3dTRXdHWUNOYXVaRnczWWg5SzA5SDg3T2NTUWxlTEdvYWc4YzdqelAyUk0xcFZyNQo0c2M9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://172.31.93.210:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubelet-bootstrap
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: c47ffb939f5ca36231d9e3121a252940

把这个文件拷贝到node节点的配置文件路径：

scp bootstrap.kubeconfig root@172.31.93.211:/opt/kubernetes/cfg

（4）systemd 管理 kubelet

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

（5）启动并设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

3.在master的apiserver上批准 kubelet 证书申请并加入集群

由于之前在apiserver上设置了启用 TLS Bootstrapping 机制，kubelet 启动时会向 kube-apiserver 发送 TLS bootstrapping 请求。

在master上使用命令kubectl get csr可以看到还未认证的节点：

[root@binary-master k8s]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-UdCmaoVbWhqqdxWo7gkP3pH5vzoqliQPtvvCzJ54DBM   7s    kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

然后在master上批准证书：kubectl certificate approve  'nodename'

[root@binary-master k8s]# kubectl certificate approve node-csr-UdCmaoVbWhqqdxWo7gkP3pH5vzoqliQPtvvCzJ54DBM
certificatesigningrequest.certificates.k8s.io/node-csr-UdCmaoVbWhqqdxWo7gkP3pH5vzoqliQPtvvCzJ54DBM approved

此时再使用命令kubectl get csr可以看到状态已经变为approved：

[root@binary-master k8s]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-UdCmaoVbWhqqdxWo7gkP3pH5vzoqliQPtvvCzJ54DBM   10m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

查看节点：

[root@binary-master k8s]# kubectl get node
NAME          STATUS     ROLES    AGE   VERSION
binary-node   NotReady   <none>   13m   v1.19.0

由于网络插件还没有部署，节点会没有准备就绪，所以是 NotReady。

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ps：这里遇到一个问题，kubectl get csr 显示No Resources Found，即apiserver没有收到kubelet的认证请求

[root@binary-master k8s]# kubectl get csr
No resources found

出现这个问题，一般检查：kubelet 使用的 bootstrap.kubeconfig 文件中User 是否是 kubelet-boostrap，token是否是kube-apiserver 使用的 token.csv 文件中的token

检查后发现都是正确的，再查看kubelet的日志，发现有错误：无法创建证书签名请。

F0330 17:59:28.096982    2887 server.go:265] failed to run Kubelet: cannot create certificate signing request: Post "https://172.31.93.210:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": x509: certificate is valid for 10.0.0.1, 127.0.0.1, 192.168.31.71, 192.168.31.72, 192.168.31.73, 192.168.31.74, 192.168.31.81, 192.168.31.82, 192.168.31.88, not 172.31.93.210
goroutine 1 [running]:

原来是在自签CA签发apiserver证书时，创建的证书申请文件server-csr.json中未添加集群的所有可信任IP。

修改这个文件，重新生成证书，并把新证书放到apiserver的证书目录，最后重启apiserver，问题解决。

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 4.部署 kube-proxy

（1）创建配置文件

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

（2）配置参数文件kube-proxy-config.yml

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
 kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: binary-node
clusterCIDR: 10.0.0.0/24
EOF

一般只要改一下--hostname-override即可。

（3）生成 kube-proxy.kubeconfig 文件

先在master节点生成 kube-proxy 证书，因为生成 .kubeconfig 文件需要用到证书：

# 切换工作目录
[root@binary-master k8s]# cd ~/TLS/k8s

# 创建证书请求文件
[root@binary-master k8s]# cat > kube-proxy-csr.json<< EOF
> {
> "CN": "system:kube-proxy",
> "hosts": [],
> "key": {
> "algo": "rsa",
> "size": 2048
> },
> "names": [
> {
> "C": "CN",
> "L": "BeiJing",
> "ST": "BeiJing",
> "O": "k8s",
> "OU": "System"
> }
> ]
> }
> EOF

# 生成证书
[root@binary-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2021/03/31 15:04:11 [INFO] generate received request
2021/03/31 15:04:11 [INFO] received CSR
2021/03/31 15:04:11 [INFO] generating key: rsa-2048
2021/03/31 15:04:11 [INFO] encoded CSR
2021/03/31 15:04:11 [INFO] signed certificate with serial number 223055766142513681177047308544368105041301659941
2021/03/31 15:04:11 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

#查看证书
[root@binary-master k8s]# ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

在master上生成kube-proxy.kubeconfig文件，然后复制到node节点

需要用到kubectl命令，所以也在master上操作：

# apiserver IP:PORT
KUBE_APISERVER="https://172.31.93.210:6443"
#生成kube-proxy.kubeconfig文件
kubectl config set-cluster kubernetes 
--certificate-authority=/opt/kubernetes/ssl/ca.pem 
--embed-certs=true 
--server=${KUBE_APISERVER} 
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy 
--client-certificate=/root/TLS/k8s/kube-proxy.pem 
--client-key=/root/TLS/k8s/kube-proxy-key.pem 
--embed-certs=true 
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default 
--cluster=kubernetes 
--user=kube-proxy 
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

一步步执行，执行后会生成一个 kube-proxy.kubeconfig文件：

[root@binary-master ~]# KUBE_APISERVER="https://172.31.93.210:6443"
[root@binary-master ~]# kubectl config set-cluster kubernetes 
> --certificate-authority=/opt/kubernetes/ssl/ca.pem 
> --embed-certs=true 
> --server=${KUBE_APISERVER} 
> --kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.
[root@binary-master ~]# kubectl config set-credentials kube-proxy --client-certificate=/root/TLS/k8s/kube-proxy.pem --client-key=/root/TLS/k8s/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
[root@binary-master ~]# kubectl config set-context default 
> --cluster=kubernetes 
> --user=kube-proxy 
> --kubeconfig=kube-proxy.kubeconfig
Context "default" created.
[root@binary-master ~]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".

把这个文件拷贝到node节点的配置文件路径：

scp kube-proxy.kubeconfig root@172.31.93.211:/opt/kubernetes/cfg

（4） systemd 管理 kube-proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

（5）启动并设置开机启动

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ps:这里遇到一个问题，kube-proxy无法启动，查看错误日志：unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

[root@binary-node logs]# cat kube-proxy.binary-node.root.log.FATAL.20210402-181634.77665
Log file created at: 2021/04/02 18:16:34
Running on machine: binary-node
Binary: Built with gc go1.15 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0402 18:16:34.526797   77665 server.go:495] unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

是因为kube-proxy-config.yml这个配置文件中，kubeconfig:/opt/kubernetes/cfg/kube-proxy.kubeconfig，这一行最前面应该有一个空格：

修改后重启即可。

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

5.部署 CNI 网络

（1）在node节点准备好 CNI 二进制文件

下载地址：https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

解压二进制包并移动到默认工作目录：

mkdir /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

 二进制文件都在 /opt/cni/bin/：

[root@binary-node ~]# ll
total 95336
-rw-------. 1 root root     1260 Mar 19 01:44 anaconda-ks.cfg
-rw-r--r--  1 root root 36878412 Mar 31 18:20 cni-plugins-linux-amd64-v0.8.6.tgz
drwxrwxr-x  2 1000 1000        6 Mar 23 14:35 docker
-rw-r--r--  1 root root 60730088 Mar 23 14:35 docker-19.03.9.tgz
-rw-r--r--  1 root root     4821 Mar 31 17:43 kube-flannel.yml
[root@binary-node ~]# tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin
./
./flannel
./ptp
./host-local
./firewall
./portmap
./tuning
./vlan
./host-device
./bandwidth
./sbr
./static
./dhcp
./ipvlan
./macvlan
./loopback
./bridge
[root@binary-node ~]# ls /opt/cni/bin/
bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan

（2）在master节点使用 kubectl apply 部署CNI网络，这一步和上一篇kubeadm方式中的操作一样

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

如下：

[root@binary-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
--2021-04-01 10:40:48--  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4821 (4.7K) [text/plain]
Saving to: ‘kube-flannel.yml’

100%[===================================================================================================================================================>] 4,821       --.-K/s   in 0s      

2021-04-01 10:40:48 (24.9 MB/s) - ‘kube-flannel.yml’ saved [4821/4821]

[root@binary-master ~]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

部署好网络插件，Node 就准备就绪：

[root@binary-master ~]# kubectl get pods -n kube-system
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-8zjwr   1/1     Running   6          9m33s
[root@binary-master ~]# kubectl get node
NAME          STATUS   ROLES    AGE   VERSION
binary-node   Ready    <none>   40h   v1.19.0

6.若要新增work node

假设需要新增work节点：172.31.93.212

（1）拷贝已部署好的 Node 相关文件到新节点

从已部署好的work节点（172.31.93.211）上将涉及的文件拷贝到新work节点（172.31.93.212）：

#kubernetes工作目录
scp -r /opt/kubernetes root@172.31.93.212:/opt/
#服务注册
scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@172.31.93.212:/usr/lib/systemd/system
#cni网络插件
scp -r /opt/cni/ root@172.31.93.212:/opt/
#证书
scp /opt/kubernetes/ssl/ca.pem root@172.31.93.212:/opt/kubernetes/ssl

（2）拷贝完成后，删除 kubelet 证书和 kubeconfig 文件

rm /opt/kubernetes/cfg/kubelet.kubeconfig

rm -f /opt/kubernetes/ssl/kubelet*

这几个文件是证书申请审批后自动生成的，每个 Node 不同，必须删除重新生成。

（3）修改主机名

#这两个文件中的主机名要修改掉
vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=
vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: 

（4） 启动并设置开机启动

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

（5） 在 Master 上批准新 Node kubelet 证书申请

kubectl get csr

kubectl certificate approve xxxxxxx

（6）查看 Node 状态验证

Kubectl get node

测试 kubernetes 集群

在 Kubernetes 集群中创建一个 pod（以nginx为例），验证是否正常运行：kubectl create deployment nginx --image=nginx

[root@binary-master ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@binary-master ~]# kubectl get pod
NAME                     READY   STATUS              RESTARTS   AGE
nginx-6799fc88d8-2tf8z   0/1     ContainerCreating   0          15s

暴露内部端口80：

kubectl expose deployment nginx --port=80 --type=NodePort

查看pod与svc：kubectl get pod,svc

[root@binary-master flannel]# kubectl get pod,svc
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6799fc88d8-6g9q2   1/1     Running   0          12m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        10d
service/nginx        NodePort    10.0.0.149   <none>        80:31503/TCP   38m

对外暴露的端口为31503，可以通过浏览器访问集群中任一work node的IP加31503端口来测试：

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ps：这里遇到一个问题，创建了nginx的pod后，该pod一直处于ContainerCreating状态：

[root@binary-master flannel]# kubectl get pod
NAME                     READY   STATUS              RESTARTS   AGE
nginx-6799fc88d8-6g9q2   0/1     ContainerCreating   0          5s

通过 kubectl describe pod nginx 查看pod详细信息：

[root@binary-master ~]# kubectl describe pod nginx
Name:           nginx-6799fc88d8-mlc22
Namespace:      default
Priority:       0
Node:           binary-node/172.31.93.211
Start Time:     Sat, 03 Apr 2021 20:59:57 +0800
Labels:         app=nginx
                pod-template-hash=6799fc88d8
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/nginx-6799fc88d8
Containers:
  nginx:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-74sf7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-74sf7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-74sf7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                 From                  Message
  ----     ------                  ----                ----                  -------
  Normal   Scheduled               95s                                       Successfully assigned default/nginx-6799fc88d8-mlc22 to binary-node
  Warning  FailedCreatePodSandBox  95s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5240a4b7f8d25d59542c8ffb75937999c7d09c58056e7e7d7b9131d99635333b" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  94s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "09f9728baca36797f2efcc111fd8bd63749e8e40dd0a84780a6ddae447c10ffd" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  93s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3453f4e978941bd442bcbaece321c79b48dff8633c87f33f7ef86c2d5de3ada4" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  92s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "25b8a4368c66c23b4ed5ab55b8f40b7020ff2fe5d1be25b0c605cd187be88ec4" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  91s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eb2dcd49db7e1366df8cb2842b37b5dc86b7b07b9a87cb8572aa5907262631d3" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  90s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8e65cc2af4f387b0dcd307d22513e074900f2879c323608a3986c960bcf3f4dd" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  89s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "67af2477a1e0aeac6f5254922ac83518a9cb54af6e667a9dc7e8643095fc2430" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  87s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "09d99848e555b858c0f76f2b55d3c55e73552f464e916ccc56547aa0d24995f1" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  86s                 kubelet, binary-node  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "5d1c24620fe5620f7f4abc098b011a3df99d0c9391e5724d6f8dcdc75568d7d2" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          83s (x12 over 94s)  kubelet, binary-node  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  82s (x4 over 85s)   kubelet, binary-node  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "662c7162eb2922ba76cd4fd3eafd917e3d7e986331896fcee3abae2130e12613" network for pod "nginx-6799fc88d8-mlc22": networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory

提示：binary-node上的kubelet报错：networkPlugin cni failed to set up pod "nginx-6799fc88d8-mlc22_default" network: open /run/flannel/subnet.env: no such file or directory

应该是cni网络插件的问题，推测是之前binary-node节点上kube-proxy没有启动成功而导致。

查了很多网上资料，也没有解决问题，最后重启了下binary-node节点的docker，就可以了。

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------