K8S在原有的集群上新增node节点（v1.19.5）

一、原集群环境说明

主机名	IP地址	说明
k8s-master01	192.168.1.100	master节点
k8s-master02	192.168.1.101	master节点
k8s-master03	192.168.1.102	master节点
k8s-master-lb（在master节点）	192.168.1.246	keepalived虚拟IP
k8s-node01	192.168.1.103	worker节点
k8s-node02	192.168.1.104	worker节点

配置信息	备注
系统版本	CentOS 7.9
Docker版本	19.03.x
Pod网段	172.168.0.0/12
Service网段	10.96.0.0/12

注意：

VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用。VIP需要和主机在同一个局域网内！

二、基础环境准备（基本上都在新的机器上操作）

2.1、配置hosts解析（master01执行）

cat >> /etc/hosts << EFO 
192.168.1.105 k8s-node03
EFO

2.2、更换yum源码

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

2.3、安装常用工具

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y

2.4、关闭防火墙、selinux、dnsmasq、swap

systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

# 关闭swap分区
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

2.5、时间同步配置

# 安装ntpdate
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install ntpdate -y

# 更改时区
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# 设置定时任务同步时间
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com

# 加入到crontab
crontab -e
*/5 * * * * ntpdate time2.aliyun.com

2.6、优化Linux

ulimit -SHn 65535

vim /etc/security/limits.conf
# 末尾添加如下内容
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

2.7、所有节点升级系统并重启，此处升级没有升级内核，下节会单独升级内核：

# CentOS7需要升级，CentOS8可以按需升级系统
yum update -y --exclude=kernel* && reboot 

2.8、设置主机名

[root@localhost ~]# hostnamectl set-hostname k8s-node03
[root@localhost ~]# bash

三、内核升级

3.1、配置免密登录（Master01上）

Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作，阿里云或者AWS上需要单独一台kubectl服务器。密钥配置如下：

# 一直回车就行
ssh-keygen -t rsa 

for i in k8s-node03;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

# 分发新的hosts文件
for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02 k8s-node03;do scp /etc/hosts $i:/etc ;done

3.2、分发升级所需安装包（Master01上）

CentOS7 需要升级内核至4.18+，本地升级的版本为4.19

# 在master01节点下载内核
cd /root
# 从master01节点传到其他节点：
for i in k8s-node03;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done

3.4、内核升级

# 安装内核
cd /root && yum localinstall -y kernel-ml*
grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

# 检查默认内核是不是4.19
grubby --default-kernel /boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64

# 所有节点重启，然后检查内核是不是4.19
reboot
[root@k8s-node02 ~]# uname -a
Linux k8s-node02 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

3.5、安装ipvsadm

yum install ipvsadm ipset sysstat conntrack libseccomp -y

配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：

# 加入以下内容
cat > /etc/modules-load.d/ipvs.conf << EFO
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack   # 4.18 改成这个nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EFO

# 然后执行
systemctl enable --now systemd-modules-load.service

3.6、开启一些k8s集群中必须的内核参数，配置k8s内核

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

# 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
reboot
[root@k8s-master01 ~]# lsmod | grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp              16384  0 
nf_nat                 32768  1 ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_sh               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs                 151552  24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  2 nf_nat,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,ip_vs

四、Docker安装

4.1、安装Docker-ce 19.03

yum install docker-ce-19.03.* -y

4.1.1温馨提示：

由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

4.1.2、所有节点设置开机自启动Docker

systemctl daemon-reload && systemctl enable --now docker

五、K8s安装及证书拷贝

5.1、将组件发送到新节点（Master01上）

[root@k8s-master01 ~]# WorkNodes='k8s-node03'
[root@k8s-master01 ~]# for NODE in $WorkNodes; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done

5.2、版本查看（新节点）

[root@k8s-node03 ~]# kubelet --version
Kubernetes v1.19.5

5.3、创建/opt/cni/bin目录（新节点）

mkdir -p /opt/cni/bin
mkdir -p /etc/kubernetes/pki

5.4、发送证书（master01）

WorkNodes='k8s-node03'
ssh $NODE "mkdir -p /etc/etcd/ssl"
     for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
       scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
     done
     
     
[root@k8s-master01 ~]# cd /etc/kubernetes/
for NODE in k8s-node03; do
     ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl
     for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do
       scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/
     done
     for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
       scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
 done
 done

六、Kubelet配置

6.1、创建相关目录

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

6.2、配置kubelet service

vim  /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

6.3、配置kubelet service的配置文件

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

6.4、kubelet的配置文件、启动所有节点kubelet

注意：如果更改了k8s的service网段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成k8s Service网段的第十个地址，比如10.96.0.10（k8s的service网段开始设置的是10.96.0.0/12）

vim /etc/kubernetes/kubelet-conf.yml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

6.5、启动kubelet

systemctl daemon-reload
systemctl enable --now kubelet
systemctl status kubelet

# 查看此时系统日志
tail -f /var/log/messages

6.6、查看集群状态（matser01上）

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE     VERSION
k8s-master01   Ready    <none>   12d     v1.19.5
k8s-master02   Ready    <none>   12d     v1.19.5
k8s-master03   Ready    <none>   12d     v1.19.5
k8s-node01     Ready    <none>   12d     v1.19.5
k8s-node02     Ready    <none>   12d     v1.19.5
k8s-node03     Ready    <none>   3m21s   v1.19.5   # 已经多了一个节点了

七、kube-proxy配置

7.1、分发配置文件（master01上）

[root@k8s-master01 ~]# cd /root/k8s-ha-install

for NODE in k8s-node03; do
     scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
     scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
     scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
 done

7.2、启动kube-proxy

systemctl daemon-reload && systemctl enable --now kube-proxy; systemctl status kube-proxy

八、查看Calico状态

# 可以看到已经有一个新的Calico部署在新的节点上了
[root@k8s-master01 k8s-ha-install]# kubectl  get po -n kube-system -owide
NAME                    READY   STATUS    RESTARTS   AGE     IP                NODE 
calico-node-xx6wk       1/1     Running   6          8m55s   192.168.1.105     k8s-node03 

九、查看Metrics Server

# 也可以获取到k8s-node03的信息
[root@k8s-master01 ~]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   516m         12%    2536Mi          61%       
k8s-master02   451m         11%    1227Mi          29%       
k8s-master03   370m         9%     1122Mi          29%       
k8s-node01     249m         6%     1944Mi          50%       
k8s-node02     236m         5%     558Mi           14%       
k8s-node03     159m         7%     524Mi           28%   

十、集群验证

# 节点安装telnet命令,有的话忽略
yum install -y telnet

# 新机器 10.96.0.1  443  kubernetes svc 443
# 新机器 10.96.0.10 53   kube-dns的service 53
# 不会自动断开就是成功了
telnet 10.96.0.1 443
telnet 10.96.0.10 53

Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.


# 使用curl命令验证（新机器）
[root@k8s-node03 ~]# curl 10.96.0.10:53
curl: (52) Empty reply from server