一、机器及环境准备
二、系统初始配置
三、添加国内镜像源
四、升级系统 && 安装软件
五、自建CA并签发二级CA
六、使用kubeadm部署K8S集群
七、安装网络插件
一、机器及环境准备
master节点三台，worker节点一台，公网负载均衡一个
| 主机名 | 内网IP | 公网IP | 域名 | 解析地址 | 系统版本 | 前端端口 | 后端端口 | 后端服务器组 |
| master1 | 172.30.0.3 | CentOS 7.6 64bit | ||||||
| master2 | 172.30.0.4 | CentOS 7.6 64bit | ||||||
| master3 | 172.30.0.5 | CentOS 7.6 64bit | ||||||
| worker-0001 | 172.30.0.1 | CentOS 7.6 64bit | ||||||
| 负载均衡 | 172.30.0.10 | 1.1.1 | 6444 | 6443 | master1-3 | |||
| 域名 | xxx.yyy.com | 1.1.1.1 |
二、系统初始配置
1、关闭swap
swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab
2、关闭所有防火墙
systemctl stop firewalld systemctl disable firewalld
3、禁用SELINUX
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
4、内核配置
cat << EOF > /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 EOF modprobe br_netfilter modprobe ip_vs
sysctl -p /etc/sysctl.d/kubernetes.conf
5、kube-proxy开启ipvs的前置配置
cat << EOF >/etc/sysconfig/modules/ipvs.modules #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
三、添加国内镜像源
1、添加Docker-ce源
yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
2、添加K8S源
cat << EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
四、升级系统 && 安装软件
1、升级安装
yum update -y yum install -y epel-release yum install -y conntrack ipvsadm ipset jq sysstat curl iptables yum install -y device-mapper-persistent-data lvm2 yum install -y container-selinux yum install -y docker-ce yum install -y kubelet kubeadm kubectl
2、配置 cgroupdriver
修改或创建/etc/docker/daemon.json，加入下面内容：
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
3、启动docker、设置docker、kubelet开机自启(这里先不要开启kubelet服务)
systemctl enable docker && systemctl start docker systemctl enable kubelet
4、验证 docker cgroup
执行以下命令, 预期输出 Cgroup Driver: systemd
docker info |grep Cgroup
五、自建CA并签发二级CA
1、创建根CA
1.1、依次创建如下目录：
mkdir -p /opt/ca/root mkdir /opt/ca/root/key
1.2、vim /opt/ca/root/openssl.cnf
[ ca ] default_ca = CA_default [ CA_default ] dir = /opt/ca/root certs = $dir/certs crl_dir = $dir/crl database = $dir/index.txt new_certs_dir = $dir/newcerts certificate = $dir/key/ca.crt serial = $dir/serial crlnumber = $dir/crlnumber crl = $dir/crl.pem private_key = $dir/key/ca.key RANDFILE = $dir/key/.rand unique_subject = no x509_extensions = usr_cert copy_extensions = copy name_opt = ca_default cert_opt = ca_default default_days = 3650 default_crl_days= 30 default_md = sha256 preserve = no policy = policy_ca [ policy_ca ] countryName = supplied stateOrProvinceName = supplied organizationName = supplied organizationalUnitName = supplied commonName = supplied emailAddress = optional [ req ] default_bits = 2048 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes x509_extensions = v3_ca string_mask = utf8only utf8 = yes prompt = no [ req_distinguished_name ] countryName = CN stateOrProvinceName = beijing localityName = beijing organizationName = Global XXX CA Inc organizationalUnitName = XXX Root CA commonName = Global XXX Root CA [ usr_cert ] basicConstraints = CA:TRUE [ v3_ca ] basicConstraints = CA:TRUE [ req_attributes ]
1.3、创建如下目录及文件
mkdir /opt/ca/root/newcerts touch /opt/ca/root/index.txt touch /opt/ca/root/index.txt.attr echo 01 > /opt/ca/root/serial
1.4、创建根CA私钥
openssl genrsa -out /opt/ca/root/key/ca.key 2048
1.5、创建根CA证书请求文件
openssl req -new -key /opt/ca/root/key/ca.key -out /opt/ca/root/key/ca.csr -config /opt/ca/root/openssl.cnf
1.6、自签根CA证书
openssl ca -selfsign -in /opt/ca/root/key/ca.csr -out /opt/ca/root/key/ca.crt -config /opt/ca/root/openssl.cnf
1.7、查看证书信息（可选）
openssl x509 -text -in /opt/ca/root/key/ca.crt
经过以上几个步骤，就生成了根CA的相关证书和私钥，可以用于签发其他的CA（二级CA），不可签发服务器证书
2、创建二级CA
2.1、创建如下目录
mkdir -p /opt/ca/agent/key
2.2、vim /opt/ca/agent/openssl.cnf
[ ca ] default_ca = CA_default [ CA_default ] dir = /opt/ca/agent certs = $dir/certs crl_dir = $dir/crl database = $dir/index.txt new_certs_dir = $dir/newcerts certificate = $dir/key/ca.crt serial = $dir/serial crlnumber = $dir/crlnumber crl = $dir/crl.pem private_key = $dir/key/ca.key RANDFILE = $dir/key/.rand unique_subject = no x509_extensions = usr_cert copy_extensions = copy name_opt = ca_default cert_opt = ca_default default_days = 3650 default_crl_days= 30 default_md = sha256 preserve = no policy = policy_ca [ policy_ca ] countryName = supplied stateOrProvinceName = supplied organizationName = supplied organizationalUnitName = supplied commonName = supplied emailAddress = optional [ req ] default_bits = 2048 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes x509_extensions = v3_ca string_mask = utf8only utf8 = yes prompt = no [ req_distinguished_name ] countryName = CN stateOrProvinceName = Guangdong localityName = Guangzhou organizationName = Global XXX CA Inc organizationalUnitName = Google 2020 CA commonName = Google 2020 CA [ usr_cert ] basicConstraints = CA:FALSE [ v3_ca ] basicConstraints = CA:TRUE [ req_attributes ]
2.3、创建如下目录及文件
mkdir /opt/ca/agent/newcerts touch /opt/ca/agent/index.txt touch /opt/ca/agent/index.txt.attr echo 01 > /opt/ca/agent/serial
2.4、创建二级CA私钥
openssl genrsa -out /opt/ca/agent/key/ca.key 2048
2.5、创建二级CA证书请求文件
openssl req -new -key /opt/ca/agent/key/ca.key -out /opt/ca/agent/key/ca.csr -config /opt/ca/agent/openssl.cnf
2.6、使用根CA签发二级CA
openssl ca -in /opt/ca/agent/key/ca.csr -out /opt/ca/agent/key/ca.crt -config /opt/ca/root/openssl.cnf
2.7、查看证书信息（可选）
openssl x509 -text -in /opt/ca/agent/key/ca.crt
经过以上几个步骤，就生成了一个二级CA，这个二级CA可以签发服务器证书（不能签发其他的CA）
3、配置kubeadm使用自定义证书
3.1、创建目录
mkdir /etc/kubernetes/pki/
3.2、将二级CA证书及私钥复制到pki目录下，kubeadm初始化中将自动使用此CA签发
cp /opt/ca/agent/key/ca.crt /etc/kubernetes/pki/ cp /opt/ca/agent/key/ca.key /etc/kubernetes/pki/
六、使用kubeadm部署K8S集群
1、打印kubeadm配置文件（master1节点操作）
kubeadm config print init-defaults > kubeadm-confi
2、编辑kubeadm配置文件（master1节点操作）
将advertiseAddress选项中的ip地址改为master1的地址；
添加controlPlaneEndpoint配置，地址为域名xxx.yyy.com，端口为负载均衡映射6443的前端端口6444；
将容器镜像仓库改为阿里仓库；
将kubeproxy的mode改为ipvs。
3、初始化master1节点
kubeadm init --config=kubeadm-config.yml --upload-certs
4、master1节点初始化完成后，逐个将master02、master03、worker01加入集群
第一段为配置kubectl；
第二段为master2、master3执行加入集群；
第三段为worker-0001执行加入集群。
5、如果添加工作节点的命令遗忘，可重新生成token，并输出显示
kubeadm token create --print-join-command --ttl 0
添加master节点则在 kubeadm join 中增加 --control-plane --certificate-key 参数
七、安装网络插件
此处网络插件选用 calico
1、获取calico网络的yaml文件
curl https://docs.projectcalico.org/v3.14/manifests/calico.yaml -o calico.yaml
2、部署flannel网络
kubectl apply -f calico.yaml
3、等待一段时间后，查看所有pod的状态为Running、所有节点的状态是Ready，则集群正常
kubectl get pods -n kube-system
kubectl get nodes -o wide
附：验证证书是否是某CA机构颁发
openssl verify -CAfile /etc/kubernetes/pki/ca.crt kubelet-client-current.pem
回显如下则kubelet-client-current.pem 证书是 /etc/kubernetes/pki/ca.crt 该CA机构颁发
kubelet-client-current.pem: OK
查看证书详情
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout