第二篇(二进制部署k8s集群---etcd集群搭建)
本文etcd集群用三台centos7搭建完成。
etcd1:192.168.206.31
etcd2:192.168.206.32
etcd3:192.168.206.33
一、创建CA证书和密钥,下面步骤是在k8s-master1上操作的。
1、所有机器上创建相关目录
mkdir -p /opt/kubernetes/{bin,ssl,yaml,conf,log,cfg}
echo 'export PATH=$PATH:/opt/kubernetes/bin' >> /etc/profile
source /etc/profile
2、下载cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
3、创建证书签发机构CA,先创建一个证书制作的文件夹。
mkdir -p /data/ssl
cd /data/ssl
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile;
signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
server auth:表示 client 可以用该 CA 对 server 提供的证书进行验证;
client auth:表示 server 可以用该 CA 对 client 提供的证书进行验证;
4、创建CA证书签名请求
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "hangzhou",
"ST": "Zhejiang",
"O": "k8s",
"OU": "System"
}
]
}
EOF
CN:Common Name,kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name),浏览器使用该字段验证网站是否合法;
O:Organization,kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group);
kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识;
6、生成 CA 证书和私钥:
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@master1 ssl]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
创建存放etcd证书文件夹
mkdir /opt/kubernetes/ssl/etcd/
cp *.pem /opt/kubernetes/ssl/
7、创建 etcd 证书签名请求,先创建一个制作证书的文件夹
mkdir /data/ssl/etcd
cd /data/ssl/etcd
cat > etcd-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"192.168.206.31",
"192.168.206.32",
"192.168.206.33"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "hangzhou",
"ST": "Zhejiang",
"O": "k8s",
"OU": "System"
}
]
}
EOF
8、生成etcd证书和对应的私钥
cfssl gencert -ca=/data/ssl/ca.pem
-ca-key=/data/ssl/ca-key.pem
-config=/data/ssl/ca-config.json
-profile=kubernetes etcd-csr.json
| cfssljson -bare etcd
cp *.pem /opt/kubernetes/ssl/etcd
注意:证书3哥etcd都要放哦。
ETCD使用证书的组件如下:
etcd:使用 ca.pem、etcd-key.pem、etcd.pem;
二、部署etcd集群
1、下载etcd安装包
wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
tar zxf etcd-v3.3.10-linux-amd64.tar.gz
cp etcd-v3.3.10-linux-amd64/etcd* /opt/kubernetes/bin/
2、创建工作目录
mkdir /opt/kubernetes/etcd
3、创建systemd unit 文件(修改对应etcd主机名称和ip)
cat > /etc/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/opt/kubernetes/etcd/
ExecStart=/opt/kubernetes/bin/etcd
--name etcd1
--cert-file=/opt/kubernetes/ssl/etcd/etcd.pem
--key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem
--peer-cert-file=/opt/kubernetes/ssl/etcd/etcd.pem
--peer-key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem
--trusted-ca-file=/opt/kubernetes/ssl/etcd/ca.pem
--peer-trusted-ca-file=/opt/kubernetes/ssl/etcd/ca.pem
--initial-advertise-peer-urls https://192.168.206.31:2380
--listen-peer-urls https://192.168.206.31:2380
--listen-client-urls https://192.168.206.31:2379,http://127.0.0.1:2379
--advertise-client-urls https://192.168.206.31:2379
--initial-cluster-token etcd-cluster-0
--initial-cluster etcd1=https://192.168.206.31:2380,etcd2=https://192.168.206.32:2380,etcd3=https://192.168.206.33:2380
--initial-cluster-state new
--data-dir=/opt/kubernetes/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
为了保证通信安全,需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书(trusted-ca-file);
创建etcd.pem 证书时使用的 etcd-csr.json 文件的 hosts 字段包含所有 etcd 节点的IP,否则证书校验会出错;
–initial-cluster-state 值为 new 时,–name 的参数值必须位于 –initial-cluster 列表中.
4、启动etcd服务并且设置开机自启动
systemctl daemon-reload
systemctl start etcd.service
systemctl status etcd.service
systemctl enable etcd.service
最先启动的 etcd 进程会卡住一段时间,等待其它节点上的 etcd 进程加入集群,为正常现象。
5、验证etcd集群状态,以及查看leader,在任何一个etcd节点执行
[root@k8s-master1 etcd]# etcdctl --ca-file=/opt/kubernetes/ssl/etcd/ca.pem --cert-file=/opt/kubernetes/ssl/etcd/etcd.pem --key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem cluster-health
member 8b7aa04311a7389f is healthy: got healthy result from https://192.168.206.32:2379
member 9b7dcd0eef5c1758 is healthy: got healthy result from https://192.168.206.33:2379
member f617b05c8dbf5231 is healthy: got healthy result from https://192.168.206.31:2379
cluster is healthy
[root@k8s-master1 etcd]# etcdctl --ca-file=/opt/kubernetes/ssl/etcd/ca.pem --cert-file=/opt/kubernetes/ssl/etcd/etcd.pem --key-file=/opt/kubernetes/ssl/etcd/etcd-key.pem member list
8b7aa04311a7389f: name=etcd2 peerURLs=https://192.168.206.32:2380 clientURLs=https://192.168.206.32:2379 isLeader=false
9b7dcd0eef5c1758: name=etcd3 peerURLs=https://192.168.206.33:2380 clientURLs=https://192.168.206.33:2379 isLeader=true
f617b05c8dbf5231: name=etcd1 peerURLs=https://192.168.206.31:2380 clientURLs=https://192.168.206.31:2379 isLeader=false
至此ETCD TLS证书集群部署完成