ETCD 集群部署

目录

• 部署ETCD集群
• Etcd 解析
  • 方法一 修改hosts文件
  • 方法二 增加bind解析
• 下载和分发etcd二进制文件
• 创建etcd证书和私钥
• 创建etcd的启动文件
• 测试 ETCD 集群状态

下面的操作依托于上一篇文章

部署前期准备工作

部署ETCD集群

这里使用的ETCD为三节点高可用集群，步骤如下

• 下载和分发etcd二进制文件
• 创建etcd集群各节点的x509证书，用于加密客户端(如kubectl)与etcd集群、etcd集群之间的数据流
• 创建etcd的system unit文件，配置服务参数
• 检查集群工作状态

注意: 没有特殊说明都在node01节点操作

Etcd 解析

本次使用etcd单独的域名解析

方法一 修改hosts文件

在所有机器上操作， /etc/hosts 文件最后增加下面内容，也可以替换成自己的IP

10.0.20.11 etcd01 etcd01.k8s.com
10.0.20.12 etcd02 etcd02.k8s.com
10.0.20.13 etcd03 etcd03.k8s.com

方法二 增加bind解析

如果使用的内网 DNS bind 做内网解析增加下面解析

etcd01  IN  A   10.0.20.11
etcd02  IN  A   10.0.20.12
etcd03  IN  A   10.0.20.13

下载和分发etcd二进制文件

二进制文件在 部署前期准备工作 文章中已经下载好，直接使用；

分发二进制文件到ETCD集群节点

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-v3.3.13-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
  done

创建etcd证书和私钥

cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.0.20.11",
    "10.0.20.12",
    "10.0.20.13",
    "etcd01.k8s.com",
    "etcd02.k8s.com",
    "etcd03.k8s.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF


#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中

生成证书和私钥

cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem 
    -ca-key=/opt/k8s/work/ca-key.pem 
    -config=/opt/k8s/work/ca-config.json 
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pem

分发证书和私钥到etcd各个节点

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert"
    scp etcd*.pem root@${node_ip}:/etc/etcd/cert/
  done

创建etcd的启动文件

这里相对应的etcd 的配置，就保存在启动文件中

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \
  --data-dir=${ETCD_DATA_DIR} \
  --wal-dir=${ETCD_WAL_DIR} \
  --name=##NODE_NAME## \
  --cert-file=/etc/etcd/cert/etcd.pem \
  --key-file=/etc/etcd/cert/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-cert-file=/etc/etcd/cert/etcd.pem \
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://##NODE_IP##:2380 \
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://##NODE_IP##:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=${ETCD_NODES} \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

配置说明 (此处不需要修改任何配置)

• WorkDirectory、–data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录 (后面跟着我操作就可以，会有创建步骤)
• –wal-dir 指定wal目录，为了提高性能，一般使用SSD和–data-dir不同的盘
• –name 指定节点名称，当–initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster列表中
• –cert-file、–key-file ETCD server与client通信时使用的证书和私钥
• –trusted-ca-file 签名client证书的CA证书，用于验证client证书
• –peer-cert-file、–peer-key-file ETCD与peer通信使用的证书和私钥
• –peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书

拆分三个配置文件，并修改信息

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))
  do
    sed -e "s/##NODE_NAME##/${ETCD_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service 
  done

etcd*.service

分发生成的etcd启动文件到对应的服务器

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
  done

启动etcd服务

etcd首次进程启动会等待其他节点加入etcd集群，执行启动命令会卡顿一会，为正常现象

远程创建对应 ETCD 的数据目录等

source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &
  done

测试 ETCD 集群状态

检查启动结果

cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status etcd|grep Active"
  done

输出结果:

[root@node01 work]# for node_ip in ${MASTER_IPS[@]}
>   do
>     echo ">>> ${node_ip}"
>     ETCDCTL_API=3 /opt/k8s/bin/etcdctl 
>     --endpoints=https://${node_ip}:2379 
>     --cacert=/etc/kubernetes/cert/ca.pem 
>     --cert=/etc/etcd/cert/etcd.pem 
>     --key=/etc/etcd/cert/etcd-key.pem endpoint health
>   done
>>> 10.0.20.11
https://10.0.20.11:2379 is healthy: successfully committed proposal: took = 1.609991ms
>>> 10.0.20.12
https://10.0.20.12:2379 is healthy: successfully committed proposal: took = 1.117871ms
>>> 10.0.20.13
https://10.0.20.13:2379 is healthy: successfully committed proposal: took = 1.49139ms

通过下面命令查看当前etcd集群leader

source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 /opt/k8s/bin/etcdctl 
  -w table --cacert=/etc/kubernetes/cert/ca.pem 
  --cert=/etc/etcd/cert/etcd.pem 
  --key=/etc/etcd/cert/etcd-key.pem 
  --endpoints=${ETCD_ENDPOINTS} endpoint status

输出结果如下：

[root@node01 work]# source /opt/k8s/bin/environment.sh
[root@node01 work]# ETCDCTL_API=3 /opt/k8s/bin/etcdctl 
>   -w table --cacert=/etc/kubernetes/cert/ca.pem 
>   --cert=/etc/etcd/cert/etcd.pem 
>   --key=/etc/etcd/cert/etcd-key.pem 
>   --endpoints=${ETCD_ENDPOINTS} endpoint status
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://etcd01.k8s.com:2379 | 6330dc0a28f62066 |  3.3.13 |   16 kB |     false |        35 |         14 |
| https://etcd02.k8s.com:2379 | 77bc4da10f4c40bb |  3.3.13 |   16 kB |      true |        35 |         14 |
| https://etcd03.k8s.com:2379 | d2573d5cc998d0f0 |  3.3.13 |   16 kB |     false |        35 |         14 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+

---

---

如果对ETCD集群安装不熟悉的，可以参考文章 CentOS 7 ETCD集群配置大全