etcd cluster安装及应用

一、环境准备：

10.10.0.170      k8s-master
10.10.0.171      k8s-node1
10.10.0.172     k8s-node2

 二、安装：

2.1  建立主机信任：

k8s-master上执行下列命令：

ssh-keygen -t  rsa                #一路回车即可
ssh-copy-id k8s-master
ssh-copy-id k8s-node1
ssh-copy-id k8s-node2

2.2  设置cfssl环境(k8s-master上执行):

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

2.3  创建CA配置文件：

cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "8760h"
},
"profiles": {
  "kubernetes-Soulmate": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
    "expiry": "8760h"
  }
}
}
}
EOF

cat >  ca-csr.json <<EOF
{
"CN": "kubernetes-Soulmate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
  "C": "CN",
  "ST": "shanghai",
  "L": "shanghai",
  "O": "k8s",
  "OU": "System"
}
]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "10.10.0.170",
    "10.10.0.171",
    "10.10.0.172"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

[root@k8s-master ssl]# ls
ca-config.json

ca.csr

ca-csr.json

ca-key.pem

ca.pem

etcd.csr

etcd-csr.json

etcd-key.pem

etcd.pem
2.4 将etcd的证书分发到k8s-node1、 k8s-node2（k8s-master上执行）：

mkdir /etc/etcd/ssl/
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n k8s-node1 "mkdir -p /etc/etcd/ssl && exit"
ssh -n k8s-node2 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem k8s-node1:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem k8s-node2:/etc/etcd/ssl/

三、安装etcd（3个节点都执行）： 

yum install etcd -y

四、etcd.service配置：

k8s-master：

[root@k8s-master ssl]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name k8s-master   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://10.10.0.170:2380   --listen-peer-urls https://10.10.0.170:2380   --listen-client-urls https://10.10.0.170:2379,http://127.0.0.1:2379   --advertise-client-urls https://10.10.0.170:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster k8s-master=https://10.10.0.170:2380,k8s-node1=https://10.10.0.171:2380,k8s-node2=https://10.10.0.172:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

k8s-node1:

[root@k8s-node1 ~]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name k8s-node1   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://10.10.0.171:2380   --listen-peer-urls https://10.10.0.171:2380   --listen-client-urls https://10.10.0.171:2379,http://127.0.0.1:2379   --advertise-client-urls https://10.10.0.171:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster k8s-master=https://10.10.0.170:2380,k8s-node1=https://10.10.0.171:2380,k8s-node2=https://10.10.0.172:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

k8s-node2:

[root@k8s-node2 ~]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/bin/etcd   --name k8s-node2   --cert-file=/etc/etcd/ssl/etcd.pem   --key-file=/etc/etcd/ssl/etcd-key.pem   --peer-cert-file=/etc/etcd/ssl/etcd.pem   --peer-key-file=/etc/etcd/ssl/etcd-key.pem   --trusted-ca-file=/etc/etcd/ssl/ca.pem   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem   --initial-advertise-peer-urls https://10.10.0.172:2380   --listen-peer-urls https://10.10.0.172:2380   --listen-client-urls https://10.10.0.172:2379,http://127.0.0.1:2379   --advertise-client-urls https://10.10.0.172:2379   --initial-cluster-token etcd-cluster-0   --initial-cluster k8s-master=https://10.10.0.170:2380,k8s-node1=https://10.10.0.171:2380,k8s-node2=https://10.10.0.172:2380   --initial-cluster-state new   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

参数解释：

--name

etcd集群中的节点名，这里可以随意，可区分且不重复就行 

--listen-peer-urls
监听的用于节点之间通信的url，可监听多个，集群内部将通过这些url进行数据交互(如选举，数据同步等)
--initial-advertise-peer-urls
建议用于节点之间通信的url，节点间将以该值进行通信。
--listen-client-urls
监听的用于客户端通信的url,同样可以监听多个。
--advertise-client-urls
建议使用的客户端通信url,该值用于etcd代理或etcd成员与etcd节点通信。
--initial-cluster-token etcd-cluster-1
节点的token值，设置该值后集群将生成唯一id,并为每个节点也生成唯一id,当使用相同配置文件再启动一个集群时，只要该token值不一样，etcd集群就不会相互影响。
--initial-cluster
也就是集群中所有的initial-advertise-peer-urls 的合集
--initial-cluster-state new
新建集群的标志

三个节点执行下列命令： 

systemctl daemon-reload

systemctl enable etcd
systemctl start etcd
systemctl status etcd

检查etcd集群健康性（可三个节点都试试）：

[root@k8s-master ssl]# etcdctl --endpoints=https://10.10.0.170:2379,https://10.10.0.171:2379,https://10.10.0.172:2379 \
>   --ca-file=/etc/etcd/ssl/ca.pem \
>   --cert-file=/etc/etcd/ssl/etcd.pem \
>   --key-file=/etc/etcd/ssl/etcd-key.pem  cluster-health
member 1c25bde2973f71cf is healthy: got healthy result from https://10.10.0.172:2379
member 3222a6aebdf856ac is healthy: got healthy result from https://10.10.0.170:2379
member 5796b25a0b404b92 is healthy: got healthy result from https://10.10.0.171:2379
cluster is healthy