Centos7环境下etcd集群的搭建

Centos7环境下etcd集群的搭建

一、简介

"A highly-available key value store for shared configuration and service discovery."

Etcd是coreos开发的分布式服务系统，内部采用raft协议作为一致性算法。作为一个高可用的配置共享、服务发现的键值存储系统，Etcd有以下的特点：

简单：安装配置简单，而且提供了 HTTP API 进行交互，使用也很简单
安全：支持 SSL 证书验证
快速：根据官方提供的数据，单实例支持每秒2k+读操作、1k写操作
可靠：采用raft算法，实现分布式系统数据的可用性和一致性
Etcd构建自身高可用集群主要有三种形式:

静态发现: 预先已知 Etcd 集群中有哪些节点，在启动时直接指定好Etcd的各个node节点地址
Etcd动态发现: 通过已有的Etcd集群作为数据交互点，然后在扩展新的集群时实现通过已有集群进行服务发现的机制
DNS动态发现: 通过DNS查询方式获取其他节点地址信息
本文主要介绍第一种方式，后续会陆续介绍剩下的两种方式。(直接docker安装请移步：quay.io/coreos/etcd 基于Docker镜像的集群搭建)

二、环境介绍

三台虚拟机，系统环境均为Centos7，对应节点名称及IP地址如下，将这个信息添加到三台主机的hosts文件中，编辑/etc/hosts

[root@node1 ~]# cat /etc/hosts

192.168.3.198 node1
192.168.3.199 node2
192.168.3.200 node3

三、安装、配置Etcd

在3个节点上都安装etcd服务

# yum install etcd -y
yum安装的etcd默认配置文件在/etc/etcd/etcd.conf，以下将三个节点上的配置贴出来，请注意不同点（未贴出的，则表明不需要更改）

node1

[root@node1 etcd]# cat etcd.conf 
#[Member]
#ETCD_CORS=""
# 数据存放位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
# 监听其他 Etcd 实例的地址
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
# 监听客户端地址
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
# 节点名称
ETCD_NAME="node1"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#
#[Clustering]
# 通知其他 Etcd 实例地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://node1:2380"
# 通知客户端地址
ETCD_ADVERTISE_CLIENT_URLS="http://node1:2379,http://node1:4001"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
# 初始化集群内节点地址
ETCD_INITIAL_CLUSTER="node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380"
# 初始化集群 token
ETCD_INITIAL_CLUSTER_TOKEN="chinasoft-etcd-cluster"
# 初始化集群状态，new 表示新建
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"

[root@node1 etcd]# egrep '^[a-Z]' etcd.conf 
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"
ETCD_NAME="node1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://node1:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://node1:2379,http://node1:4001"
ETCD_INITIAL_CLUSTER="node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380"
ETCD_INITIAL_CLUSTER_TOKEN="chinasoft-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2

#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="node2"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://node2:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://node2:2379,http://node2:4001"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380"
ETCD_INITIAL_CLUSTER_TOKEN="chinasoft-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"

node3

#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http://0.0.0.0:4001"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="node3"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://node3:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://node3:2379,http://node3:4001"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380"
ETCD_INITIAL_CLUSTER_TOKEN="chinasoft-etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"

改好配置之后，在各个节点上开启etcd服务：
[root@node1 etcd]# systemctl restart etcd
[root@node1 etcd]# systemctl enable etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
[root@node1 etcd]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-10 11:30:20 CST; 10s ago
 Main PID: 31636 (etcd)
   CGroup: /system.slice/etcd.service
           └─31636 /usr/bin/etcd --name=node1 --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://0.0.0.0:2379,http://0.0.0.0:4001

Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)
Jan 10 11:30:30 node1 etcd[31636]: request cluster ID mismatch (got c304f47342ea490a want cdf818194e3a8c32)


四、测试验证
[root@node2 etcd]# etcdctl set testdir/testkey0 0
0
[root@node2 etcd]# etcdctl set testdir/testkey1 1
1
[root@node2 etcd]# etcdctl set testdir/testkey2 2
2
[root@node2 etcd]# etcdctl ls
/testdir
[root@node2 etcd]# etcdctl ls testdir
/testdir/testkey0
/testdir/testkey1
/testdir/testkey2
[root@node2 etcd]# etcdctl get testdir/testkey2
2
[root@node2 etcd]# etcdctl member list
3914cfb876a6deeb: name=node2 peerURLs=http://node2:2380 clientURLs=http://node2:2379,http://node2:4001 isLeader=true
4d3a7c30064f762d: name=node1 peerURLs=http://node1:2380 clientURLs= isLeader=false
a68deb812f62fac3: name=node3 peerURLs=http://node3:2380 clientURLs=http://node3:2379,http://node3:4001 isLeader=false


在其他节点也能看到创建的节点，说明集群搭建成功
[root@node3 ~]# etcdctl ls testdir
/testdir/test0
/testdir/test1
/testdir/test88