之前写过一篇手动搭建etcd 3.1集群的文章《etcd 3.1 高可用集群搭建》，最近要初始化一套新的环境，考虑用ansible自动化部署整套环境， 先从部署etcd 3.2集群开始。
需要部署etcd的主机信息如下：
node1 192.168.61.11
node2 192.168.61.12
node3 192.168.61.13
配置管理项目目录结构
├── inventories
│ ├── staging
│ │ ├── group_vars
│ │ │ ├── all.yml
│ │ │ └── etcd-nodes.yml
│ │ ├── host_vars
│ │ │ ├── node1.yml
│ │ │ ├── node2.yml
│ │ │ └── node3.yml
│ │ └── hosts
│ └── production
├── roles
│ ├── common
│ │ ├── defaults
│ │ │ └── main.yml
│ │ └── tasks
│ │ └── main.yml
│ ├── etcd3
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ └── make-ca-cert.sh
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ ├── create_etcd_user.yml
│ │ ├── etcd-restart.yml
│ │ ├── etcd-start.yml
│ │ ├── etcd-stop.yml
│ │ ├── gen-etcd-certs.yml
│ │ ├── gen-etcd-systemd.yml
│ │ ├── install_etcd_bin.yml
│ │ └── main.yml
│ └── templates
│ ├── etcd.conf.j2
│ └── etcd.service.j2
├── deploy-etcd3.yml
roles/etcd3/defaults/main.yml：
---
etcd_version: 3.2.0
etcd_download_url_base: "https://github.com/coreos/etcd/releases/download/v{{ etcd_version }}"
etcd_release: "etcd-v{{ etcd_version }}-linux-amd64"
etcd_download_url: "{{ etcd_download_url_base }}/{{ etcd_release}}.tar.gz"
etcd_bin_path: /usr/bin
etcd_data_dir: /var/lib/etcd
etcd_conf_dir: /etc/etcd
etcd_certs_dir: "{{ etcd_conf_dir }}/ssl"
etcd_cert_group: root
etcd_ca_file: "{{ etcd_certs_dir }}/ca.crt"
etcd_cert_file: "{{ etcd_certs_dir }}/server.crt"
etcd_key_file: "{{ etcd_certs_dir }}/server.key"
etcd_peer_ca_file: "{{ etcd_certs_dir }}/ca.crt"
etcd_peer_cert_file: "{{ etcd_certs_dir }}/peer.crt"
etcd_peer_key_file: "{{ etcd_certs_dir }}/peer.key"
etcd_client_cert_file: "{{ etcd_certs_dir }}/client.crt"
etcd_client_key_file: "{{ etcd_certs_dir }}/client.key"
etcd_client_cert_auth: true
etcd_peer_client_cert_auth: true
etcd_client_port: 2379
etcd_peer_port: 2380
etcd_initial_cluster_state: new
etcd_initial_cluster_token: etcd-k8s-cluster
etcd_initial_advertise_peer_urls: "https://{{ etcd_machine_address }}:{{ etcd_peer_port }}"
etcd_listen_peer_urls: "https://{{ etcd_machine_address }}:{{ etcd_peer_port }}"
etcd_advertise_client_urls: "https://{{ etcd_machine_address }}:{{ etcd_client_port }}"
etcd_listen_client_urls: "https://{{ etcd_machine_address }}:2379,https://127.0.0.1:2379"
创建etcd用户和数据目录
创建etcd用户、用户组和数据目录。
- name: create system etcd group
group:
name: etcd
state: present
- name: create system etcd user
user:
name: etcd
comment: "etcd user"
shell: /sbin/nologin
state: present
system: yes
home: "{{ etcd_data_dir }}"
groups: etcd
- name: ensure etcd_data_dir exists
file:
path: "{{ etcd_data_dir }}"
recurse: yes
state: directory
owner: etcd
group: etcd
下载和解压etcd
下载和解压缩etcd release tar包，并将可执行文件etcd, etcdctl拷贝到/usr/bin。
---
- name: set github s3 host on the first etcd server
lineinfile:
dest: /etc/hosts
regexp: '.*github-production-release-asset-2e65be.s3.amazonaws.com$'
line: "219.76.4.4 github-production-release-asset-2e65be.s3.amazonaws.com"
state: present
delegate_to: "{{ groups['etcd-nodes'][0] }}"
run_once: true
- name: check whether etcd release tar extracted on the first etcd server
stat:
path: "{{ ansible_temp_dir }}/{{ etcd_release }}"
register: etcd_release_tar_check
delegate_to: "{{ groups['etcd-nodes'][0] }}"
run_once: true
- name: download etcd release tar file on first the etcd server
get_url:
url: "{{ etcd_download_url }}"
dest: "{{ ansible_temp_dir }}"
validate_certs: no
timeout: 20
register: download_etcd
delegate_to: "{{ groups['etcd-nodes'][0] }}"
run_once: true
when: not etcd_release_tar_check.stat.exists
- name: extract etcd tar file
unarchive:
src: "{{ download_etcd.dest }}"
dest: "{{ ansible_temp_dir }}"
remote_src: yes
run_once: true
delegate_to: "{{ groups['etcd-nodes'][0] }}"
when: not etcd_release_tar_check.stat.exists
- name: fetch etcd bins from the first etcd server
fetch:
src: "{{ ansible_temp_dir }}/{{ etcd_release }}/{{ item }}"
dest: "tmp/etcd3/{{ item }}"
flat: yes
register: fetch_etcd
run_once: true
delegate_to: "{{ groups['etcd-nodes'][0] }}"
with_items:
- etcd
- etcdctl
- name: copy etcd binary
copy:
src: "tmp/etcd3/{{ item }}"
dest: "{{ etcd_bin_path }}"
owner: etcd
group: etcd
mode: 0750
with_items:
- etcd
- etcdctl
生成并分发etcd TLS证书
---
- name: ensure etcd certs directory
file:
path: "{{ etcd_certs_dir }}"
state: directory
owner: etcd
group: etcd
mode: 0750
recurse: yes
- name: copy make-ca-cert.sh
copy:
src: make-ca-cert.sh
dest: "{{ etcd_certs_dir }}"
owner: root
group: root
mode: "0500"
run_once: true
delegate_to: "{{ groups['etcd-nodes'][0] }}"
- name: gen certs on the first etcd server
command:
"{{ etcd_certs_dir }}/make-ca-cert.sh"
args:
creates: "{{ etcd_certs_dir }}/server.crt"
run_once: true
delegate_to: "{{ groups['etcd-nodes'][0] }}"
environment:
NODE_IPS: "{% for host in groups['etcd-nodes'] %}{{ hostvars[host]['etcd_machine_address'] }}{% if not loop.last %},{% endif %}{% endfor %}"
NODE_DNS: "{{ groups['etcd-nodes']|join(',') }}"
CERT_DIR: "{{ etcd_certs_dir }}"
CERT_GROUP: "{{ etcd_cert_group }}"
- name: slurp etcd certs
slurp:
src: "{{ item }}"
register: pki_certs
run_once: true
delegate_to: "{{ groups['etcd-nodes'][0] }}"
with_items:
- "{{ etcd_ca_file }}"
- "{{ etcd_cert_file }}"
- "{{ etcd_key_file }}"
- "{{ etcd_peer_ca_file }}"
- "{{ etcd_peer_cert_file }}"
- "{{ etcd_peer_key_file }}"
- "{{ etcd_client_cert_file }}"
- "{{ etcd_client_key_file }}"
- name: copy etcd certs to other etcd servers
copy:
dest: "{{ item.item }}"
content: "{{ item.content | b64decode }}"
owner: etcd
group: "{{ etcd_cert_group }}"
mode: 0400
with_items: "{{ pki_certs.results }}"
when: inventory_hostname != groups['etcd-nodes'][0]
systemd和配置
---
- name: create etcd systemd unit file
template:
src: etcd.service.j2
dest: /etc/systemd/system/etcd.service
- name: create etcd env conf
template:
src: etcd.conf.j2
dest: /etc/etcd/etcd.conf
owner: etcd
group: etcd
mode: 0540
启动etcd
---
- name: start etcd
systemd:
name: etcd
daemon_reload: yes
state: started
enabled: yes
- name: restart etcd
systemd:
name: etcd
state: restarted
查看集群状态
检查集群是否健康，在任一节点执行：
etcdctl
--ca-file=/etc/etcd/ssl/ca.crt
--cert-file=/etc/etcd/ssl/client.crt
--key-file=/etc/etcd/ssl/client.key
--endpoints=https://node1:2379,https://node2:2379,https://node3:2379
cluster-health
member 1e3da2bf674fd07 is healthy: got healthy result from https://192.168.61.11:2379
member 88548a72a2e9a749 is healthy: got healthy result from https://192.168.61.13:2379
member c3bda13bf78ed2ab is healthy: got healthy result from https://192.168.61.12:2379
cluster is healthy
etcdctl
--ca-file=/etc/etcd/ssl/ca.crt
--cert-file=/etc/etcd/ssl/client.crt
--key-file=/etc/etcd/ssl/client.key
--endpoints=https://node1:2379,https://node2:2379,https://node3:2379
member list
1e3da2bf674fd07: name=node1 peerURLs=https://192.168.61.11:2380 clientURLs=https://192.168.61.11:2379 isLeader=false
88548a72a2e9a749: name=node3 peerURLs=https://192.168.61.13:2380 clientURLs=https://192.168.61.13:2379 isLeader=false
c3bda13bf78ed2ab: name=node2 peerURLs=https://192.168.61.12:2380 clientURLs=https://192.168.61.12:2379 isLeader=true
标题：使用Ansible部署etcd 3.2高可用集群
本文链接：http://blog.frognew.com/2017/06/using-ansible-deploy-etcd-cluster.html
转载请注明出处。
本文链接：http://blog.frognew.com/2017/06/using-ansible-deploy-etcd-cluster.html
转载请注明出处。