etcd+confd+nginx实现服务自动注册与发现

一、etcd集群搭建

etcd集群搭建可参考：https://www.cnblogs.com/shenjianping/p/14399264.html

二、nginx的安装与启动

1、安装前环境准备

yum install gcc-c++  #gcc编译
yum install -y pcre pcre-devel  #nginx http正则
yum install -y zlib zlib-devel  #nginx http 包进行内容gzip
yum install -y openssl openssl-devel #nginx也支持https

2、下载源码包

[root@localhost software]# wget -c https://nginx.org/download/nginx-1.12.0.tar.gz

3、解压缩

[root@localhost software]# tar -xzvf nginx-1.12.0.tar.gz

4、配置安装目录

[root@localhost nginx-1.12.0]# ./configure --prefix=/usr/local/nginx

5、编译和安装

[root@localhost nginx-1.12.0]# make && make install

6、启动nginx

[root@localhost nginx-1.12.0]# cd /usr/local/nginx/sbin/
[root@localhost sbin]# ./nginx 

此时可以通过ip地址进行访问测试。

三、confd

1、下载与安装

# 下载二进制文件
[root@localhost software]# wget https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64

# 重命名并移动到PATH路径下
[root@localhost software]# mv confd-0.16.0-linux-amd64 /usr/local/bin/confd

# 给予执行权限
[root@localhost software]# chmod +x /usr/local/bin/confd

# 验证是否安装成功
[root@localhost software]# confd --help

2、confd的配置

• 创建confdir

[root@localhost bin]# mkdir -p /etc/confd/{conf.d,templates}
[root@localhost bin]# ll /etc/confd/
total 0
drwxr-xr-x 2 root root 6 Feb 15 20:49 conf.d
drwxr-xr-x 2 root root 6 Feb 15 20:49 templates

创建conf.d和templates目录：

• conf.d：confd的配置文件
• templates：配置模板Template

3、Template Resources

模板源TOML格式的配置文件，主要包含配置的生成逻辑，例如模板源，后端存储对应的keys，命令执行等。默认目录在/etc/confd/conf.d。

• /etc/confd/conf.d/myapp.toml

[template]
src = "nginx.tmpl"
dest = "/project/temp/mynginx.conf"
keys = [
  "/myapp/web/nginx/upstream",
  "/myapp/web/nginx/subdomain"
]
check_cmd = "/usr/local/nginx/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/local/nginx/sbin/nginx reload"

4、Template

Template定义了单一应用配置的模板，默认存储在 /etc/confd/templates目录下。模板语法详见：https://github.com/kelseyhightower/confd/blob/master/docs/templates.md

•  /etc/confd/templates/nginx.tmpl

pstream {{getv "/myapp/web/nginx/subdomain"}} {
{{range getvs "/myapp/web/nginx/upstream/*"}}
server {{.}};
{{end}}
}
server {
server_name {{getv "/myapp/web/nginx/subdomain"}}.example.com;
location / {
proxy_pass http://{{getv "/myapp/web/nginx/subdomain"}};
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

四、进行测试

1、向etcd集群添加key和value

[root@localhost etcd-v3.3.10-linux-amd64]# export ETCDCTL_API=3

[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/subdomain myapp

[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/upstream/app1 "192.168.159.132:80"
[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/upstream/app2 "192.168.159.133:80"

2、执行confd

[root@localhost etc]# confd -watch -backend etcdv3 -node http://192.168.159.128:2379
2021-02-16T13:02:26+08:00 localhost.localdomain confd[5674]: INFO Backend set to etcdv3
2021-02-16T13:02:26+08:00 localhost.localdomain confd[5674]: INFO Starting confd
...

3、查看结果

查看生成的配置文件/project/temp/mynginx.conf

upstream myapp {

server 192.168.159.132:80;
server 192.168.159.133:80;

}
server {
server_name myapp.example.com;
location / {
proxy_pass http://myapp;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

4、整合nginx.conf

/project/temp/mynginx.conf是生成的配置，需要将其整合到nginx启动项的配置中/usr/local/nginx/conf/nginx.conf，在http模块下使用include命令即可：

http {
    include       mime.types;
    include       /project/temp/mynginx.conf   #进行整合
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;
...
}

五、注意事项

• confd执行后台参数不正确

confd的执行confd -watch -backend etcdv3 -node http://192.168.159.128:2379中后台需要使用etcdv3，不能使用etcd，否则会报错，其报错内容如下：

[root@localhost etc]# confd -watch -backend etcd -node http://192.168.159.128:2379
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Backend set to etcd
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Starting confd
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Backend source(s) set to http://192.168.159.128:2379
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: ERROR 100: Key not found (/myapp) [61]

• confd执行域名地址不存在或者不正确

向etcd中添加nginx检查无法通过的地址会报错，所以一定要是配置正确的地址，其报错内容如下：

[root@localhost etc]# confd -watch -backend etcdv3 -node http://192.168.159.128:2379
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Backend set to etcdv3
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Starting confd
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Backend source(s) set to http://192.168.159.128:2379
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Target config /project/temp/mynginx.conf out of sync
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: ERROR "/bin/sh: -c: line 0: syntax error near unexpected token `newline' /bin/sh: -c: line 0: `/usr/local/nginx/sbin/nginx -t -c <no value>' "
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: ERROR Config check failed: exit status 1