目录
Octavia 为什么需要自建 CA 证书？
Note: For production use the ca issuing the client certificate and the ca issuing the server certificate need to be different so a hacker can’t just use the server certificate from a compromised amphora to control all the others.
可以想象，如果 octavia-worker、amphora 通信使用的证书与 User、Dashboard 通信使用的证书相同，那么 User 进出 amphora 就如入无人之地。简而言之，Octavia 自建 CA 证书是为了防止恶意用户登录并利用 amphora 作为 “肉鸡” 来攻击 OpenStack 的内部网络。
Octavia 提供了脚本 octavia/bin/create_certificates.sh 和配置文件 octavia/etc/certificates/openssl.cnf，只需执行下述指令即可完成自建 CA。
$ source /opt/rocky/octavia/bin/create_certificates.sh /etc/octavia/certs/ /opt/rocky/octavia/etc/certificates/openssl.cnf
$ ll /etc/octavia/certs/
total 44
-rw-r--r-- 1 stack stack 1294 Oct 26 12:51 ca_01.pem
-rw-r--r-- 1 stack stack 989 Oct 26 12:51 client.csr
-rw-r--r-- 1 stack stack 1708 Oct 26 12:51 client.key
-rw-r--r-- 1 stack stack 4405 Oct 26 12:51 client-.pem
-rw-r--r-- 1 stack stack 6113 Oct 26 12:51 client.pem
-rw-r--r-- 1 stack stack 71 Oct 26 12:51 index.txt
-rw-r--r-- 1 stack stack 21 Oct 26 12:51 index.txt.attr
-rw-r--r-- 1 stack stack 0 Oct 26 12:51 index.txt.old
drwxr-xr-x 2 stack stack 20 Oct 26 12:51 newcerts
drwx------ 2 stack stack 23 Oct 26 12:51 private
-rw-r--r-- 1 stack stack 3 Oct 26 12:51 serial
-rw-r--r-- 1 stack stack 3 Oct 26 12:51 serial.old
- ca_01.pem：CA 证书文件
- client.csr：Server CSR 证书签名请求文件
- client.key：Server 私钥文件
- client-.pem：PEM 编码的 Server 证书文件
- client.pem：结合了 client-.pem 和 client.key 的文件
与 CA 认证相关的主要有以下配置：
[certificates]
ca_private_key_passphrase = foobar
ca_private_key = /etc/octavia/certs/private/cakey.pem
ca_certificate = /etc/octavia/certs/ca_01.pem
[haproxy_amphora]
server_ca = /etc/octavia/certs/ca_01.pem
client_cert = /etc/octavia/certs/client.pem
[controller_worker]
client_ca = /etc/octavia/certs/ca_01.pem
-
[haproxy_amphora]section 在 octavia-worker AmphoraAPIClient 被应用，AmphoraAPIClient 拿着 client.pem（包含 Server 证书、Server 私钥）和 CA 公钥向 amphora-agent 发起 SSL 通信。 -
[certificates]section 应用在 create amphora flow 的 GenerateServerPEMTask 中，指定 CA 中心为 amphora 签署服务端证书。 -
[controller_worker]的 client_ca 在 Task:CertComputeCreate 中被应用，指定了 CA 证书路径。
下面主要围绕这 3 个 section 来进行分析。
GenerateServerPEMTask
从 UML 图可以看出当 octavia-worker 需要使用 REST 方式与 amphora-agent 通信时 create_amphora_flow 才会加载 GenerateServerPEMTask 用于签发一张 PEM 编码的 Amphora 证书用于建立 HTTPS 通信。
class GenerateServerPEMTask(BaseCertTask):
"""Create the server certs for the agent comm
Use the amphora_id for the CN
"""
def execute(self, amphora_id):
cert = self.cert_generator.generate_cert_key_pair(
cn=amphora_id,
validity=CERT_VALIDITY)
return cert.certificate + cert.private_key
octavia.certificates 实现了 local_cert_generator(default) 和 anchor_cert_generator 两种证书生成器，通过配置 [certificates] cert_generator 选定，这里以 local_cert_generator 为例：
# file: /opt/rocky/octavia/octavia/certificates/generator/local.py
@classmethod
def generate_cert_key_pair(cls, cn, validity, bit_length=2048,
passphrase=None, **kwargs):
pk = cls._generate_private_key(bit_length, passphrase)
csr = cls._generate_csr(cn, pk, passphrase)
cert = cls.sign_cert(csr, validity, **kwargs)
cert_object = local_common.LocalCert(
certificate=cert,
private_key=pk,
private_key_passphrase=passphrase
)
return cert_object
method generate_cert_key_pair 的语义为：
- 生成 Amphora 私钥
- 生成 Amphora 证书签名请求（CSR）
- 向 CA 中心申请签署 Amphora 证书
区别于 create_certificates.sh 脚本的 openssl 指令，octavia-worker 使用了内建的 cryptography 库来实现。GenerateServerPEMTask 的是 Amphora 的 server.pem，包含了 Amphora 证书（公钥）和 Amphora 私钥。
(Pdb) cert
<octavia.certificates.common.local.LocalCert object at 0x7ff1b879ee50>
(Pdb) cert.certificate + cert.private_key
'-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIRAO8gS0SKikCiuMCRUiKkLYUwDQYJKoZIhvcNAQELBQAw
XDELMAkGA1UEBhMCVVMxDzANBgNVBAgMBkRlbmlhbDEUMBIGA1UEBwwLU3ByaW5n
ZmllbGQxDDAKBgNVBAoMA0RpczEYMBYGA1UEAwwPd3d3LmV4YW1wbGUuY29tMB4X
DTE4MTExNDA2MjY0NVoXDTIwMTExMzA2MjY0NVowLzEtMCsGA1UEAwwkZjQ5M2M5
MDQtZWFhMy00NTljLThjNzctMTU2OGYyOWMwYzI3MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+h
gtLKhMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4K
AB8502uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0
MwvQVfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e
0UAYS/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BV
vzD68Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABo3MwcTAMBgNVHRMB
Af8EAjAAMC8GA1UdEQQoMCaCJGY0OTNjOTA0LWVhYTMtNDU5Yy04Yzc3LTE1Njhm
MjljMGMyNzAOBgNVHQ8BAf8EBAMCA7gwIAYDVR0lAQH/BBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQAgP+yoiW/Otfl9Sx/VBh2Ehw0H
+82rPT7Yvs8UfnOv/AqDiUzOZMbmJQNfwMgzwY9l7yY4h0DMQsTortYHCuOMTaWB
IsPRCCZeFtSxgYnHjW8Ggw1bEHNUWlkazENzA/2fKNnAkj/ddcr26TOADTGi4wB1
VqvU51+UckA+Qn19fDdmlYWhRDhh3ye1wgQNZmIGiBoFuwJpQBBYST9AS+xcr80g
tCgThHgjqTMr0I3K1Dx/zMA+/eTIWb4e8T7jcd0iW7fTFtux5/NVjcTWjcAFvTAd
Z9mNB5DTKrc3V61AybpIyZeI545Had0GM+SdHXNzFI2tCtgTvT/ZEAsqRiRv
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA2spT4jGxt1nrfRwFburC2ErkdPaO3KE15ndoc17Blw+hgtLK
hMhqCGGmWx7h9XsLV98JtglYuJDwh2vI7iToqN22izIhyUfef9j6asgUDs4KAB85
02uPPeVwxmB60KeZQzES2PxgxpngUeMktpMv90/YIyJyEPgeEKj4/41C8Pr0MwvQ
VfJD28O2A/S5GgJWWjSgZA3vvEVflyhe0nnCWdJNwSFSzoR68IAF8RCBOm2e0UAY
S/p9nabvkCPxpO8fhNbqzg/NCwIb+Vi06mC3L0tzyApI+Oks1Jd7uMM4O/BVvzD6
8Me/KbirYr6JEUHYAlIvyQ51ljAC0nWWJviW1wIDAQABAoIBAEAJpE+6V9fgm8p8
nyJ92BXSpdeOKvZswQf5vzq1a1g5nP5bkCcZOd/GJRjaiyx8nS9U+tSrG6q50Yzx
gVgiuW5jpoBLZhQx0u/8pB8I/MXwjIDIovY8rypgs4d8ybW0uGkwPeIAzJqUg1G0
eBRwNEPgvNRbyqMo3DPoISk7QXKilmk/h5em1KmodBU9YtgIqi8F+Rs60csOPo5Z
X70YGq5dICN1xIqW8eqzvbVO4JF/oU7JqhEyfcG6S029Ilj29bAPCRLEMZIUxeP5
cMjGVP59PCt1z42UEnaxLEmzIpD2vVq06xEC3/0BvLLGTY6FOfzXjeEt3i/fdcUq
LiNIejECgYEA/QxXwmPu9c9v4uUqhMcG6AUJd1Hlrfq/gJIaNRGsp/UtK4qtPMgw
6GTv9Sv1D4Vuad65recdrGXYgAPWj1TRbIqimBE0Yu0z5vuAfScu7yrrZsWZXHJO
VIP2310ZPnJFXwRSCA1VpIM8MoKfoAfpAMfDr7PM0gEjDapPpFLsl4UCgYEA3Veu
8fTqqgxCWFe0IRZRiRT5FciRupBvuOVmYMIqIGtSD/l21dOQ0rd5o5ZBZsInO80y
dXWk6YEni6dqSjxYwlaTHfG3ZyZlSEeaSHEhGiLvItxaMh6DJGcbTKUgkDiHFf5K
W/KpJ91C4ByykHZmGYesz/z4Z8vadjPrf8gpLasCgYEAoJHedjlXfp88jiuAyXRJ
i5z2nsJXDgkYz4rmGlq2xnUrTn/W4cTeU/kI0vgrrseqgn+ULyeCisytjr3gvl7B
7TAjcH8qUMPXtXBN3hypCZagfTxRznmx/qsmUiIPTLLSFjL1oqpjd9rWre55P+EF
Fzurjqh3BaM3DQrPMqR0AMkCgYEAkt5BqS7YHulvhGr9jQ7gH1OZS8kAWYjJeShO
XFm51jUgCJWBMrTlXcx8m/1xfBvMKLQpjSL4wDAA63u03XlZc+o6SB5BkeI6RlGs
n/DhBBS2FK2d86+nWRpJVPwktU2s5P0MniJP97GrVEX2fkDx0nLiSkgTE9yCIvik
hO9t020CgYEAxbRcW1jAB9cFeWKo4YxXAQv3DWc3PNYgNTAX8/GvS98DQC8LprtC
jMUPwaDQZBt1gMiPBYyYE8nqlIt6lHTJ8jZxQNIDYkbC8NNzPU26nj5UPfXGo5Io
LYi4xiLXfbAZhTP7M2Bf3sJXAjENtPUh/AHexcBSHF3ZRPxxokxx9aQ=
-----END RSA PRIVATE KEY-----
'
CertComputeCreate
CertComputeCreate 用于创建具有证书的 Amphora 虚拟机。
# file: /opt/rocky/octavia/octavia/controller/worker/tasks/compute_tasks.py
class CertComputeCreate(ComputeCreate):
def execute(self, amphora_id, server_pem,
build_type_priority=constants.LB_CREATE_NORMAL_PRIORITY,
server_group_id=None, ports=None):
"""Create an amphora
:returns: an amphora
"""
# load client certificate
with open(CONF.controller_worker.client_ca, 'r') as client_ca:
ca = client_ca.read()
config_drive_files = {
'/etc/octavia/certs/server.pem': server_pem,
'/etc/octavia/certs/client_ca.pem': ca}
return super(CertComputeCreate, self).execute(
amphora_id, config_drive_files=config_drive_files,
build_type_priority=build_type_priority,
server_group_id=server_group_id, ports=ports)
形参 server_pem 的实参是 GenerateServerPEMTask 的 return，也就是说 config_drive_files 包含了 CA 证书、Amphora 证书、Amphora 私钥三者。最终通过 novaclient 调用 Nova 的 userdata 和 config drive 机制来完成 config_drive_files 文件注入。
# file: /opt/rocky/octavia/octavia/compute/drivers/nova_driver.py
amphora = self.manager.create(
name=name, image=image_id, flavor=amphora_flavor,
key_name=key_name, security_groups=sec_groups,
nics=nics,
files=config_drive_files,
userdata=user_data,
config_drive=True,
scheduler_hints=server_group,
availability_zone=CONF.nova.availability_zone
)
登录到 Amphora Instance 可以查看到这些文件：
ubuntu@amphora-3a980a2f-4954-47cc-bf56-fc206fc50ae3:~$ ll /etc/octavia/certs/
total 16
drwxr-xr-x 2 root root 4096 Nov 5 02:36 ./
drwxr-xr-x 3 root root 4096 Nov 5 02:36 ../
-rw-rw---- 1 root root 1294 Nov 5 02:36 client_ca.pem
-rw-rw---- 1 root root 2964 Nov 5 02:36 server.pem
/etc/octavia/amphora-agent.conf 配置文件的内容如下：
[DEFAULT]
debug = True
[haproxy_amphora]
base_cert_dir = /var/lib/octavia/certs
base_path = /var/lib/octavia
bind_host = ::
bind_port = 9443
haproxy_cmd = /usr/sbin/haproxy
respawn_count = 2
respawn_interval = 2
use_upstart = True
[health_manager]
controller_ip_port_list = 192.168.0.4:5555
heartbeat_interval = 10
heartbeat_key = insecure
[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem
agent_request_read_timeout = 120
amphora_id = a47ce718-1b3e-40a9-9fd9-a6ac5e1deba1
amphora_udp_driver = keepalived_lvs
[health_manager]section 描述了 amphora-agent 如何向 octavia-health-manager 上报监控状态。[amphora_agent]section 中的 agent_server_ca 和 agent_server_cert 在启动 amphora-agent 时候被加载启用 SSL 通信，后文再继续展开。
Amphora Agent
amphora-agent 服务进程启动脚本：
# file: /usr/local/bin/amphora-agent
#!/opt/amphora-agent-venv/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from octavia.cmd.agent import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script.pyw?|.exe)?$', '', sys.argv[0])
sys.exit(main())
amphora-agent 服务进程的入口函数：
# file: /opt/rocky/octavia/octavia/cmd/agent.py
# start api server
def main():
# comment out to improve logging
service.prepare_service(sys.argv)
gmr.TextGuruMeditation.setup_autorun(version)
health_sender_proc = multiproc.Process(name='HM_sender',
target=health_daemon.run_sender,
args=(HM_SENDER_CMD_QUEUE,))
health_sender_proc.daemon = True
health_sender_proc.start()
# Initiate server class
server_instance = server.Server()
bind_ip_port = utils.ip_port_str(CONF.haproxy_amphora.bind_host,
CONF.haproxy_amphora.bind_port)
options = {
'bind': bind_ip_port,
'workers': 1,
'timeout': CONF.amphora_agent.agent_request_read_timeout,
'certfile': CONF.amphora_agent.agent_server_cert,
'ca_certs': CONF.amphora_agent.agent_server_ca,
'cert_reqs': True,
'preload_app': True,
'accesslog': '/var/log/amphora-agent.log',
'errorlog': '/var/log/amphora-agent.log',
'loglevel': 'debug',
}
AmphoraAgent(server_instance.app, options).run()
AmphoraAgent Application Class 继承自 gunicorn.app.base.BaseApplication，通过 options 中的 CONF.amphora_agent.agent_server_cert 与 CONF.amphora_agent.agent_server_ca 加载 Amphora 的证书文件路径并启用 SSL 通信。
实际上 amphora-agent 使用的是 Flask 框架，在生成 Web Application 的过程中也完成了路由函数和视图函数的初始化。
# file: /opt/rocky/octavia/octavia/amphorae/backends/agent/api_server/server.py
class Server(object):
def __init__(self):
self.app = flask.Flask(__name__)
self._osutils = osutils.BaseOS.get_os_util()
self._keepalived = keepalived.Keepalived()
self._listener = listener.Listener()
self._udp_listener = (udp_listener_base.UdpListenerApiServerBase.
get_server_driver())
self._plug = plug.Plug(self._osutils)
self._amphora_info = amphora_info.AmphoraInfo(self._osutils)
register_app_error_handler(self.app)
self.app.add_url_rule(rule=PATH_PREFIX +
'/listeners/<amphora_id>/<listener_id>/haproxy',
view_func=self.upload_haproxy_config,
methods=['PUT'])
...
AmphoraAPIClient
再回过头来看看 AmphoraAPIClient 的实现，AmphoraAPIClient 在建立 REST 连接的 session 时会导入了 create_certificates.sh 创建的 Server 证书，由 CONF.haproxy_amphora.client_cert 和 CONF.haproxy_amphora.server_ca 指定。
class AmphoraAPIClient(object):
def __init__(self):
super(AmphoraAPIClient, self).__init__()
self.secure = False
self.get = functools.partial(self.request, 'get')
self.post = functools.partial(self.request, 'post')
self.put = functools.partial(self.request, 'put')
self.delete = functools.partial(self.request, 'delete')
self.head = functools.partial(self.request, 'head')
self.start_listener = functools.partial(self._action,
consts.AMP_ACTION_START)
self.stop_listener = functools.partial(self._action,
consts.AMP_ACTION_STOP)
self.reload_listener = functools.partial(self._action,
consts.AMP_ACTION_RELOAD)
self.start_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_START)
self.stop_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_STOP)
self.reload_vrrp = functools.partial(self._vrrp_action,
consts.AMP_ACTION_RELOAD)
self.session = requests.Session()
self.session.cert = CONF.haproxy_amphora.client_cert
self.ssl_adapter = CustomHostNameCheckingAdapter()
self.session.mount('https://', self.ssl_adapter)
...
def request(self, method, amp, path='/', timeout_dict=None, **kwargs):
cfg_ha_amp = CONF.haproxy_amphora
if timeout_dict is None:
timeout_dict = {}
req_conn_timeout = timeout_dict.get(
consts.REQ_CONN_TIMEOUT, cfg_ha_amp.rest_request_conn_timeout)
req_read_timeout = timeout_dict.get(
consts.REQ_READ_TIMEOUT, cfg_ha_amp.rest_request_read_timeout)
conn_max_retries = timeout_dict.get(
consts.CONN_MAX_RETRIES, cfg_ha_amp.connection_max_retries)
conn_retry_interval = timeout_dict.get(
consts.CONN_RETRY_INTERVAL, cfg_ha_amp.connection_retry_interval)
LOG.debug("request url %s", path)
_request = getattr(self.session, method.lower())
_url = self._base_url(amp.lb_network_ip) + path
LOG.debug("request url %s", _url)
reqargs = {
'verify': CONF.haproxy_amphora.server_ca,
'url': _url,
'timeout': (req_conn_timeout, req_read_timeout), }
reqargs.update(kwargs)
headers = reqargs.setdefault('headers', {})
...
由于 Server 证书和 Amphora 证书都是由同一个 CA 中心签发的，存在信任链关系，所以 AmphoraAPIClient 也就可以与 Amphora 进行 HTTPS 通信了。
最后
一言以蔽之，Octavia 使用 CA 认证是为了保证 octavia-worker 和 Amphora 的通信安全。我们需要注意的是相关配置项目的意义与证书的作用，通过分析代码实现的方式能够对这个认证配置了解得更加深入。在 R 版本中，octavia-worker 和 Amphora 的通信仍不能称之为非常稳定，所以还是很有必要了解一下前因后果，以便候查。