zoukankan      html  css  js  c++  java
  • 探索 OpenStack 之(15):oslo.messaging 和 Cinder 中 MessageQueue 消息的发送和接收

    前言:上一篇文章 只是 RabbitMQ 的科普,本文将仔细分析 Cinder 中 RabbitMQ 的各组件的使用、消息的发送和接收等。由于各流程步骤很多,本文只会使用若干流程图来加以阐述,尽量做到图文自解释,不会添加很细的文字说明了。

    1. Cinder 中创建卷的端到端过程

    该过程主要包括两部分:

    第一部分即初始化部分:cinder-api 服务启动过程中 (参见另一篇文章),APIRouter 类被初始化,接着它会初始化 VolumeController 类,最终,SchedulerAPI 类以及 VolumeAPI 类会被初始化,它们分别会调用 get_client 方法获取一个 RPCClient 的实例,其中,target 即发送消息的目的 RabbitMQ exchange,它由 cinder.conf 定义。该 RPCClient 实例的方法会分别被 SchedulerAPI 类和 VolumeAPI类来发送。

    第二部分即流程处理部分:当 cinder-api 的 WSGI server 收到一个卷创建的请求后,APIRouter 会负责把该请求分发到 VolumeController 类的 create 方法 (其过程请参考另一篇文章)。该方法会依次:

    (16)生成一个 task flow,其中的几个 task 依次提取request中的数据、预留 Quota、在 db 中创建 entery、提交 quota,最后调用 VolumeCastTask::exectue 方法。

    (17)调用 SchedulerAPI 的 create_volume 方法,它会调用 RPCClient 的 cast 方法来发出一个创建volume 的 message。

    (18 ~ 20)最终 SchedulerManager 的 create_volume 方法会被调用,它会执行 filtering 和 weighting 来选择一个最优的 host。

    (23)确定好该 host 后,调用 volume.API 的 create_volume 方法。

    (24 ~ 25)该方法会调用 RPCClient 的 cast 方法发出一个 volume creation 消息给 cinder-volume

    (26)最终,指定 host 上的 VolumeManager 的 create_volume 方法会被调用,它会调用 cinder driver 来创建 volume。

    该过程显示了 volume 创建过程中,cinder 的三大组件 cinder-api、cinder-scheduler 和 cinder-volume 使用 RabbitMQ 来传递和接收消息来执行操作的过程。

    2. 消息发送机制 (olso.messaging.RPCClient 的 cast 和 call 方法)

    几个要点:

    (0)cast 方法把消息发出去就算了,不需要返回值;call 方法在消息发出后需要等待消息结果返回,因此它在发消息(步骤 27)之前,会创建一个 waiter (步骤26)来等待消息返回。

    (3)get_transport 方法会使用 stevedore 库来加载特定的 messaging driver 类,默认情况下 conf.rpc_backend 的值为 'rabbit', 因此其对应的 driver 类 oslo.messaging._drivers.impl_rabbit.RabbitDriver会被加载;然后生成一个 Transport 实例,该实例实际上只是 driver 实例的一个wrapper。

    (6)初始化 Target 实例,它实际上表明了 message 的目的地,包括消息被发往的地方或者一个应用需要监听的对象。其属性包括:      

    def __init__(self, exchange=None, topic=None, namespace=None, version=None, server=None, fanout=None):
            self.exchange = exchange // MQ 的exchange,使用 cinder.conf 中的定义,默认是 control_exchange : openstack
            self.topic = topic //MQ 的 queue,由 cinder.conf 指定: scheduler_topic = cinder-scheduler,volume_topic = cinder-volume 
            self.namespace = namespace //消息处理类的方法集合,默认是 null
            self.version = version
            self.server = server //消息的特定目的consumer application,是 cinder 里面的一个消息处理类
            self.fanout = fanout //一个flag,设置的话,消息将发给所有queue的consumers

    (15 ~ 21) cast 方法的实现,最终是使用 RabbitMQ 将消息发出去

    (20 ~ 35)call 方法的实现,在第 27 步将消息发出去之前先创建一个 waiter 来等待返回的消息,它等到返回的消息后会把结果传回到caller。

     3. 消息接收和分发机制 (olso.messaging.RPCDispatcher)

    (2~3)cinder-scheduler 和在各节点上的 cinder-volume 作为 service 启动的时候,会初始化并启动一个RPC server,该 server 会监听 message queu并分发消息。

    (8,30)RPCDispatcher 负责为接收到的 PRC message 调用一个 endpoint 类 的 method。

    • endpoint 类:
      • 对 cinder-scheduler 的 rpcapi 发送的消息来说,endpoint 是类 SchedulerManager;
      • 对 cinder-volume 的 rpcapi 发出的消息来说,endpoint 是类 VolumeManager。
    • endpoint 类的方法:被调用的方法名称会在 cast 方法中作为参数传入,比如下面 create_volume 方法 cast 这个消息后,它会分发到 SchedulerManager 的 create_volume 方法。
    //cinder/scheduler/rpcapi.py
    def
    create_volume(self, ctxt, topic, volume_id, snapshot_id=None, image_id=None, request_spec=None, filter_properties=None): cctxt = self.client.prepare(version='1.2') request_spec_p = jsonutils.to_primitive(request_spec) return cctxt.cast(ctxt, 'create_volume', topic=topic, volume_id=volume_id, snapshot_id=snapshot_id, image_id=image_id, request_spec=request_spec_p, filter_properties=filter_properties)

     (11,26 ~ 28)EventletExecutor 会初始化一个大小为 conf.rpc_thread_pool_size 的线程池,它调用 AMQPListener 的 poll 方法获取 message 后,会在一个线程中调用 RPCDispatcher 的 __call__ 方法来调用消息处理方法。

    (20, 26)AMQPListener 会无限循环去获取 message,直到它被停止。获取到的 message 会被交给 EventletExecutor ,它会调用RCPDispatcher的方法将其分发给某个类的某个方法。

    4. Cinder 所用到的 RabbitMQ 组件

    注:有些组件,比如 cinder_scheduler.controller 尚不清楚其用途。

     5. 消息示例

    5.1 每个节点上的 cinder-volume 服务向 cinder-scheduler 服务更新其 capabilities 的消息

    消息发送者 发送频率 MQ exchange 消息接收者类 备注
    每个cinder-volume 节点上的cinder-volume 服务 1 分钟 cinder-scheduler_fanout SchedulerManager::update_service_capabilities cinder-scheduler 的 HostManager 类维护一个数组service_states,每个cinder-volume 节点占该数组的一项,其值为该节点上 cinder-volume 服务的 capabilities,该值通过消息机制定期更新。在发生特定操作比如删除卷时,会进行立刻更新。

    消息内容示例:

    {u'_context_domain': None, u'_context_request_id': u'req-c0a0babc-3fec-41fe-ae6a-45418beec8fc', u'_context_quota_class': None, u'_context_service_catalog': [], u'_context_auth_token': '<SANITIZED>', u'_context_user': None, u'_context_user_id': None, u'_context_is_admin': True, u'version': u'1.0', u'_context_project_domain': None, u'_context_timestamp': u'2015-03-16T10:13:14.676590', u'method': u'update_service_capabilities', u'_context_remote_address': None, u'_context_roles': [u'admin'], u'args': {u'service_name': u'volume', u'host': u'network@lvmdriver-network', u'capabilities': {u'pools': [{u'pool_name': u'lvmbackend', u'QoS_support': False, u'allocated_capacity_gb': 4, u'free_capacity_gb': 0.36, u'location_info': u'LVMVolumeDriver:network:system:default:0', u'total_capacity_gb': 9.06, u'reserved_percentage': 0}], u'driver_version': u'2.0.0', u'vendor_name': u'Open Source', u'volume_backend_name': u'lvmbackend', u'storage_protocol': u'iSCSI'}}, u'_unique_id': u'f18ce7979c3148ce933ba93ee77e4f47', u'_context_project_name': None, u'_context_read_deleted': u'no', u'_context_user_identity': u'- - - - -', u'_context_tenant': None, u'_context_project_id': None, u'_context_user_domain': None}

     5.2 创建卷时 cinder-api 服务发给 cinder-scheduler 服务的消息

    
    
    MQ Exchange: openstack
    
    
    Message body:
    {"oslo.message": "{"_context_domain": null, "_context_request_id": "req-7bb7ff5b-97e5-481a-97d4-c636b2e0687a", "_context_quota_class": null, "_context_service_catalog": [{"endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "id": "055951b654364d918aaa37bbe365906e", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "endpoints_links": [], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...==", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.2", "_context_timestamp": "2015-03-21T07:26:33.337831", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "create_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"request_spec": {"source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "volume_metadata": [], "volume_admin_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "display_description": null, "availability_zone": "az1", "attach_status": "detached", "source_volid": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}, "volume_type": {}, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "source_volid": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc"}, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "filter_properties": {}, "topic": "cinder-volume", "image_id": null, "snapshot_id": null}, "_unique_id": "73f2789be1124ac095909d7e4e542d8d", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}
    Message routing_key: 'cinder-scheduler'
    Endpoint method: SchedulerManager::create_volume

    5.3 创建卷时 cinder-scheduler 服务发给某个节点上 cinder-volume 服务的消息

    MQ Exchange: openstack
    
    
    Message body:
    '{"oslo.message": "{"_context_domain": null, "_context_request_id": "req-7bb7ff5b-97e5-481a-97d4-c636b2e0687a", "_context_quota_class": null, "_context_service_catalog": [{"endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "055951b654364d918aaa37bbe365906e", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "endpoints_links": [], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...==", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.4", "_context_timestamp": "2015-03-21T07:26:33.337831", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "create_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"request_spec": {"source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "availability_zone": "az1", "consistencygroup_id": null, "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "volume_admin_metadata": [], "attach_status": "detached", "display_description": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "volume_metadata": [], "replication_status": "disabled", "snapshot_id": null, "source_volid": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}, "volume_type": {}, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "source_volid": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "resource_properties": {"status": "creating", "volume_type_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "volume_metadata": [], "volume_admin_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "display_description": null, "availability_zone": "az1", "attach_status": "detached", "source_volid": null, "metadata": {}, "source_replicaid": null, "encryption_key_id": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "display_name": null, "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1}}, "source_replicaid": null, "allow_reschedule": true, "filter_properties": {"config_options": {}, "availability_zone": "az1", "request_spec": {"volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "source_replicaid": null, "volume_properties": {"status": "creating", "volume_type_id": null, "display_name": null, "volume_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "availability_zone": "az1", "user_id": "1dc0db32a936496ebfc50be54924a7cc", "attach_status": "detached", "display_description": null, "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "replication_status": "disabled", "snapshot_id": null, "encryption_key_id": null, "source_volid": null, "volume_admin_metadata": [], "source_replicaid": null, "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "consistencygroup_id": null, "size": 1, "metadata": {}}, "volume_type": {}, "image_id": null, "consistencygroup_id": null, "source_volid": null, "snapshot_id": null, "resource_properties": {"status": "creating", "volume_type_id": null, "volume_metadata": [], "reservations": ["881c6dda-bf43-4535-952c-1c30ae24de9e", "b0949e21-f7e6-41ac-a8c1-8314e6b4a13d"], "source_volid": null, "consistencygroup_id": null, "replication_status": "disabled", "snapshot_id": null, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "id": "89dd8904-b893-42db-b158-7f14c13bf1cc", "size": 1, "display_name": null, "source_replicaid": null, "availability_zone": "az1", "attach_status": "detached", "display_description": null, "encryption_key_id": null, "volume_admin_metadata": [], "project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "metadata": {}}}, "qos_specs": null, "size": 1, "retry": {"num_attempts": 1, "hosts": ["block2@lvmdriver-b21#lvmbackend"]}, "user_id": "1dc0db32a936496ebfc50be54924a7cc", "volume_type": {}, "resource_type": {}, "metadata": {}}, "source_volid": null, "image_id": null, "snapshot_id": null, "consistencygroup_id": null, "volume_id": "89dd8904-b893-42db-b158-7f14c13bf1cc"}, "_unique_id": "e730b03f68cb426f906e39c64e265956", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}'
    Message routing_key: 'cinder-volume.block2@lvmdriver-b21'
    Endpoint method: VolumerManager::create_volume

    5.4 删除卷时 cinder-api 服务发给目的节点上 cinder-volume 服务的消息

    MQ Exchange: openstack
    Message body:
    {"oslo.message": "{"_context_domain": null, "_context_request_id": "req-a5297b18-7b87-4ea1-914d-0174790a1ca5", "_context_quota_class": null, "_context_service_catalog": [{"endpoints_links": [], "endpoints": [{"adminURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "region": "regionOne", "publicURL": "http://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6", "id": "055951b654364d918aaa37bbe365906e", "internalURL": "hhttp://controller:8774/v2/43f66bb82e684bbe9eb9ef6892bd7fd6"}], "type": "compute", "name": "nova"}], "_context_auth_token": "PKIZ_...", "_context_user_id": "1dc0db32a936496ebfc50be54924a7cc", "_context_is_admin": true, "version": "1.15", "_context_timestamp": "2015-03-21T07:16:02.443178", "_context_project_domain": null, "_context_user": "1dc0db32a936496ebfc50be54924a7cc", "method": "delete_volume", "_context_remote_address": "192.168.1.20", "_context_roles": ["admin", "_member_"], "args": {"unmanage_only": false, "volume_id": "a8a4bf9b-b233-4bd4-9ec4-d132d3b4b0af"}, "_unique_id": "370304cba81641ee82e8eaf84b519e25", "_context_project_name": "admin", "_context_read_deleted": "no", "_context_user_identity": "1dc0db32a936496ebfc50be54924a7cc 43f66bb82e684bbe9eb9ef6892bd7fd6 - - -", "_context_tenant": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_project_id": "43f66bb82e684bbe9eb9ef6892bd7fd6", "_context_user_domain": null}", "oslo.version": "2.0"}

    Message routing_key: 'cinder-volume.block2@lvmdriver-b21'
    Endpoint method: VolumeManager::delete_volume
  • 相关阅读:
    关于celery踩坑
    关于git的分批提交pull requests流程
    SymGAN—Exploiting Images for Video Recognition: Heterogeneous Feature Augmentation via Symmetric Adversarial Learning学习笔记
    AFN—Larger Norm More Transferable: An Adaptive Feature Norm Approach for Unsupervised Domain Adaptation学习笔记
    Learning to Transfer Examples for Partial Domain Adaptation学习笔记
    Partial Adversarial Domain Adaptation学习笔记
    Partial Transfer Learning with Selective Adversarial Networks学习笔记
    Importance Weighted Adversarial Nets for Partial Domain Adaptation学习笔记
    Exploiting Images for Video Recognition with Hierarchical Generative Adversarial Networks学习笔记
    improved open set domain adaptation with backpropagation 学习笔记
  • 原文地址:https://www.cnblogs.com/sammyliu/p/4335339.html
Copyright © 2011-2022 走看看