cinder侧挂载卷流程分析

cinder侧挂载卷流程分析，存储类型以lvm+iscsi的方式为分析基础
cinder侧主要调用了三个接口
1)reserve_volume: 把volume的状态改为attaching，阻止其它节点执行挂载操作。
2)initialize_connection: 这个方法负责构建和返回nova调用者需要的所有信息。返回的信息中包括CHAP credential, target-iqn 和lun 信息。
3)attach_volume: 把volume状态改为in-use，挂载成功,并创建对应的attach记录。

1、nova侧调用cinder的reserve_volume方法

nova/volume/cinder.py
    @translate_volume_exception
    def reserve_volume(self, context, volume_id):
        cinderclient(context).volumes.reserve(volume_id)

1)cinderclient端接受到nova发送的reserve操作的http请求，其入口处理函数为

cinder/api/contrib/volume_actions.py:VolumeActionsController
    @wsgi.action('os-reserve')
    def _reserve(self, req, id, body):
        """Mark volume as reserved."""
        context = req.environ['cinder.context']
        # Not found exception will be handled at the wsgi level
        volume = self.volume_api.get(context, id)

        self.volume_api.reserve_volume(context, volume)
        return webob.Response(status_int=http_client.ACCEPTED)

该函数的主要作用是通过volume 的uuid，获取volume实例信息，并调用volume目录下的api模块

2)进一步调用cinder volume的api模块的reserve_volume函数，进行数据库的操作，更新卷的状态为attaching
该函数的主要作用是检查指定的卷是否为available,如果卷的状态是available，更新cinder数据库，把卷的状态标记为attaching来预留这个卷，防止其他api在别的地方使用这个卷
对于支持多路挂载的卷，有效状态包括in-use

def reserve_volume(self, context, volume):
    expected = {'multiattach': volume.multiattach,
                'status': (('available', 'in-use') if volume.multiattach
                           else 'available')}

    result = volume.conditional_update({'status': 'attaching'}, expected)

    if not result:
        expected_status = utils.build_or_str(expected['status'])
        msg = _('Volume status must be %(expected)s to reserve, but the '
                'status is %(current)s.') % {'expected': expected_status,
                                             'current': volume.status}
        LOG.error(msg)
        raise exception.InvalidVolume(reason=msg)

    LOG.info(_LI("Reserve volume completed successfully."),
             resource=volume)

2、nova侧向cinder发送initialize_connection请求，请求获取卷的所有连接信息

nova/virt/block_device.py:DriverVolumeBlockDevice
    def attach(self, context, instance, volume_api, virt_driver,
               do_check_attach=True, do_driver_attach=False, **kwargs):
        volume = volume_api.get(context, self.volume_id)
        if do_check_attach:
            volume_api.check_attach(context, volume, instance=instance)

        volume_id = volume['id']
        context = context.elevated()

        connector = virt_driver.get_volume_connector(instance)
        connection_info = volume_api.initialize_connection(context,
                                                           volume_id,
                                                           connector)
        if 'serial' not in connection_info:
            connection_info['serial'] = self.volume_id
        self._preserve_multipath_id(connection_info)
        ........
    

1)cinderclient接受nova发送过来的os-initialize_connection请求

@wsgi.action('os-initialize_connection')
def _initialize_connection(self, req, id, body):
    """Initialize volume attachment."""
    context = req.environ['cinder.context']
    # Not found exception will be handled at the wsgi level
    volume = self.volume_api.get(context, id)
    try:
        connector = body['os-initialize_connection']['connector']
    except KeyError:
        raise webob.exc.HTTPBadRequest(
            explanation=_("Must specify 'connector'"))
    try:
        info = self.volume_api.initialize_connection(context,volume,connector)    
        ....

2)进一步调用volume目录下的api模块的initialize_connection函数,对该请求进行处理

cinder/volume/api.py:API类
    @wrap_check_policy
    def initialize_connection(self, context, volume, connector):
        if volume.status == 'maintenance':
            LOG.info(_LI('Unable to initialize the connection for '
                         'volume, because it is in '
                         'maintenance.'), resource=volume)
            msg = _("The volume connection cannot be initialized in "
                    "maintenance mode.")
            raise exception.InvalidVolume(reason=msg)
        init_results = self.volume_rpcapi.initialize_connection(context,
                                                                volume,
                                                                connector)
        LOG.info(_LI("Initialize volume connection completed successfully."),
                 resource=volume)
        return init_results

3)cinder api进一步发送RPC请求给volume所在的cinder-volume服务节点，最终在cinder-volume节点，
由cinder/volume/manager.py:VolumeManager的initialize_connection处理，该函数的处理，主要包括如下内容

   def initialize_connection(self, context, volume, connector):
         ....
           utils.require_driver_initialized(self.driver)
            step 1: self.driver.validate_connector(connector)
            step 2: model_update = self.driver.create_export(context.elevated(),volume, connector)
            step 3: volume.update(model_update)
            setp 4: conn_info = self.driver.initialize_connection(volume, connector)
        return conn_info

step 1:
对于LVM + iSCSI方式，validate_connector就是检查有没有initiator字段，即nova-compute节点的initiator信息
代码跳转过程如下：drivers/lvm.py -> targets/lio.py -> targets/iscsi.py。

cinder/volume/targets/iscsi.py:ISCSITarget
def validate_connector(self, connector):
    # NOTE(jdg): api passes in connector which is initiator info
    if 'initiator' not in connector:
        err_msg = (_LE('The volume driver requires the iSCSI initiator '
                       'name in the connector.'))
        LOG.error(err_msg)
        raise exception.InvalidConnectorException(missing='initiator')
    return True

step 2 :调用cinder-rtstool工具创建target，并把卷volume添加到target中创建出lun,认证信息。

def create_export(self, context, volume, connector, vg=None):
    if vg is None:
        vg = self.configuration.volume_group
    volume_path = "/dev/%s/%s" % (vg, volume['name'])
    export_info = self.target_driver.create_export(
        context,
        volume,
        volume_path)
    return {'provider_location': export_info['location'],
            'provider_auth': export_info['auth'], }

最终调用的是cinder/volume/targets/iscsi.py:ISCSITarget类

    def create_export(self, context, volume, volume_path):
        """Creates an export for a logical volume."""
        # 'iscsi_name': 'iqn.2010-10.org.openstack:volume-00000001'
        iscsi_name = "%s%s" % (self.configuration.iscsi_target_prefix,------设置iscsi name,形式为iqn.2010-10.org.openstack:volume-uuid
                               volume['name'])
        iscsi_target, lun = self._get_target_and_lun(context, volume)---返回target，和lun的编号，值为(0,0)

        # Verify we haven't setup a CHAP creds file already
        # if DNE no big deal, we'll just create it
        chap_auth = self._get_target_chap_auth(context, volume)------从数据库volumes表中，读取该卷的provider_auth字段，获取认证信息，若没有，则创建
        if not chap_auth:
            chap_auth = (vutils.generate_username(),-----创建auth认证信息
                         vutils.generate_password())

        # Get portals ips and port
        portals_config = self._get_portals_config()-------获取portals配置，该函数返回的字典格式如下 {'portals_ips': portals_ips,'portals_port': self.configuration.iscsi_port}

        # NOTE(jdg): For TgtAdm case iscsi_name is the ONLY param we need
        # should clean this all up at some point in the future
        tid = self.create_iscsi_target(iscsi_name,-----------------创建target
                                       iscsi_target,
                                       lun,
                                       volume_path,
                                       chap_auth,
                                       **portals_config)
        data = {}
        data['location'] = self._iscsi_location(
            self.configuration.iscsi_ip_address, tid, iscsi_name, lun,
            self.configuration.iscsi_secondary_ip_addresses)
        LOG.debug('Set provider_location to: %s', data['location'])
        data['auth'] = self._iscsi_authentication(
            'CHAP', *chap_auth)
        return data

创建target操作，调用的是cinder/volume/targets/lio.py中的create_iscsi_target方法
这个函数下发的参数为
name：iqn.2010-10.org.openstack:volume-uuid
tid:0
lun:0
path:卷的路径
chap_auth:{username,passowrd}
kwargs:{'portals_ips': portals_ips,存储服务器的ip
'portals_port': self.configuration.iscsi_port,一般是3260
}

def create_iscsi_target(self, name, tid, lun, path,chap_auth=None, **kwargs):
    # tid and lun are not used
    vol_id = name.split(':')[1]
    LOG.info(_LI('Creating iscsi_target for volume: %s'), vol_id)
    chap_auth_userid = ""
    chap_auth_password = ""
    if chap_auth is not None:
        (chap_auth_userid, chap_auth_password) = chap_auth

    optional_args = []
    if 'portals_port' in kwargs:
        optional_args.append('-p%s' % kwargs['portals_port'])

    if 'portals_ips' in kwargs:
        optional_args.append('-a' + ','.join(kwargs['portals_ips']))

    try:
        command_args = ['cinder-rtstool',
                        'create',
                        path,
                        name,
                        chap_auth_userid,
                        chap_auth_password,
                        self.iscsi_protocol == 'iser'] + optional_args
        self._execute(*command_args, run_as_root=True)
    except putils.ProcessExecutionError:
        LOG.exception(_LE("Failed to create iscsi target for volume "
                          "id:%s."), vol_id)

        raise exception.ISCSITargetCreateFailed(volume_id=vol_id)

    iqn = '%s%s' % (self.iscsi_target_prefix, vol_id)
    tid = self._get_target(iqn)
    if tid is None:
        LOG.error(_LE("Failed to create iscsi target for volume "
                      "id:%s."), vol_id)
        raise exception.NotFound()

    # We make changes persistent
    self._persist_configuration(vol_id)
    return tid

def _iscsi_location(self, ip, target, iqn, lun=None, ip_secondary=None):
    ip_secondary = ip_secondary or []
    port = self.configuration.iscsi_port
    portals = map(lambda x: "%s:%s" % (x, port), [ip] + ip_secondary)
    return ("%(portals)s,%(target)s %(iqn)s %(lun)s"
            % ({'portals': ";".join(portals),
                'target': target, 'iqn': iqn, 'lun': lun}))

step 3:创建完target以后，更新cinder数据库volumes表中，该volume的provider_location，provider_auth两个字段的值

step 4：调用cinder-rtstool的add-initiator子命令，把计算节点的initiator增加到刚刚创建的target acls中,并把所有的信息拼装返回给nova使用。

cinder/volume/targets/lio.py：
   def initialize_connection(self, volume, connector):
        volume_iqn = volume['provider_location'].split(' ')[1]
        (auth_method, auth_user, auth_pass) = 
            volume['provider_auth'].split(' ', 3)
        # Add initiator iqns to target ACL
        try:
            self._execute('cinder-rtstool', 'add-initiator',
                          volume_iqn,
                          auth_user,
                          auth_pass,
                          connector['initiator'],
                          run_as_root=True)
        ......
        return super(LioAdm, self).initialize_connection(volume, connector)

3、nova给cinderclient发送attach_volume命令，更改cinder数据库中，volume状态

nova/virt/block_device.py:API
    @translate_volume_exception
    def attach(self, context, volume_id, instance_uuid, mountpoint, mode='rw'):
        cinderclient(context).volumes.attach(volume_id, instance_uuid,
                                             mountpoint, mode=mode)

1)cinder侧接受nova更新cinder数据库的入口函数

cinder/api/contrib/volume_actions.py
    @wsgi.action('os-attach')
    def _attach(self, req, id, body):
            .....
            self.volume_api.attach(context, volume,instance_uuid, host_name, mountpoint, mode)
            ....

2)最后cinder-api通过RPC请求到cinder-volume节点，更新数据库，把volume状态改为in-use，并创建对应的attach记录。

cinder/volume/manager.py:VolumeManager
    def attach_volume(self, context, volume_id, instance_uuid, host_name,
                      mountpoint, mode, volume=None):
        """Updates db to show volume is attached.""
   ......
     attachment = volume.begin_attach(mode
    ......