解决删除Volume报错的问题

很久没有遇到过删除Volume出错使得Volume处于Error_Deleting状态的情况了，昨天删除一个Volume时又出现了这个问题，这里顺便把解决方法记录一下。 注意我这里针对的是后端采用iscsi方式的，具体到我这里是tgt+lvm方式。

原因

目前我所遇到的删除Volume出错的原因只有一个“设备正忙”，如果你查看Volume所在的存储节点的日志就会看如下类似的内容

 [-] Exception during message handling
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
     rval = self.proxy.dispatch(ctxt, version, method, **args)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
     return getattr(proxyobj, method)(ctxt, **kwargs)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 206, in delete_volume
     {'status': 'error_deleting'})
   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
     self.gen.next()
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 195, in delete_volume
     self.driver.delete_volume(volume_ref)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 203, in delete_volume
     self._delete_volume(volume, volume['size'])
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 155, in _delete_volume
     run_as_root=True)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 98, in _try_execute
     self._execute(*command, **kwargs)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/utils.py", line 187, in execute
     cmd=' '.join(cmd))
 ProcessExecutionError: Unexpected error while running command.
 Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf dmsetup remove -f /dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1
 Exit code: 1
 Stdout: ''
 Stderr: 'device-mapper: remove ioctl failed: Device or resource busy
Command failed
'

那设备为什么会忙呢？首先我们在Volume所在的存储节点通过tgtadm --lld iscsi --mode target --op show 命令来查看我们要删除的Volume的信息如下

Target 7: iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 24
            Initiator: iqn.1994-05.com.redhat:351f5adb9c85
            Connection: 0
                IP Address: 10.61.2.5
    LUN information:
    ......

原来是我们有一个计算节点到这个Volume的链接没有释放。这里我要先说下iscsi+lvm的大致的工作方式，创建Volume时先在配置项指定的VG中创建一个LV大小Volume的大小， 然后通过tgtadm这个管理命令创建一个iscsi Target并将所创建的LV作为Target的后端存储。给实例挂载时，实例所在的计算节点调用iscsiadm命令链接到Volume对应的Target上 然后底层虚拟化软件再把这个Target指定给实例用。卸载Volume的时候就是底层虚拟化软件先从实例中移除这个Target，然后计算节点再释放它。所以如果有计算节点有链接由于某种 原因没有释放Target的话就会出现上面的情况，当时一般情况是不会出现这种问题的，但是我发现如果一个挂载有Volume的实例从一个节点迁移到另一个节点，源节点到Target的链接 并不会释放，也就是同时有两个节点链接到Target上，然后就会出现上面的问题。

解决方法

过程和原因清楚后，解决起来也就简单了，释放链接，重置Volume的状态，然后删除。首先我们登录计算节点用iscsiadm命令释放链接

[root@stack5 ~]# iscsiadm -m node -T iqn.2010-10.org.openstack:volume-c1374407-d1b3-407f-bbcc-7416756071c1 -u

修改数据库，重置Volume的状态

[root@stack5 ~]# mysql -h 10.61.2.12 -u cinder -p cinder -e "update volumes set status ='available' where id = 'c1374407-d1b3-407f-bbcc-7416756071c1'"

以前这样操作后就能正常删除Volume了，但这次竟然还报错了，看来得上绝招了

 Clear capabilities
 Removing volume: c1374407-d1b3-407f-bbcc-7416756071c1
 [-] Exception during message handling
 Traceback (most recent call last):
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/amqp.py", line 276, in _process_data
     rval = self.proxy.dispatch(ctxt, version, method, **args)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/openstack/common/rpc/dispatcher.py", line 145, in dispatch
     return getattr(proxyobj, method)(ctxt, **kwargs)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 206, in delete_volume
     {'status': 'error_deleting'})
   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
     self.gen.next()
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/manager.py", line 193, in delete_volume
     self.driver.remove_export(context, volume_ref)
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/driver.py", line 474, in remove_export
     self.tgtadm.remove_iscsi_target(iscsi_target, 0, volume['id'])
   File "/usr/lib/python2.6/site-packages/cinder-2012.2.1-py2.6.egg/cinder/volume/iscsi.py", line 168, in remove_iscsi_target
     raise exception.ISCSITargetRemoveFailed(volume_id=vol_id)
 ISCSITargetRemoveFailed: Failed to remove iscsi target for volume c1374407-d1b3-407f-bbcc-7416756071c1.

解决方法二

这个方法就比较暴力了，直接手动搞了。更新数据库标记Volume为已删除，在存储节点用tgtadm命令删除对应的Target，清空对应LV的数据，用lvremove命令删除LV，搞完收工。

[root@store2 ~]# mysql -h 10.61.2.12 -u cinder -p cinder -e "update volumes set deleted = 1, deleted_at = now() where id = 'c1374407-d1b3-407f-bbcc-7416756071c1'"
[root@store2 ~]# tgtadm --lld iscsi --mode target --op delete --tid 7
[root@store2 ~]# dd if=/dev/zero of=/dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1 bs=1M
[root@store2 ~]# lvremove /dev/mapper/nova--volumes-volume--c1374407--d1b3--407f--bbcc--7416756071c1

北方工业大学 | 云计算研究中心 | 姜永