目录
前言
Cinder 支持 front-end、back-end 和 both 三种类型的存储 QoS，front-end 由 Hypervisor（QEMU）实现（e.g. 通过 Libvirt 设置虚拟机的存储 QoS），back-end 指由后端 Storage Provider（e.g. 物理存储设备）设置的 QoS。back-end QoS 需要 Storage Provider 的支持。常见的存储设备支持 QoS，例如：EMC，最新版的 Ceph Mimic 的 RBD 也已经支持 IOPS 的 QoS。
QoS 项目：
- total_bytes_sec - the total allowed bandwidth for the guest per second
- read_bytes_sec - sequential read limitation
- write_bytes_sec - sequential write limitation
- total_iops_sec - the total allowed IOPS for the guest per second
- read_iops_sec - random read limitation
- write_iops_sec - random write limitation
在 Rocky 版本，Cinder 开始支持基于容量的 QoS 参数：
Cinder now allows for capacity based QoS which can be useful in environments where storage performance scales with consumption (such as RBD backed storage). The newly added QoS specs are read_iops_sec_per_gb, write_iops_sec_per_gb, total_iops_sec_per_gb, read_bytes_sec_per_gb, write_bytes_sec_per_gb and total_bytes_sec_per_gb. These values will be multiplied by the size of the volume and passed to the consumer. For example, setting total_iops_sec_per_gb to 30 and setting total_bytes_sec_per_gb to 1048576 (1MB) then creating a 100 GB volume with that QoS will result in a volume with 3,000 total IOPs and 100MB/s throughput limit.
Cinder now allows for a minimum value when using the capacity based QoS in order to make sure small volumes can get a minimum allocation for them to be usable. The newly added QoS specs are read_iops_sec_per_gb_min, write_iops_sec_per_gb_min, total_iops_sec_per_gb_min, read_bytes_sec_per_gb_min, write_bytes_sec_per_gb_min and total_bytes_sec_per_gb_min
操作步骤
创建具有 QoS Policy 的 Volume Type 并创建 Volume：
[root@controller ~]# openstack volume type create frontend_qos
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| description | None |
| id | 139a3dc3-345d-47fc-8b05-499c4757f6e2 |
| is_public | True |
| name | frontend_qos |
+-------------+--------------------------------------+
[root@controller ~]# openstack volume qos create qos1 --consumer front-end --property total_iops_sec=200 --property total_bytes_sec=2048000
+------------+-------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------+
| consumer | front-end |
| id | b6d79c1a-9c6b-4a8a-8d8d-b0106232deed |
| name | qos1 |
| properties | total_bytes_sec='2048000', total_iops_sec='200' |
+------------+-------------------------------------------------+
[root@controller ~]# openstack volume qos associate qos1 frontend_qos
[root@controller ~]# openstack volume create --type frontend_qos --size 1 volume01
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-04-29T01:46:32.000000 |
| description | None |
| encrypted | False |
| id | bd6013bf-1dcd-4b64-9731-05528624d28e |
| migration_status | None |
| multiattach | False |
| name | volume01 |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | frontend_qos |
| updated_at | None |
| user_id | 92602c24daa24f019f05ecb95f1ce68e |
+---------------------+--------------------------------------+NOTE：一个 VolumeType 只能 associated 一个 QoS Policy。e.g.
[root@controller ~]# openstack volume qos create qos2 --consumer back-end --property total_iops_sec=200 --property total_bytes_sec=2048000
+------------+-------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------+
| consumer | back-end |
| id | 8622845e-188d-4cc4-8aef-0ac5f701dff9 |
| name | qos2 |
| properties | total_bytes_sec='2048000', total_iops_sec='200' |
+------------+-------------------------------------------------+
[root@controller ~]# openstack volume qos associate qos2 frontend_qos
Invalid volume type: Type 139a3dc3-345d-47fc-8b05-499c4757f6e2 is already associated with another qos specs: b6d79c1a-9c6b-4a8a-8d8d-b0106232deed (HTTP 400) (Request-ID: req-6cd926cc-27b7-422f-b2e2-1d5a2df67d77)将 Volume 挂载到 Libvirt 虚拟机并查看 XML 中的 QoS 参数：
[root@controller ~]# openstack server show VM1
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute |
| OS-EXT-SRV-ATTR:instance_name | instance-00000029 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-04-29T01:48:55.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | vlan-net-100=192.168.1.7 |
| config_drive | |
| created | 2019-04-29T01:48:40Z |
| flavor | mini (66ddc38d-452a-40b6-a0f3-f867658754ff) |
| hostId | 489693032f8676c0ce48995cffce9c4e00bd0b10739e5c0ca33f8559 |
| id | 1125f797-f4e3-4f7f-a511-e445b4e27135 |
| image | cirros_raw (d18923bd-86fc-4f77-b5e8-976d3b1c367c) |
| key_name | None |
| name | VM1 |
| progress | 0 |
| project_id | a2b55e37121042a1862275a9bc9b0223 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2019-04-29T01:48:55Z |
| user_id | 92602c24daa24f019f05ecb95f1ce68e |
| volumes_attached | |
+-------------------------------------+----------------------------------------------------------+
[root@controller ~]# openstack server add volume VM1 volume01
[root@controller ~]# openstack volume show volume01
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'1125f797-f4e3-4f7f-a511-e445b4e27135', u'attachment_id': u'07beff11-cc81-4fa6-9eda-f51590539044', u'attached_at': u'2019-04-29T01:50:46.000000', u'host_name': u'compute', u'volume_id': u'bd6013bf-1dcd-4b64-9731-05528624d28e', u'device': u'/dev/vdb', u'id': u'bd6013bf-1dcd-4b64-9731-05528624d28e'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-04-29T01:46:32.000000 |
| description | None |
| encrypted | False |
| id | bd6013bf-1dcd-4b64-9731-05528624d28e |
| migration_status | None |
| multiattach | False |
| name | volume01 |
| os-vol-host-attr:host | controller@lvm#lvm |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | a2b55e37121042a1862275a9bc9b0223 |
| properties | |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | frontend_qos |
| updated_at | 2019-04-29T01:50:52.000000 |
| user_id | 92602c24daa24f019f05ecb95f1ce68e |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@compute ~]# virsh dumpxml instance-00000029
...
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<auth username='cinder'>
<secret type='ceph' uuid='4810c760-dc42-4e5f-9d41-7346db7d7da2'/>
</auth>
<source protocol='rbd' name='vms/1125f797-f4e3-4f7f-a511-e445b4e27135_disk'>
<host name='172.18.22.234' port='6789'/>
<host name='172.18.22.235' port='6789'/>
<host name='172.18.22.236' port='6789'/>
</source>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/sdb'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<iotune>
<total_bytes_sec>2048000</total_bytes_sec>
<total_iops_sec>200</total_iops_sec>
</iotune>
<serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>NOTE：Cinder 暂时是不支持动态 QoS 的，所以即便我们修改了 QoS Policy 的数值也不能马上被应用到 Libvirt 虚拟机。
[root@controller ~]# openstack volume qos list
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
| ID | Name | Consumer | Associations | Properties |
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
| b6d79c1a-9c6b-4a8a-8d8d-b0106232deed | qos1 | front-end | frontend_qos | total_bytes_sec='2048000', total_iops_sec='200' |
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
[root@controller ~]# openstack volume qos set --property total_bytes_sec="1024000" qos1
[root@controller ~]# openstack volume qos list
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
| ID | Name | Consumer | Associations | Properties |
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
| b6d79c1a-9c6b-4a8a-8d8d-b0106232deed | qos1 | front-end | frontend_qos | total_bytes_sec='1024000', total_iops_sec='200' |
+--------------------------------------+------+-----------+--------------+-------------------------------------------------+
[root@compute ~]# virsh dumpxml instance-00000029
...
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/sdb'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<iotune>
<total_bytes_sec>2048000</total_bytes_sec>
<total_iops_sec>200</total_iops_sec>
</iotune>
<serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>但可以直接通过 virsh 指令来设定 Libvirt 虚拟机的 QoS 属性：
[root@compute ~]# virsh blkdeviotune instance-00000029 vdb --total-bytes-sec 1024000
[root@compute ~]# virsh blkdeviotune instance-00000029 vdb --total_iops_sec 100
[root@compute ~]# virsh dumpxml instance-00000029
...
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/sdb'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<iotune>
<total_bytes_sec>1024000</total_bytes_sec>
<total_iops_sec>100</total_iops_sec>
</iotune>
<serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>NOTE：Qos 一般来说是在一个建议值上下浮动的，对于严格要求的 QoS 还可以通过指定 --total-iops-sec-max 来固定上限。
参考文章
https://docs.openstack.org/cinder/latest/admin/blockstorage-capacity-based-qos.html
相关阅读：