zoukankan      html  css  js  c++  java
  • Cinder 的 I/O QoS

    目录

    前言

    Cinder 支持 front-end、back-end 和 both 三种类型的存储 QoS,front-end 由 Hypervisor(QEMU)实现(e.g. 通过 Libvirt 设置虚拟机的存储 QoS),back-end 指由后端 Storage Provider(e.g. 物理存储设备)设置的 QoS。back-end QoS 需要 Storage Provider 的支持。常见的存储设备支持 QoS,例如:EMC,最新版的 Ceph Mimic 的 RBD 也已经支持 IOPS 的 QoS。

    QoS 项目

    • total_bytes_sec - the total allowed bandwidth for the guest per second
    • read_bytes_sec - sequential read limitation
    • write_bytes_sec - sequential write limitation
    • total_iops_sec - the total allowed IOPS for the guest per second
    • read_iops_sec - random read limitation
    • write_iops_sec - random write limitation

    在 Rocky 版本,Cinder 开始支持基于容量的 QoS 参数:

    Cinder now allows for capacity based QoS which can be useful in environments where storage performance scales with consumption (such as RBD backed storage). The newly added QoS specs are read_iops_sec_per_gb, write_iops_sec_per_gb, total_iops_sec_per_gb, read_bytes_sec_per_gb, write_bytes_sec_per_gb and total_bytes_sec_per_gb. These values will be multiplied by the size of the volume and passed to the consumer. For example, setting total_iops_sec_per_gb to 30 and setting total_bytes_sec_per_gb to 1048576 (1MB) then creating a 100 GB volume with that QoS will result in a volume with 3,000 total IOPs and 100MB/s throughput limit.

    Cinder now allows for a minimum value when using the capacity based QoS in order to make sure small volumes can get a minimum allocation for them to be usable. The newly added QoS specs are read_iops_sec_per_gb_min, write_iops_sec_per_gb_min, total_iops_sec_per_gb_min, read_bytes_sec_per_gb_min, write_bytes_sec_per_gb_min and total_bytes_sec_per_gb_min

    操作步骤

    创建具有 QoS Policy 的 Volume Type 并创建 Volume

    [root@controller ~]# openstack volume type create frontend_qos
    +-------------+--------------------------------------+
    | Field       | Value                                |
    +-------------+--------------------------------------+
    | description | None                                 |
    | id          | 139a3dc3-345d-47fc-8b05-499c4757f6e2 |
    | is_public   | True                                 |
    | name        | frontend_qos                         |
    +-------------+--------------------------------------+
    
    [root@controller ~]# openstack volume qos create qos1 --consumer front-end --property total_iops_sec=200 --property total_bytes_sec=2048000
    +------------+-------------------------------------------------+
    | Field      | Value                                           |
    +------------+-------------------------------------------------+
    | consumer   | front-end                                       |
    | id         | b6d79c1a-9c6b-4a8a-8d8d-b0106232deed            |
    | name       | qos1                                            |
    | properties | total_bytes_sec='2048000', total_iops_sec='200' |
    +------------+-------------------------------------------------+
    
    [root@controller ~]# openstack volume qos associate qos1 frontend_qos
    
    [root@controller ~]# openstack volume create --type frontend_qos --size 1 volume01
    +---------------------+--------------------------------------+
    | Field               | Value                                |
    +---------------------+--------------------------------------+
    | attachments         | []                                   |
    | availability_zone   | nova                                 |
    | bootable            | false                                |
    | consistencygroup_id | None                                 |
    | created_at          | 2019-04-29T01:46:32.000000           |
    | description         | None                                 |
    | encrypted           | False                                |
    | id                  | bd6013bf-1dcd-4b64-9731-05528624d28e |
    | migration_status    | None                                 |
    | multiattach         | False                                |
    | name                | volume01                             |
    | properties          |                                      |
    | replication_status  | None                                 |
    | size                | 1                                    |
    | snapshot_id         | None                                 |
    | source_volid        | None                                 |
    | status              | creating                             |
    | type                | frontend_qos                         |
    | updated_at          | None                                 |
    | user_id             | 92602c24daa24f019f05ecb95f1ce68e     |
    +---------------------+--------------------------------------+

    NOTE:一个 VolumeType 只能 associated 一个 QoS Policy。e.g.

    [root@controller ~]# openstack volume qos create qos2 --consumer back-end --property total_iops_sec=200 --property total_bytes_sec=2048000
    +------------+-------------------------------------------------+
    | Field      | Value                                           |
    +------------+-------------------------------------------------+
    | consumer   | back-end                                        |
    | id         | 8622845e-188d-4cc4-8aef-0ac5f701dff9            |
    | name       | qos2                                            |
    | properties | total_bytes_sec='2048000', total_iops_sec='200' |
    +------------+-------------------------------------------------+
    
    [root@controller ~]# openstack volume qos associate qos2 frontend_qos
    Invalid volume type: Type 139a3dc3-345d-47fc-8b05-499c4757f6e2 is already associated with another qos specs: b6d79c1a-9c6b-4a8a-8d8d-b0106232deed (HTTP 400) (Request-ID: req-6cd926cc-27b7-422f-b2e2-1d5a2df67d77)

    将 Volume 挂载到 Libvirt 虚拟机并查看 XML 中的 QoS 参数:

    [root@controller ~]# openstack server show VM1
    +-------------------------------------+----------------------------------------------------------+
    | Field                               | Value                                                    |
    +-------------------------------------+----------------------------------------------------------+
    | OS-DCF:diskConfig                   | MANUAL                                                   |
    | OS-EXT-AZ:availability_zone         | nova                                                     |
    | OS-EXT-SRV-ATTR:host                | compute                                                  |
    | OS-EXT-SRV-ATTR:hypervisor_hostname | compute                                                  |
    | OS-EXT-SRV-ATTR:instance_name       | instance-00000029                                        |
    | OS-EXT-STS:power_state              | Running                                                  |
    | OS-EXT-STS:task_state               | None                                                     |
    | OS-EXT-STS:vm_state                 | active                                                   |
    | OS-SRV-USG:launched_at              | 2019-04-29T01:48:55.000000                               |
    | OS-SRV-USG:terminated_at            | None                                                     |
    | accessIPv4                          |                                                          |
    | accessIPv6                          |                                                          |
    | addresses                           | vlan-net-100=192.168.1.7                                 |
    | config_drive                        |                                                          |
    | created                             | 2019-04-29T01:48:40Z                                     |
    | flavor                              | mini (66ddc38d-452a-40b6-a0f3-f867658754ff)              |
    | hostId                              | 489693032f8676c0ce48995cffce9c4e00bd0b10739e5c0ca33f8559 |
    | id                                  | 1125f797-f4e3-4f7f-a511-e445b4e27135                     |
    | image                               | cirros_raw (d18923bd-86fc-4f77-b5e8-976d3b1c367c)        |
    | key_name                            | None                                                     |
    | name                                | VM1                                                      |
    | progress                            | 0                                                        |
    | project_id                          | a2b55e37121042a1862275a9bc9b0223                         |
    | properties                          |                                                          |
    | security_groups                     | name='default'                                           |
    | status                              | ACTIVE                                                   |
    | updated                             | 2019-04-29T01:48:55Z                                     |
    | user_id                             | 92602c24daa24f019f05ecb95f1ce68e                         |
    | volumes_attached                    |                                                          |
    +-------------------------------------+----------------------------------------------------------+
    
    [root@controller ~]# openstack server add volume VM1 volume01
    
    [root@controller ~]# openstack volume show volume01
    +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Field                          | Value                                                                                                                                                                                                                                                                                                                        |
    +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | attachments                    | [{u'server_id': u'1125f797-f4e3-4f7f-a511-e445b4e27135', u'attachment_id': u'07beff11-cc81-4fa6-9eda-f51590539044', u'attached_at': u'2019-04-29T01:50:46.000000', u'host_name': u'compute', u'volume_id': u'bd6013bf-1dcd-4b64-9731-05528624d28e', u'device': u'/dev/vdb', u'id': u'bd6013bf-1dcd-4b64-9731-05528624d28e'}] |
    | availability_zone              | nova                                                                                                                                                                                                                                                                                                                         |
    | bootable                       | false                                                                                                                                                                                                                                                                                                                        |
    | consistencygroup_id            | None                                                                                                                                                                                                                                                                                                                         |
    | created_at                     | 2019-04-29T01:46:32.000000                                                                                                                                                                                                                                                                                                   |
    | description                    | None                                                                                                                                                                                                                                                                                                                         |
    | encrypted                      | False                                                                                                                                                                                                                                                                                                                        |
    | id                             | bd6013bf-1dcd-4b64-9731-05528624d28e                                                                                                                                                                                                                                                                                         |
    | migration_status               | None                                                                                                                                                                                                                                                                                                                         |
    | multiattach                    | False                                                                                                                                                                                                                                                                                                                        |
    | name                           | volume01                                                                                                                                                                                                                                                                                                                     |
    | os-vol-host-attr:host          | controller@lvm#lvm                                                                                                                                                                                                                                                                                                           |
    | os-vol-mig-status-attr:migstat | None                                                                                                                                                                                                                                                                                                                         |
    | os-vol-mig-status-attr:name_id | None                                                                                                                                                                                                                                                                                                                         |
    | os-vol-tenant-attr:tenant_id   | a2b55e37121042a1862275a9bc9b0223                                                                                                                                                                                                                                                                                             |
    | properties                     |                                                                                                                                                                                                                                                                                                                              |
    | replication_status             | None                                                                                                                                                                                                                                                                                                                         |
    | size                           | 1                                                                                                                                                                                                                                                                                                                            |
    | snapshot_id                    | None                                                                                                                                                                                                                                                                                                                         |
    | source_volid                   | None                                                                                                                                                                                                                                                                                                                         |
    | status                         | in-use                                                                                                                                                                                                                                                                                                                       |
    | type                           | frontend_qos                                                                                                                                                                                                                                                                                                                 |
    | updated_at                     | 2019-04-29T01:50:52.000000                                                                                                                                                                                                                                                                                                   |
    | user_id                        | 92602c24daa24f019f05ecb95f1ce68e                                                                                                                                                                                                                                                                                             |
    +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    
    [root@compute ~]# virsh dumpxml instance-00000029
    ...
        <disk type='network' device='disk'>
          <driver name='qemu' type='raw' cache='writeback'/>
          <auth username='cinder'>
            <secret type='ceph' uuid='4810c760-dc42-4e5f-9d41-7346db7d7da2'/>
          </auth>
          <source protocol='rbd' name='vms/1125f797-f4e3-4f7f-a511-e445b4e27135_disk'>
            <host name='172.18.22.234' port='6789'/>
            <host name='172.18.22.235' port='6789'/>
            <host name='172.18.22.236' port='6789'/>
          </source>
          <target dev='vda' bus='virtio'/>
          <alias name='virtio-disk0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
        </disk>
        <disk type='block' device='disk'>
          <driver name='qemu' type='raw' cache='none' io='native'/>
          <source dev='/dev/sdb'/>
          <backingStore/>
          <target dev='vdb' bus='virtio'/>
          <iotune>
            <total_bytes_sec>2048000</total_bytes_sec>
            <total_iops_sec>200</total_iops_sec>
          </iotune>
          <serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
          <alias name='virtio-disk1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </disk>

    NOTE:Cinder 暂时是不支持动态 QoS 的,所以即便我们修改了 QoS Policy 的数值也不能马上被应用到 Libvirt 虚拟机。

    [root@controller ~]# openstack volume qos list
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    | ID                                   | Name | Consumer  | Associations | Properties                                      |
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    | b6d79c1a-9c6b-4a8a-8d8d-b0106232deed | qos1 | front-end | frontend_qos | total_bytes_sec='2048000', total_iops_sec='200' |
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    
    [root@controller ~]# openstack volume qos set --property total_bytes_sec="1024000" qos1
    
    [root@controller ~]# openstack volume qos list
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    | ID                                   | Name | Consumer  | Associations | Properties                                      |
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    | b6d79c1a-9c6b-4a8a-8d8d-b0106232deed | qos1 | front-end | frontend_qos | total_bytes_sec='1024000', total_iops_sec='200' |
    +--------------------------------------+------+-----------+--------------+-------------------------------------------------+
    
    [root@compute ~]# virsh dumpxml instance-00000029
    ...
        <disk type='block' device='disk'>
          <driver name='qemu' type='raw' cache='none' io='native'/>
          <source dev='/dev/sdb'/>
          <backingStore/>
          <target dev='vdb' bus='virtio'/>
          <iotune>
            <total_bytes_sec>2048000</total_bytes_sec>
            <total_iops_sec>200</total_iops_sec>
          </iotune>
          <serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
          <alias name='virtio-disk1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </disk>

    但可以直接通过 virsh 指令来设定 Libvirt 虚拟机的 QoS 属性:

    [root@compute ~]# virsh blkdeviotune instance-00000029 vdb --total-bytes-sec 1024000
    
    [root@compute ~]# virsh blkdeviotune instance-00000029 vdb --total_iops_sec 100
    
    [root@compute ~]# virsh dumpxml instance-00000029
    ...
        <disk type='block' device='disk'>
          <driver name='qemu' type='raw' cache='none' io='native'/>
          <source dev='/dev/sdb'/>
          <backingStore/>
          <target dev='vdb' bus='virtio'/>
          <iotune>
            <total_bytes_sec>1024000</total_bytes_sec>
            <total_iops_sec>100</total_iops_sec>
          </iotune>
          <serial>bd6013bf-1dcd-4b64-9731-05528624d28e</serial>
          <alias name='virtio-disk1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </disk>

    NOTE:Qos 一般来说是在一个建议值上下浮动的,对于严格要求的 QoS 还可以通过指定 --total-iops-sec-max 来固定上限。

    参考文章

    https://docs.openstack.org/cinder/latest/admin/blockstorage-capacity-based-qos.html

    相关阅读:

  • 相关阅读:
    吴裕雄--天生自然轻量级JAVA EE企业应用开发Struts2Sping4Hibernate整合开发学习笔记:Spring_Around
    吴裕雄--天生自然轻量级JAVA EE企业应用开发Struts2Sping4Hibernate整合开发学习笔记:Spring_Args
    吴裕雄--天生自然轻量级JAVA EE企业应用开发Struts2Sping4Hibernate整合开发学习笔记:Spring_AfterThrowing
    (05)mongodb 查询表达式
    (04)mongodb 关于列的操作
    (03)mongodb 数据添加、修改、删除
    (02)mongodb 数据库与collection的基本操作
    (01)mongodb 安装、配置、启动、关闭
    (002)Linux vim命令模式下快捷键保存并退出ZZ
    Linux备份与恢复
  • 原文地址:https://www.cnblogs.com/hzcya1995/p/13310146.html
Copyright © 2011-2022 走看看