zoukankan      html  css  js  c++  java
  • 云计算OpenStack核心组件---glance镜像服务(6)

    一、glance介绍:

      Glance是Openstack项目中负责镜像管理的模块,其功能包括虚拟机镜像的查找、注册和检索等。 Glance提供Restful API可以查询虚拟机镜像的metadata及获取镜像。 Glance可以将镜像保存到多种后端存储上,比如简单的文件存储或者对象存储。

    1、Image

      要理解 Image Service,先得搞清楚什么是 Image 以及为什么要用 Image?
      在传统 IT 环境下,安装一个系统要么从安装 CD 从头安装,要么用 Ghost 等克隆工具恢复。这两种方式有如下几个问题:
    1. 如果要安装的系统多了效率就很低
    2. 时间长,工作量大
    3. 安装完还要进行手工配置,比如安装其他的软件,设置 IP 等
    4. 备份和恢复系统不灵活
      云环境下需要更高效的方案,这就是 Image。 Image 是一个模板,里面包含了基本的操作系统和其他的软件。
      举例来说,有家公司需要为每位员工配置一套办公用的系统,一般需要一个 Win7 系统再加 MS office 软件。 OpenStack 是这么玩的:
    1. 先手工安装好这么一个虚机
    2. 然后对虚机执行 snapshot,这样就得到了一个 image
    3. 当有新员工入职需要办公环境时,立马启动一个或多个该 image 的 instance(虚机)就可以了
      在这个过程中,第 1 步跟传统方式类似,需要手工操作和一定时间,但第 2、3 步非常快,全自动化,一般都是秒级别。而且 2、3 步可以循环做。 比如公司新上了一套 OA 系统,每个员工的 PC 上都得有客户端软件。 那么可以在某个现有虚机中先手工安装好 OA 客户端,然后执行 snapshot 操作,得到新的 image,以后可以就直接使用新 image 创建虚机了。另外,snapshot 还有备份的作用,能够非常方便的恢复系统。

    2、Image Service

      Image Service 的功能是管理 Image,让用户能够发现、获取和保存 Image。在 OpenStack 中,提供 Image Service 的是 Glance,其具体功能如下:
    1. 提供 REST API 让用户能够查询和获取 image 的元数据和 image 本身
    2. 支持多种方式存储 image,包括普通的文件系统、Swift、Amazon S3 等
    3. 对 Instance 执行 Snapshot 创建新的 image

    3、Glance架构

    glance的架构图如下:

    4、glance-api

      glance-api 是系统后台运行的服务进程。 对外提供 REST API,响应 image 查询、获取和存储的调用
      glance-api 不会真正处理请求。 如果操作是与 image metadata(元数据)相关,glance-api 会把请求转发给 glance-registry; 如果操作是与 image 自身存取相关,glance-api 会把请求转发给该 image 的 store backend。
      在控制节点上可以查看 glance-api 进程

    5、glance-registry

      glance-registry 是系统后台运行的服务进程。 负责处理和存取 image 的 metadata,例如 image 的大小和类型。在控制节点上可以查看 glance-registry 进程

    Glance 支持多种格式的 image,包括

    6、Database

      Image 的 metadata 会保持到 database 中,默认是 MySQL。 在控制节点上可以查看 glance 的 database 信息

    7、Store backend

      Glance 自己并不存储 image。 真正的 image 是存放在 backend 中的。 Glance 支持多种 backend,包括:
    1. A directory on a local file system(这是默认配置)
    2. GridFS
    3. Ceph RBD
    4. Amazon S3
    5. Sheepdog
    6. OpenStack Block Storage (Cinder)
    7. OpenStack Object Storage (Swift)
    8. VMware ESX
    具体使用哪种 backend,是在 /etc/glance/glance-api.conf 中配置的

      其他 backend 的配置可参考http://docs.openstack.org/liberty/config-reference/content/configuring-image-service-backends.html
    查看目前已经存在的 image

    查看保存目录

    每个 image 在目录下都对应有一个文件,文件以 image 的 ID 命名。

    二、glance创建镜像:

      OpenStack 为终端用户提供了 Web UI(Horizon)和命令行 CLI 两种交换界面。两种方式我们都要会用。可能有些同学觉得既然有更友好的 Web UI 了,干嘛还要用 CLI? 这里有下面的理由:
    1、Web UI 的功能没有 CLI 全,有些操作只提供了 CLI。 即便是都有的功能,CLI 可以使用的参数更多
    2、一般来说,CLI 返回结果更快,操作起来更高效
    4、CLI 可放在脚本中进行批处理
    5、有些耗时的操作 CLI 更合适,比如创建镜像(后面将涉及)

    (1)Web UI创建image

    (2)CLI创建image

    将上传的镜像传到控制节点

    执行image上传镜像命令:
    openstack image create "cirros"   --file cirros-0.3.3-x86_64-disk.img.img   --disk-format qcow2 --container-format bare --public

    三、安装配置glance服务(控制节点)

    https://docs.openstack.org/ocata/install-guide-rdo/glance-install.html

    1、在数据库中创建glance服务的数据库及用户,并设置权限

    (1)使用root用户登录数据库

    [root@ren3 ~]# mysql -u root -proot

    (2)创建glance数据库

    MariaDB [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | keystone           |
    | mysql              |
    | performance_schema |
    +--------------------+
    4 rows in set (0.05 sec)
    
    MariaDB [(none)]> create database glance;
    Query OK, 1 row affected (0.00 sec)
    
    MariaDB [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | glance             |
    | information_schema |
    | keystone           |
    | mysql              |
    | performance_schema |
    +--------------------+
    5 rows in set (0.00 sec)

    (3)创建glance用户并授权

    MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' 
           IDENTIFIED BY 'GLANCE_DBPASS';
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' 
           IDENTIFIED BY 'GLANCE_DBPASS';
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> select user,host,password from mysql.user;
    +----------+-----------+-------------------------------------------+
    | user     | host      | password                                  |
    +----------+-----------+-------------------------------------------+
    | root     | localhost | *81F5E21E35407D884A6CD4A731AEBFB6AF209E1B |
    | root     | 127.0.0.1 | *81F5E21E35407D884A6CD4A731AEBFB6AF209E1B |
    | root     | ::1       | *81F5E21E35407D884A6CD4A731AEBFB6AF209E1B |
    | glance   | localhost | *C0CE56F2C0C7234791F36D89700B02691C1CAB8E |
    | keystone | localhost | *442DFE587A8B6BE1E9538855E8187C1EFB863A73 |
    | keystone | %         | *442DFE587A8B6BE1E9538855E8187C1EFB863A73 |
    | glance   | %         | *C0CE56F2C0C7234791F36D89700B02691C1CAB8E |
    +----------+-----------+-------------------------------------------+
    7 rows in set (0.00 sec)

    退出数据库

    2、加载OpenStack系统变量文件

    [root@ren3 ~]# ls
    anaconda-ks.cfg  openrc  yum-repo.sh
    [root@ren3 ~]# source openrc 

    3、创建服务凭证

    (1)创建glance用户:

    [root@ren3 ~]# openstack user create --domain default --password=glance glance
    +---------------------+----------------------------------+
    | Field               | Value                            |
    +---------------------+----------------------------------+
    | domain_id           | default                          |
    | enabled             | True                             |
    | id                  | 2807e20f8405473e831602b6b12588c7 |
    | name                | glance                           |
    | options             | {}                               |
    | password_expires_at | None                             |
    +---------------------+----------------------------------+
    [root@ren3 ~]# openstack user list
    +----------------------------------+--------+
    | ID                               | Name   |
    +----------------------------------+--------+
    | 2807e20f8405473e831602b6b12588c7 | glance |
    | 372fccfd264c4edfb600af3f56052ec7 | admin  |
    | 37c7c00d574146e8817413b7a091f594 | demo   |
    +----------------------------------+--------+

    (2)添加admin角色和服务项目到glance用户:

    [root@ren3 ~]# openstack role add --project service --user glance admin

    (3)创建glance服务

    [root@ren3 ~]# openstack service create --name glance 
       --description "OpenStack Image" image
    +-------------+----------------------------------+
    | Field       | Value                            |
    +-------------+----------------------------------+
    | description | OpenStack Image                  |
    | enabled     | True                             |
    | id          | a7cf08799d4b4b509530ae6c21453b08 |
    | name        | glance                           |
    | type        | image                            |
    +-------------+----------------------------------+
    [root@ren3 ~]# openstack service list
    +----------------------------------+----------+----------+
    | ID                               | Name     | Type     |
    +----------------------------------+----------+----------+
    | a7cf08799d4b4b509530ae6c21453b08 | glance   | image    |
    | ab70227ae28c4fb7a774ed4808489e76 | keystone | identity |
    +----------------------------------+----------+----------+

    4、创建image服务API端点

    [root@ren3 ~]# openstack endpoint list
    +----------+----------+--------------+--------------+---------+-----------+----------+
    | ID       | Region   | Service Name | Service Type | Enabled | Interface | URL      |
    +----------+----------+--------------+--------------+---------+-----------+----------+
    | 26d8cf53 | RegionOn | keystone     | identity     | True    | public    | http://r |
    | 020e4e2b | e        |              |              |         |           | en3:5000 |
    | a2c1ee0f |          |              |              |         |           | /v3/     |
    | a962137a |          |              |              |         |           |          |
    | 721facd6 | RegionOn | keystone     | identity     | True    | internal  | http://r |
    | 645b4efb | e        |              |              |         |           | en3:5000 |
    | b7fd6956 |          |              |              |         |           | /v3/     |
    | 42c156d4 |          |              |              |         |           |          |
    | 91bf5a46 | RegionOn | keystone     | identity     | True    | admin     | http://r |
    | 04e74aa2 | e        |              |              |         |           | en3:3535 |
    | bb229f96 |          |              |              |         |           | 7/v3/    |
    | 58bddc26 |          |              |              |         |           |          |
    +----------+----------+--------------+--------------+---------+-----------+----------+
    [root@ren3 ~]# openstack endpoint create --region RegionOne 
       image public http://ren3:9292
    +--------------+----------------------------------+
    | Field        | Value                            |
    +--------------+----------------------------------+
    | enabled      | True                             |
    | id           | cfe4773e4a3b480e8bdb2062fbdcdd33 |
    | interface    | public                           |
    | region       | RegionOne                        |
    | region_id    | RegionOne                        |
    | service_id   | a7cf08799d4b4b509530ae6c21453b08 |
    | service_name | glance                           |
    | service_type | image                            |
    | url          | http://ren3:9292                 |
    +--------------+----------------------------------+
    [root@ren3 ~]# openstack endpoint create --region RegionOne 
       image internal http://ren3:9292
    +--------------+----------------------------------+
    | Field        | Value                            |
    +--------------+----------------------------------+
    | enabled      | True                             |
    | id           | 80f23b7a63c1416da77841119d5b5169 |
    | interface    | internal                         |
    | region       | RegionOne                        |
    | region_id    | RegionOne                        |
    | service_id   | a7cf08799d4b4b509530ae6c21453b08 |
    | service_name | glance                           |
    | service_type | image                            |
    | url          | http://ren3:9292                 |
    +--------------+----------------------------------+
    [root@ren3 ~]# openstack endpoint create --region RegionOne 
       image admin http://ren3:9292
    +--------------+----------------------------------+
    | Field        | Value                            |
    +--------------+----------------------------------+
    | enabled      | True                             |
    | id           | a80b58f2db9c4aea9b8748914d16d028 |
    | interface    | admin                            |
    | region       | RegionOne                        |
    | region_id    | RegionOne                        |
    | service_id   | a7cf08799d4b4b509530ae6c21453b08 |
    | service_name | glance                           |
    | service_type | image                            |
    | url          | http://ren3:9292                 |
    +--------------+----------------------------------+
    [root@ren3 ~]# openstack endpoint list
    +----------+----------+--------------+--------------+---------+-----------+----------+
    | ID       | Region   | Service Name | Service Type | Enabled | Interface | URL      |
    +----------+----------+--------------+--------------+---------+-----------+----------+
    | 26d8cf53 | RegionOn | keystone     | identity     | True    | public    | http://r |
    | 020e4e2b | e        |              |              |         |           | en3:5000 |
    | a2c1ee0f |          |              |              |         |           | /v3/     |
    | a962137a |          |              |              |         |           |          |
    | 721facd6 | RegionOn | keystone     | identity     | True    | internal  | http://r |
    | 645b4efb | e        |              |              |         |           | en3:5000 |
    | b7fd6956 |          |              |              |         |           | /v3/     |
    | 42c156d4 |          |              |              |         |           |          |
    | 80f23b7a | RegionOn | glance       | image        | True    | internal  | http://r |
    | 63c1416d | e        |              |              |         |           | en3:9292 |
    | a7784111 |          |              |              |         |           |          |
    | 9d5b5169 |          |              |              |         |           |          |
    | 91bf5a46 | RegionOn | keystone     | identity     | True    | admin     | http://r |
    | 04e74aa2 | e        |              |              |         |           | en3:3535 |
    | bb229f96 |          |              |              |         |           | 7/v3/    |
    | 58bddc26 |          |              |              |         |           |          |
    | a80b58f2 | RegionOn | glance       | image        | True    | admin     | http://r |
    | db9c4aea | e        |              |              |         |           | en3:9292 |
    | 9b874891 |          |              |              |         |           |          |
    | 4d16d028 |          |              |              |         |           |          |
    | cfe4773e | RegionOn | glance       | image        | True    | public    | http://r |
    | 4a3b480e | e        |              |              |         |           | en3:9292 |
    | 8bdb2062 |          |              |              |         |           |          |
    | fbdcdd33 |          |              |              |         |           |          |
    +----------+----------+--------------+--------------+---------+-----------+----------+

    5、安装glance软件包

    [root@ren3 ~]# yum install openstack-glance -y

    6、编辑/etc/glance/glance-api.conf文件

    (1)在[database]部分,配置数据库访问:

    [database]
    # ...
    connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance

    (2)在[keystone_authtoken]和[paste_deploy]部分,配置身份服务访问:

    [keystone_authtoken]
    # ...
    auth_uri = http://controller:5000
    auth_url = http://controller:35357
    memcached_servers = controller:11211
    auth_type = password
    project_domain_name = default
    user_domain_name = default
    project_name = service
    username = glance
    password = GLANCE_PASS
    
    [paste_deploy]
    # ...
    flavor = keystone

    (3)在[glance_store]部分,配置本地文件系统存储和图像文件的位置:

    [glance_store]
    # ...
    stores = file,http
    default_store = file
    filesystem_store_datadir = /var/lib/glance/images/

    编辑好的配置文件:

    [root@ren3 ~]# vim /etc/glance/glance-api.conf
    [DEFAULT]
    
    [cors]
    
    [cors.subdomain]
    
    [database]
    connection = mysql+pymysql://glance:GLANCE_DBPASS@ren3/glance
    
    [glance_store]
    stores = file,http
    default_store = file
    filesystem_store_datadir = /var/lib/glance/images/
    
    [image_format]
    
    [keystone_authtoken]
    auth_uri = http://ren3:5000
    auth_url = http://ren3:35357
    memcached_servers = ren3:11211
    auth_type = password
    project_domain_name = default
    user_domain_name = default
    project_name = service
    username = glance
    password = glance
    
    [matchmaker_redis]
    [DEFAULT]
    
    [cors]
    
    [cors.subdomain]
    
    [database]
    connection = mysql+pymysql://glance:GLANCE_DBPASS@ren3/glance
    
    [glance_store]
    stores = file,http
    default_store = file
    filesystem_store_datadir = /var/lib/glance/images/
    
    [image_format]
    
    [keystone_authtoken]
    auth_uri = http://ren3:5000
    auth_url = http://ren3:35357
    memcached_servers = ren3:11211
    auth_type = password
    project_domain_name = default
    user_domain_name = default
    project_name = service
    username = glance
    password = glance
    
    [matchmaker_redis]
    
    [oslo_concurrency]
    
    [oslo_messaging_amqp]
    
    [oslo_messaging_kafka]
    
    [oslo_messaging_notifications]
    
    [oslo_messaging_rabbit]
    
    [oslo_messaging_zmq]
    
    [oslo_middleware]
    
    [oslo_policy]
    
    [paste_deploy]
    flavor = keystone
    
    [profiler]
    
    [store_type_location_strategy]
    
    [task]
    
    [taskflow_executor]

    7、编辑/etc/glance/glance- registration .conf文件

    (1)在[database]部分,配置数据库访问:

    [database]
    # ...
    connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance

    (2)在[keystone_authtoken]和[paste_deploy]部分,配置身份服务访问:

    [keystone_authtoken]
    # ...
    auth_uri = http://controller:5000
    auth_url = http://controller:35357
    memcached_servers = controller:11211
    auth_type = password
    project_domain_name = default
    user_domain_name = default
    project_name = service
    username = glance
    password = GLANCE_PASS
    
    [paste_deploy]
    # ...
    flavor = keystone

    编辑好的配置文件:

    [root@ren3 ~]# vim /etc/glance/glance-registry.conf
    [DEFAULT]
    
    [database]
    connection = mysql+pymysql://glance:GLANCE_DBPASS@ren3/glance
    
    [keystone_authtoken]
    auth_uri = http://ren3:5000
    [DEFAULT]
    
    [database]
    connection = mysql+pymysql://glance:GLANCE_DBPASS@ren3/glance
    
    [keystone_authtoken]
    auth_uri = http://ren3:5000
    auth_url = http://ren3:35357
    memcached_servers = ren3:11211
    auth_type = password
    project_domain_name = default
    user_domain_name = default
    project_name = service
    username = glance
    password = glance
     
    [matchmaker_redis]
     
    [oslo_messaging_amqp]
     
    [oslo_messaging_kafka]
     
    [oslo_messaging_notifications]
     
    [oslo_messaging_rabbit]
     
    [oslo_messaging_zmq]
     
    [oslo_policy]
    
    [paste_deploy]
    flavor = keystone
    
    [profiler]

    8、同步数据库

    [root@ren3 ~]# su -s /bin/sh -c "glance-manage db_sync" glance
    [root@ren3 ~]# mysql -u glance -pGLANCE_DBPASS
    
    MariaDB [(none)]> use glance;
    
    MariaDB [glance]> show tables;
    +----------------------------------+
    | Tables_in_glance                 |
    +----------------------------------+
    | alembic_version                  |
    | artifact_blob_locations          |
    | artifact_blobs                   |
    | artifact_dependencies            |
    | artifact_properties              |
    | artifact_tags                    |
    | artifacts                        |
    | image_locations                  |
    | image_members                    |
    | image_properties                 |
    | image_tags                       |
    | images                           |
    | metadef_namespace_resource_types |
    | metadef_namespaces               |
    | metadef_objects                  |
    | metadef_properties               |
    | metadef_resource_types           |
    | metadef_tags                     |
    | migrate_version                  |
    | task_info                        |
    | tasks                            |
    +----------------------------------+
    21 rows in set (0.00 sec)

    9、启动glance服务

    [root@ren3 ~]# systemctl enable openstack-glance-api.service 
       openstack-glance-registry.service
    [root@ren3 ~]# systemctl start openstack-glance-api.service 
       openstack-glance-registry.service
    [root@ren3 ~]# systemctl status openstack-glance-api.service 
       openstack-glance-registry.service |grep active | wc -l
    2
    [root@ren3 ~]# ss -tnl
    State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
    LISTEN     0      128      *:9191                 *:*                  
    LISTEN     0      128    192.168.11.3:5672                 *:*                  
    LISTEN     0      128      *:25672                *:*                  
    LISTEN     0      128    192.168.11.3:3306                 *:*                  
    LISTEN     0      128    192.168.11.3:11211                *:*                  
    LISTEN     0      128    127.0.0.1:11211                *:*                  
    LISTEN     0      128      *:9292                 *:*                  
    LISTEN     0      128      *:4369                 *:*                  
    LISTEN     0      128      *:22                   *:*                  
    LISTEN     0      128      *:15672                *:*                  
    LISTEN     0      100    127.0.0.1:25                   *:*                  
    LISTEN     0      128     :::5000                :::*                  
    LISTEN     0      128    ::1:11211               :::*                  
    LISTEN     0      128     :::80                  :::*                  
    LISTEN     0      128     :::22                  :::*                  
    LISTEN     0      100    ::1:25                  :::*                  
    LISTEN     0      128     :::35357               :::*                  
    [root@ren3 ~]# netstat -anp |grep 9191
    tcp        0      0 0.0.0.0:9191            0.0.0.0:*               LISTEN      13084/python2       
    [root@ren3 ~]# firewall-cmd --list-ports
    4369/tcp 5672/tcp 15672/tcp 25672/tcp 3306/tcp 11211/tcp 80/tcp 35357/tcp 5000/tcp
    [root@ren3 ~]# firewall-cmd --add-port=9292/tcp
    success
    [root@ren3 ~]# firewall-cmd --add-port=9292/tcp --permanent
    success
    [root@ren3 ~]# firewall-cmd --add-port=9191/tcp --permanent
    success
    [root@ren3 ~]# firewall-cmd --add-port=9191/tcp 
    success

    10、下载源镜像

    网络源:
    wget http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
    [root@ren3 ~]# wget http://download2.yunwei.edu/shell/openstack_app.tar.gz
    [root@ren3 ~]# ls
    anaconda-ks.cfg  openrc                yum-repo.sh
    --description    openstack_app.tar.gz 
    [root@ren3 ~]# tar xvf openstack_app.tar.gz 
    [root@ren3 ~]# ls
    anaconda-ks.cfg  openrc                openstack-ocata
    --description    openstack_app.tar.gz  yum-repo.sh
    [root@ren3 ~]# cd openstack-ocata/
    [root@ren3 openstack-ocata]# ls
    cirros-0.3.3-x86_64-disk.img  openstack-compute-yilai

    11、使用QCOW2磁盘格式、bare容器格式和公共可见性将镜像上传至image服务,以便所有项目都可以访问:

    [root@ren3 openstack-ocata]# openstack image create "cirros" --file cirros-0.3.3-x86_64-disk.img --disk-format qcow2 --container-format bare --public
    +------------------+---------------------------------------------------+
    | Field            | Value                                             |
    +------------------+---------------------------------------------------+
    | checksum         | 133eae9fb1c98f45894a4e60d8736619                  |
    | container_format | bare                                              |
    | created_at       | 2019-10-12T06:12:16Z                              |
    | disk_format      | qcow2                                             |
    | file             | /v2/images/d8e9a113-edef-                         |
    |                  | 41a6-9778-622edf76de39/file                       |
    | id               | d8e9a113-edef-41a6-9778-622edf76de39              |
    | min_disk         | 0                                                 |
    | min_ram          | 0                                                 |
    | name             | cirros                                            |
    | owner            | 640da7a471524d35a3efca2692b9555a                  |
    | protected        | False                                             |
    | schema           | /v2/schemas/image                                 |
    | size             | 13200896                                          |
    | status           | active                                            |
    | tags             |                                                   |
    | updated_at       | 2019-10-12T06:12:16Z                              |
    | virtual_size     | None                                              |
    | visibility       | public                                            |
    +------------------+---------------------------------------------------+

    12、确认镜像是否上传成功

    [root@ren3 openstack-ocata]# glance image-list
    +--------------------------------------+--------+
    | ID                                   | Name   |
    +--------------------------------------+--------+
    | d8e9a113-edef-41a6-9778-622edf76de39 | cirros |
    +--------------------------------------+--------+
    [root@ren3 openstack-ocata]# cd /var/lib/glance/images/
    [root@ren3 images]# ls
    d8e9a113-edef-41a6-9778-622edf76de39

    或者使用openstack命令:

    [root@ren3 images]# openstack image list
    +--------------------------------------+--------+--------+
    | ID                                   | Name   | Status |
    +--------------------------------------+--------+--------+
    | d8e9a113-edef-41a6-9778-622edf76de39 | cirros | active |
    +--------------------------------------+--------+--------+
  • 相关阅读:
    梦断代码,读书计划
    《梦断代码》读后感1
    四则运算2详细过程
    四则运算2初步构思
    四则运算
    阅读《梦断代码》计划
    四则运算2设计思路
    随机输出30道四则运算题
    软件工程阅读计划
    面试题04
  • 原文地址:https://www.cnblogs.com/renyz/p/11656808.html
Copyright © 2011-2022 走看看