描述
最近在使用rally做openstack的性能测试，发现执行task的时候总是报错，openrc什么的环境变量也都ok了，但是执行rally task start --task list-images.json总是报错如标题所示。
详细过程如下:
Running Rally version 0.9.1~dev6
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Task is:
{
"GlanceImages.list_images": [
{
"runner": {
"type": "constant",
"times": 10,
"concurrency": 1
},
"context": {
"users": {
"tenants": 2,
"users_per_tenant": 2
},
"images": {
"image_url": "http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img",
"image_type": "qcow2",
"image_container": "bare",
"images_per_tenant": 4
}
}
}
]
}
Task syntax is correct :)
2017-04-17 04:25:43.028 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Task validation.
2017-04-17 04:25:43.039 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Task validation of scenarios names.
2017-04-17 04:25:43.042 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Completed: Task validation of scenarios names.
2017-04-17 04:25:43.043 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Task validation of syntax.
2017-04-17 04:25:43.055 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Completed: Task validation of syntax.
2017-04-17 04:25:43.056 31424 INFO rally.task.engine [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Task validation of semantic.
2017-04-17 04:25:43.548 31424 INFO rally.plugins.openstack.context.keystone.users [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Enter context: `users`
2017-04-17 04:25:43.838 31424 WARNING rally.common.broker [-] Failed to consume a task from the queue: The resource could not be found. (HTTP 404) (Request-ID: req-0fb5eea5-37ba-4b61-b984-b715258214d8)
2017-04-17 04:25:43.839 31424 INFO rally.plugins.openstack.context.keystone.users [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Starting: Exit context: `users`
2017-04-17 04:26:00.349 31424 INFO rally.plugins.openstack.context.keystone.users [-] Task aa59de44-2171-49f6-ac4e-6fe3fab76d3e | Completed: Exit context: `users`
Task config is invalid: `Unable to setup context 'users': 'Failed to create the requested number of tenants.'`
在解决这个问题之前，顺便说一下rally task的命令，在使用rally task start --task xxx.json/xxx.yaml的时候，就算执行失败了我们也会在rally.sqlite（rally默认数据库）中产生失败的task id，作为一个精神洁癖患者的我，我想只保存成功的task id。怎么办？
很简单！我们在执行rally task start --task xxx.json/xxx.yaml之前可以通过命令rally task validate --task xxx.json/xxx.yaml来验证脚步xxx.json/xxx.yaml的合法性，比如出现如下：
rally task validate --task list-images.yaml
2017-04-17 16:12:16.632 16594 WARNING rally.task.validation [-] Plugin 'NeutronNetworks.create_and_show_subnets' uses validator 'required_openstack'. That validator is deprecated in favor of 'required_platform' in Rally v0.10.0.
2017-04-17 16:12:16.638 16594 WARNING rally.task.validation [-] Plugin 'NeutronNetworks.set_and_clear_router_gateway' uses validator 'required_openstack'. That validator is deprecated in favor of 'required_platform' in Rally v0.10.0.
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Task is:
---
GlanceImages.list_images:
-
runner:
type: "constant"
times: 10
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
images:
image_url: "http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img"
image_type: "qcow2"
image_container: "bare"
images_per_tenant: 4
Task syntax is correct :)
2017-04-17 16:12:16.898 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Task validation.
2017-04-17 16:12:16.899 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Task validation of scenarios names.
2017-04-17 16:12:16.901 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Task validation of scenarios names.
2017-04-17 16:12:16.902 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Task validation of syntax.
2017-04-17 16:12:16.911 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Task validation of syntax.
2017-04-17 16:12:16.911 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Task validation of semantic.
2017-04-17 16:12:17.213 16594 INFO rally.plugins.openstack.context.keystone.users [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Enter context: `users`
2017-04-17 16:12:17.783 16594 INFO rally.plugins.openstack.context.keystone.users [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Enter context: `users`
2017-04-17 16:12:17.935 16594 INFO rally.plugins.openstack.context.keystone.users [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Starting: Exit context: `users`
2017-04-17 16:12:20.207 16594 INFO rally.plugins.openstack.context.keystone.users [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Exit context: `users`
2017-04-17 16:12:20.207 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Task validation of semantic.
2017-04-17 16:12:20.208 16594 INFO rally.task.engine [-] Task e2b1237f-79db-4ada-a5f4-9902d9061891 | Completed: Task validation.
Task config is valid :)
这个时候我们知道list-images.yaml语法以及rally config是正确的，这个时候在执行rally task start --task list-images.yaml就可以了。
同时为了详细发现验证问题我们可以通过命令rally -dv task validate --task xxx.json/xxx.yaml来显示详细显示（精确到代码行）。
如下：
(rally) [stack@localhost result-rally]$ rally -dv task validate --task list-images.json
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-04-17 04:45:47.851 1850 RALLYDEBUG rally.api [-] INFO logs from urllib3 and requests module are hide.
2017-04-17 04:45:47.852 1850 RALLYDEBUG rally.api [-] urllib3 insecure warnings are hidden.
2017-04-17 04:45:47.852 1850 RALLYDEBUG rally.api [-] ERROR log from boto module is hide.
2017-04-17 04:45:47.853 1850 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'rally.common.db.sqlalchemy.api' _load_backend /opt/stack/rally/lib/python2.7/site-packages/oslo_db/api.py:238
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Task is:
{
"GlanceImages.list_images": [
{
"runner": {
"type": "constant",
"times": 10,
"concurrency": 1
},
"context": {
"users": {
"tenants": 2,
"users_per_tenant": 2
},
"images": {
"image_url": "http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img",
"image_type": "qcow2",
"image_container": "bare",
"images_per_tenant": 4
}
}
}
]
}
Task syntax is correct :)
2017-04-17 04:45:49.010 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Task validation.
2017-04-17 04:45:49.011 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Task validation of scenarios names.
2017-04-17 04:45:49.015 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Completed: Task validation of scenarios names.
2017-04-17 04:45:49.015 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Task validation of syntax.
2017-04-17 04:45:49.028 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Completed: Task validation of syntax.
2017-04-17 04:45:49.028 1850 INFO rally.task.engine [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Task validation of semantic.
2017-04-17 04:45:49.411 1850 INFO rally.plugins.openstack.context.keystone.users [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Enter context: `users`
2017-04-17 04:45:49.412 1850 RALLYDEBUG rally.plugins.openstack.context.keystone.users [-] Creating 1 tenants using 20 threads
2017-04-17 04:45:49.624 1850 WARNING rally.common.broker [-] Failed to consume a task from the queue: The resource could not be found. (HTTP 404) (Request-ID: req-96724a10-afd6-4d58-92e0-a804f22f85e5)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker [-] The resource could not be found. (HTTP 404) (Request-ID: req-96724a10-afd6-4d58-92e0-a804f22f85e5)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker Traceback (most recent call last):
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/common/broker.py", line 43, in _consumer
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker consume(cache, args)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/plugins/openstack/context/keystone/users.py", line 183, in consume
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker tenant = cache["client"].create_project(domain_name=domain)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/service.py", line 145, in wrapper
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return func(instance, *args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/plugins/openstack/services/identity/identity.py", line 43, in create_project
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker domain_name=domain_name)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/service.py", line 145, in wrapper
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return func(instance, *args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/plugins/openstack/services/identity/keystone_v2.py", line 191, in create_project
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker tenant = self._impl.create_tenant(project_name)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/service.py", line 145, in wrapper
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return func(instance, *args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker f = func(self, *args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/rally/plugins/openstack/services/identity/keystone_v2.py", line 29, in create_tenant
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return self._clients.keystone("2").tenants.create(tenant_name)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneclient/v2_0/tenants.py", line 99, in create
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return self._post('/tenants', params, "tenant")
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneclient/base.py", line 174, in _post
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker resp, body = self.client.post(url, body=body, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 229, in post
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return self.request(url, 'POST', **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 380, in request
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 148, in request
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return self.session.request(url, method, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker return wrapped(*args, **kwargs)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker File "/opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/session.py", line 655, in request
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker raise exceptions.from_response(resp, method, url)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-96724a10-afd6-4d58-92e0-a804f22f85e5)
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker
2017-04-17 04:45:49.628 1850 INFO rally.plugins.openstack.context.keystone.users [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Starting: Exit context: `users`
2017-04-17 04:45:51.121 1850 DEBUG neutronclient.v2_0.client [-] GET call to neutron for http://192.168.39.165:9696/v2.0/extensions.json used request id req-0431ad8e-ee9d-495a-aed3-d17c253da6c7 _append_request_id /opt/stack/rally/lib/python2.7/site-packages/neutronclient/v2_0/client.py:128
2017-04-17 04:45:51.270 1850 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X GET http://192.168.39.165:8774/v2.1/0788a6f1e847478db06be597b4ada38b/os-services -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}924d03ab9a5abc0e2fcf15168c5b5539ef54a89b" _http_log_request /opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/session.py:347
2017-04-17 04:45:51.642 1850 DEBUG novaclient.v2.client [-] RESP: [200] Content-Length: 801 Content-Type: application/json X-Openstack-Nova-Api-Version: 2.1 Vary: X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-6ab3539c-0e37-4385-82dd-83b48e6e90a6 Date: Mon, 17 Apr 2017 08:45:51 GMT Connection: keep-alive
RESP BODY: {"services": [{"status": "enabled", "binary": "nova-conductor", "host": "localhost.localdomain", "zone": "internal", "state": "up", "disabled_reason": null, "id": 7, "updated_at": "2017-04-17T08:45:48.000000"}, {"status": "enabled", "binary": "nova-scheduler", "host": "localhost.localdomain", "zone": "internal", "state": "up", "disabled_reason": null, "id": 11, "updated_at": "2017-04-17T08:45:43.000000"}, {"status": "enabled", "binary": "nova-consoleauth", "host": "localhost.localdomain", "zone": "internal", "state": "up", "disabled_reason": null, "id": 12, "updated_at": "2017-04-17T08:45:50.000000"}, {"status": "enabled", "binary": "nova-compute", "host": "localhost.localdomain", "zone": "nova", "state": "up", "disabled_reason": null, "id": 13, "updated_at": "2017-04-17T08:45:42.000000"}]}
_http_log_response /opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/session.py:395
2017-04-17 04:45:51.642 1850 DEBUG novaclient.v2.client [-] GET call to compute for http://192.168.39.165:8774/v2.1/0788a6f1e847478db06be597b4ada38b/os-services used request id req-6ab3539c-0e37-4385-82dd-83b48e6e90a6 request /opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/session.py:640
2017-04-17 04:45:51.647 1850 INFO rally.plugins.openstack.context.keystone.users [-] Task a8f62843-4a34-4b3f-b047-b45376fd055b | Completed: Exit context: `users`
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine [-] Unable to setup context 'users': 'Failed to create the requested number of tenants.'
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine Traceback (most recent call last):
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/engine.py", line 372, in validate
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine self._validate_config_semantic(self.config)
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/common/logging.py", line 197, in wrapper
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine result = f(self, *args, **kwargs)
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/engine.py", line 350, in _validate_config_semantic
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine workloads_with_users, self.deployment)
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/task/engine.py", line 295, in _validate_config_semantic_helper
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine ctx.setup()
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/common/logging.py", line 197, in wrapper
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine result = f(self, *args, **kwargs)
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine File "/opt/stack/rally/lib/python2.7/site-packages/rally/plugins/openstack/context/keystone/users.py", line 292, in setup
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine msg=_("Failed to create the requested number of tenants."))
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine ContextSetupFailure: Unable to setup context 'users': 'Failed to create the requested number of tenants.'
2017-04-17 04:45:51.647 1850 ERROR rally.task.engine
Task config is invalid: `Unable to setup context 'users': 'Failed to create the requested number of tenants.'`
通过对上面的结果分析，我发现有用的信息有两个:
1.
2017-04-17 04:45:49.624 1850 ERROR rally.common.broker [-] The resource could not be found. (HTTP 404) (Request-ID: req-96724a10-afd6-4d58-92e0-a804f22f85e5)
2.
2017-04-17 04:45:51.642 1850 DEBUG novaclient.v2.client [-] GET call to compute for http://192.168.39.165:8774/v2.1/0788a6f1e847478db06be597b4ada38b/os-services used request id req-6ab3539c-0e37-4385-82dd-83b48e6e90a6 request /opt/stack/rally/lib/python2.7/site-packages/keystoneauth1/session.py:640
解决办法
是session的因素，于是我重启一下openstack的数据库sudo systemctl restart mysql.service然后再执行rally task validate --task xxx.json/xxx.yaml就好使了。
note：上面提到的xxx.json/xxx.yaml为你需要进行测试的脚步文件，json格式或者yaml格式（怕小白误解，补充一下）。
补充：
我使用的openstack是单机版的，通过devstack安装的，有时候上述方法还不好使的时候试试使用admin-openrc-v3.sh这个认证试试，清除一下系统缓存再试试。我第一次遇到这个问题重启好了，第二次的时候修改认证版本并清除缓存好使了。