Benchmarking with predefined users

Registered by Mikhail Dubov

Currently the benchmark engine takes the admin user credentials as its input and generates a set of temporary tenants/users that it uses later to run benchmark scenarios against the cloud. Rally should support passing these users explicitly as well. This should be particularly useful when one does not want to pass the admin user credentials to Rally, but has a set of predefined users which one would like to use for benchmarking.

This blueprint will be implemented in 5 steps:

1. Add support for user/admin roles in the deploy object.
Instead of storing the endpoints as a simple admin credentials dictionary (containing the username, password etc.) the deploy object should now store more information, i.e. whether the deployment is going to be used with the predefined set of users (the "user" role) or the users should be generated (the "admin" role) and what are the actual credentials of these users. This information should be stored in a dictionary like:

...
"endpoints" : {
    "role": "admin|user",
    "credentials": [
        {"username": ..., "password": ..., "tenant_name": ..., "auth_url": ...},
        {"username": ..., "password": ..., "tenant_name": ..., "auth_url": ...},
        ...
    ]
}

2. Refactor the Scenario runner.
The scenario runner should now support different workflows for the two cases:
    a) "admin" role: the scenario runner should generate temporary tenants/users and exploit them while launching benchmark scenarios;
    b) "user" role: the scenario runner should just initialize OpenStack clients for the given set of users and then use them during benchmarking.

3. Modify the config validation procedure.
The diffentiation between the two roles ("admin"/"user") implies that the benchmark config should be different as well for the cases when users are generated before benchmarking and when they are not. Precisely, the "tenants" and "users_per_tenant" now make no sense when the users are passed explicitly and thus should be absent from the config in this case.

4. Refactor the dummy engine.
Dummy engine refactoring will be the only change in this blueprint noticeable for the end-user. Its input config format should now be changed to get similarly to the one described in step 1.

5. Implement and add the "admin" decorator to benchmark scenarios
The potential problem is now that the user may use a "user-role" deployment but launch the benchmark scenarios which require access to admin OpenStack clients (e.g. the Keystone scenarios). This behaviour should be prevented by explicitly adding an "@admin" sign to such benchmark scenario methods; The scneario runner will then be able to recognize the scenarios it cannot run in the "user" mode.

Blueprint information

Status:
Complete
Approver:
Boris Pavlovic
Priority:
High
Drafter:
Mikhail Dubov
Direction:
Approved
Assignee:
Mikhail Dubov
Definition:
Drafting
Series goal:
None
Implementation:
Implemented
Milestone target:
None
Started by
Boris Pavlovic
Completed by
Boris Pavlovic

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/benchmarking-with-predefined-users,n,z

Addressed by: https://review.openstack.org/67154
    Add support for user/admin roles to the deploy object

Addressed by: https://review.openstack.org/67275
    Refactor unit tests for dummy engine

Addressed by: https://review.openstack.org/67276
    Introduce the Endpoint object

Addressed by: https://review.openstack.org/67643
    Refactor the Scenario runner

Addressed by: https://review.openstack.org/67710
    Modify the config validation procedure

Addressed by: https://review.openstack.org/67720
    Refactor the dummy engine

Addressed by: https://review.openstack.org/70771
    Implement ScenarioRunner execution types via inheritance

Gerrit topic: https://review.openstack.org/#q,topic:bp/benchmark-context-predefined-users,n,z

Addressed by: https://review.openstack.org/85300
    Improve @base.scenario to specify it is admin only

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.