Benchmarking with predefined users

Registered by Mikhail Dubov on 2014-01-16

Currently the benchmark engine takes the admin user credentials as its input and generates a set of temporary tenants/users that it uses later to run benchmark scenarios against the cloud. Rally should support passing these users explicitly as well. This should be particularly useful when one does not want to pass the admin user credentials to Rally, but has a set of predefined users which one would like to use for benchmarking.

This blueprint will be implemented in 5 steps:

1. Add support for user/admin roles in the deploy object.
Instead of storing the endpoints as a simple admin credentials dictionary (containing the username, password etc.) the deploy object should now store more information, i.e. whether the deployment is going to be used with the predefined set of users (the "user" role) or the users should be generated (the "admin" role) and what are the actual credentials of these users. This information should be stored in a dictionary like:

"endpoints" : {
    "role": "admin|user",
    "credentials": [
        {"username": ..., "password": ..., "tenant_name": ..., "auth_url": ...},
        {"username": ..., "password": ..., "tenant_name": ..., "auth_url": ...},

2. Refactor the Scenario runner.
The scenario runner should now support different workflows for the two cases:
    a) "admin" role: the scenario runner should generate temporary tenants/users and exploit them while launching benchmark scenarios;
    b) "user" role: the scenario runner should just initialize OpenStack clients for the given set of users and then use them during benchmarking.

3. Modify the config validation procedure.
The diffentiation between the two roles ("admin"/"user") implies that the benchmark config should be different as well for the cases when users are generated before benchmarking and when they are not. Precisely, the "tenants" and "users_per_tenant" now make no sense when the users are passed explicitly and thus should be absent from the config in this case.

4. Refactor the dummy engine.
Dummy engine refactoring will be the only change in this blueprint noticeable for the end-user. Its input config format should now be changed to get similarly to the one described in step 1.

5. Implement and add the "admin" decorator to benchmark scenarios
The potential problem is now that the user may use a "user-role" deployment but launch the benchmark scenarios which require access to admin OpenStack clients (e.g. the Keystone scenarios). This behaviour should be prevented by explicitly adding an "@admin" sign to such benchmark scenario methods; The scneario runner will then be able to recognize the scenarios it cannot run in the "user" mode.

Blueprint information

Boris Pavlovic
Mikhail Dubov
Mikhail Dubov
Series goal:
Milestone target:
Started by
Boris Pavlovic on 2014-01-16
Completed by
Boris Pavlovic on 2015-10-22

Related branches



Gerrit topic:,topic:bp/benchmarking-with-predefined-users,n,z

Addressed by:
    Add support for user/admin roles to the deploy object

Addressed by:
    Refactor unit tests for dummy engine

Addressed by:
    Introduce the Endpoint object

Addressed by:
    Refactor the Scenario runner

Addressed by:
    Modify the config validation procedure

Addressed by:
    Refactor the dummy engine

Addressed by:
    Implement ScenarioRunner execution types via inheritance

Gerrit topic:,topic:bp/benchmark-context-predefined-users,n,z

Addressed by:
    Improve @base.scenario to specify it is admin only


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.