Performance & convenience improvements to integration tests, part1

Registered by Timur Sufiev on 2015-12-23

As integration tests are now stable enough, the next step is to make them pass faster and their internal architecture to be less redundant.

While the number of tests is relatively small, the major part of time spent running dsvm-integration job is due to preparing devstack environment. As number of tests grow (and hopefully, it will), the fraction of time spent actually running tests will increase and then the performance issues will arise. Let's fix some of these issues before they hit us.

Also there is a number of items (such as simplifying selectors) eluded from the scope of previous blueprint, integration-tests-hardening, time to address them now.


1. Every table wrapper currently holds both selector and table name. Table selector could be easily calculated using table name, so 2 should be reduced to 1.

2. [MOVED OUT OF THE SCOPE OF THIS BLUEPRINT] The constructors for almost every region and control element in testing Page Objects are put inside method marked with @property decorator, which means that every time you access that element or a region, its constructor is called again, which may take a significant time for complex regions including many nested elements. Caching the results of such constructors would save us a lot of time running tests. On the other hand, in some cases (especially, if some fields appear only after providing an input in previous fields) it may be desired to re-run constructors, so the caching feature should be switchable (by means of config). Also, the proposed approach is to first add the caching facility and then start adding it gradually to different categories of properties.

3. Table columns are matched by their index. This will require to change tests when additional columns are inserted into the beginning of a table. To avoid this, columns should be matched by their id/name and not by index.

4. Existing Page Objects have a very serious flaw: many forms/views that appear on them as a result of some actions being initiated by user/test are put inside @property and thus seem to be accessible all the time. This creates false expectations for the test writers who will be using these Page Objects and makes the code using these @properties quite fragile (as the success of using such properties depends on some previous actions being done). Instead of this we need to return such Page Regions as a result of table/row actions and never put them inside Page Object properties. To achieve this goal, current implementation of actions in integration tests has to be heavily reworked. As a final outcome we'll have each action defined explicitly as a method (so we could put the logic for creation new Page Regions into it) and being tied to an HTML control element by means of @decorators.

5. For every passing test we should also ensure in test that an expected success message was shown and no error messages were shown. This check could be combined with expected messages dismissing in order to make tests pass faster (no need to wait until the message is auto-dismissed).

6. Since is replaced with submit() action, it seems natural to put waiting for spinner to vanish (which always happens after submitting a form) into this action, thus reducing the amount of boilerplate .wait_for_spinner_to_disappear() calls.

The end user experience should be unaffected.

Wireframes, Mocks, Videos and UI Markup


Outside Dependencies

Requirements Update Required

Doc Impact
The changes will be reflected in an upcoming documentation for integration tests writers. Actually, the necessity to document some of idiosyncrasies of existing tests led me to an idea of rewriting the tests machinery (easier to fix it than to explain it).

Blueprint information

David Lyle
Timur Sufiev
Timur Sufiev
Series goal:
Milestone target:
Started by
Timur Sufiev on 2016-01-06
Completed by
Timur Sufiev on 2016-01-24

Related branches



Gerrit topic:,topic:bp/integration-tests-improvements-part1,n,z

Addressed by:
    Move integration_tests.tests.decorators one level up

Addressed by:
    Simplify tables page regions

Addressed by:
    Add @cached_property decorator to integration tests

Addressed by:
    In integration tests reference table columns by name

Addressed by:
    Rework table actions in integration tests

Addressed by:
    Reduce number of calls to Selenium for form fields

Addressed by:
    Improve working with messages in integration tests

Addressed by:
    Don't wait for spinner to disappear explicitly in i9n tests

Addressed by:
    Remove redundant dashboard.login_url setting from horizon.conf

Addressed by:
    Introduce field mappings for FormRegion descendants in i9n tests

I'm moving (2) @cached_property out of the scope of this blueprint, because possible performance gain is not clear, while complexity is obviously increased.


Work Items

This blueprint contains Public information 
Everyone can see this information.


No subscribers.