Make Horizon integration tests less fragile

Registered by Timur Sufiev on 2015-08-19

Summary
=======
To make Horizon integration tests less fragile and reduce amount of random breakages.

Motivation
=========
Integration tests are hardly useful when they break due to any minor change in Horizon page structure and styling issues. The fragility comes from several sources, partly because tests are not written good enough, partly because of underlying OpenStack issues. The point of this blueprint is to identify where the integration tests are not good enough and make them more durable (less tied to page structure) so they catch the actual OpenStack/Horizon problems.

Description
=========
There are several things that make tests more fragile:

1. Matching the form fields based on their ordering in actual Horizon forms. With the addition of new fields, changing the ordering of existing fields tests break - which is not desired. Instead they should catch only the fact that previously existing fields disappear (and thus some scenario should be rewritten). This can be achieved by referencing tied test fields to actual fields by their 'name' attribute (which every field has) and not by their position.

2. Same holds true for table actions. Every table action (as Horizon entity) has a name, which can be used in tests.

3. Checking for the entity presence in a table by its display name. UX improvements made recently to Horizon include name truncation (given it's too long) - so we should first match against additional attribute of a cell which contains original cell value, and if it's missing fall back to its display value.

4. The most difficult part is to untie the CSS and XPATH selectors used in tests from a specific page layout in Horizon. Everyone who is altering page layout and doing some theming should understand what causes integration tests to break after their changes. The idea here is to introduce an orthogonal hierarchy of CSS classes to which tests are tied. These classes are not used for Horizon theming/layout construction and should not be changed. The tests' selectors SHOULD NOT rely on page structure, they should rely SOLELY on these classes - that way they would be agnostic of page layout.

5. Reduce non-determinism with Selenium timeouts when trying to get some specific element (due to the element being completely replaced and we're trying to take something out of a destroyed box).

UX
===
The end user experience should be unaffected.

Wireframes, Mocks, Videos and UI Markup
---------------------------------------------------------------
N/A

Testing
======
No additional tests are provided besides the ones that are being fixed.

Outside Dependencies
==================
N/A

Requirements Update Required
========================
N/A

Doc Impact
=========
N/A

Blueprint information

Status:
Complete
Approver:
David Lyle
Priority:
High
Drafter:
Timur Sufiev
Direction:
Approved
Assignee:
Timur Sufiev
Definition:
Approved
Series goal:
Accepted for mitaka
Implementation:
Implemented
Milestone target:
milestone icon mitaka-1
Started by
David Lyle on 2015-08-26
Completed by
Timur Sufiev on 2015-12-02

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bug/1467950,n,z

Addressed by: https://review.openstack.org/214238 -- merged
    Integration tests no longer use form's fields ordering for matching

Addressed by: https://review.openstack.org/214671 -- merged
    Integration tests no longer rely on entities' display names in tables

Addressed by: https://review.openstack.org/214670 -- merged
    Adapt integration tests to recent devstack and page layout changes

Addressed by: https://review.openstack.org/215101 -- merged
    Reduce the chance of Selenium timeouts in integration tests

Gerrit topic: https://review.openstack.org/#q,topic:bug/1467950-patch4,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/integration-tests-hardening,n,z

Addressed by: https://review.openstack.org/218301
    Do not match table actions by ordering in integration tests

Addressed by: https://review.openstack.org/117091
    Integration tests - Instances tests

Gerrit topic: https://review.openstack.org/#q,topic:bp/launch-instance-selenium,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/horizon-theme-css-reorg-login,n,z

Addressed by: https://review.openstack.org/229004
    Wait until additional window is opened in integration tests

Addressed by: https://review.openstack.org/236203
    Fix accidentally broken Instances integration test

Addressed by: https://review.openstack.org/241025
    Integration tests - Instances tests-VNC

Gerrit topic: https://review.openstack.org/#q,topic:bugs/#1511879,n,z

[tsufiev 02.12.15] I think we can mark this one as Implemented since (4) is not going to be done in the nearest future. We have higher priority things to do in order to improve integration tests, coverage is the first one. The tests were already made voting, so if any other serious stability-breaking issue is found, it'll be filed as a red-code bug.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.