Make Horizon integration tests less fragile
Summary
=======
To make Horizon integration tests less fragile and reduce amount of random breakages.
Motivation
=========
Integration tests are hardly useful when they break due to any minor change in Horizon page structure and styling issues. The fragility comes from several sources, partly because tests are not written good enough, partly because of underlying OpenStack issues. The point of this blueprint is to identify where the integration tests are not good enough and make them more durable (less tied to page structure) so they catch the actual OpenStack/Horizon problems.
Description
=========
There are several things that make tests more fragile:
1. Matching the form fields based on their ordering in actual Horizon forms. With the addition of new fields, changing the ordering of existing fields tests break - which is not desired. Instead they should catch only the fact that previously existing fields disappear (and thus some scenario should be rewritten). This can be achieved by referencing tied test fields to actual fields by their 'name' attribute (which every field has) and not by their position.
2. Same holds true for table actions. Every table action (as Horizon entity) has a name, which can be used in tests.
3. Checking for the entity presence in a table by its display name. UX improvements made recently to Horizon include name truncation (given it's too long) - so we should first match against additional attribute of a cell which contains original cell value, and if it's missing fall back to its display value.
4. The most difficult part is to untie the CSS and XPATH selectors used in tests from a specific page layout in Horizon. Everyone who is altering page layout and doing some theming should understand what causes integration tests to break after their changes. The idea here is to introduce an orthogonal hierarchy of CSS classes to which tests are tied. These classes are not used for Horizon theming/layout construction and should not be changed. The tests' selectors SHOULD NOT rely on page structure, they should rely SOLELY on these classes - that way they would be agnostic of page layout.
5. Reduce non-determinism with Selenium timeouts when trying to get some specific element (due to the element being completely replaced and we're trying to take something out of a destroyed box).
UX
===
The end user experience should be unaffected.
Wireframes, Mocks, Videos and UI Markup
-------
N/A
Testing
======
No additional tests are provided besides the ones that are being fixed.
Outside Dependencies
==================
N/A
Requirements Update Required
=======
N/A
Doc Impact
=========
N/A
Blueprint information
- Status:
- Complete
- Approver:
- David Lyle
- Priority:
- High
- Drafter:
- Timur Sufiev
- Direction:
- Approved
- Assignee:
- Timur Sufiev
- Definition:
- Approved
- Series goal:
- Accepted for mitaka
- Implementation:
-
Implemented
- Milestone target:
-
mitaka-1
- Started by
- David Lyle
- Completed by
- Timur Sufiev
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Integration tests no longer use form's fields ordering for matching
Addressed by: https:/
Integration tests no longer rely on entities' display names in tables
Addressed by: https:/
Adapt integration tests to recent devstack and page layout changes
Addressed by: https:/
Reduce the chance of Selenium timeouts in integration tests
Gerrit topic: https:/
Gerrit topic: https:/
Addressed by: https:/
Do not match table actions by ordering in integration tests
Addressed by: https:/
Integration tests - Instances tests
Gerrit topic: https:/
Gerrit topic: https:/
Addressed by: https:/
Wait until additional window is opened in integration tests
Addressed by: https:/
Fix accidentally broken Instances integration test
Addressed by: https:/
Integration tests - Instances tests-VNC
Gerrit topic: https:/
[tsufiev 02.12.15] I think we can mark this one as Implemented since (4) is not going to be done in the nearest future. We have higher priority things to do in order to improve integration tests, coverage is the first one. The tests were already made voting, so if any other serious stability-breaking issue is found, it'll be filed as a red-code bug.