Write selenium tests for Horizon running in a Devstack

Registered by Kieran Spear

We have unit tests with many many lines of mox stubbing. It'd be nice to be able to write selenium tests without bothering with all of that. This blueprint proposes the use of Selenium in integration tests run against a devstack, so we can catch higher-level integration issues earlier and have a layer of defence against missing unit tests or incorrect stubs.

Blueprint information

Status:
Complete
Approver:
Gabriel Hurley
Priority:
Medium
Drafter:
None
Direction:
Approved
Assignee:
Julie Pichon
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon 2014.1
Started by
Julie Pichon
Completed by
Matthias Runge

Related branches

Sprints

Whiteboard

Selenium tests also give us an easy way to generate screenshots for user documentation.

[gabriel] -- I fully agree, however I have thoughts on different directions this could go in terms of implementation. We should discuss at the testing session during the summit.

[jpichon 2013.10.09] If I recall correctly, there was no time to go into that topic during the last summit. Either way, it would be great to see notes on the blueprint as to the implementation ideas people had/have in mind.

Icehouse summit session: https://etherpad.openstack.org/p/icehouse-summit-Integration-Testing-for-Horizon . Decision: The Horizon functional tests will live in-tree to begin with, although the end goal remain to gate all projects on it.

Wiki: https://wiki.openstack.org/wiki/Horizon/Testing/UI

[jpichon 2013.12.19] Wiki update with information on the Page Object Pattern, which we're planning on using for the tests.

[jpichon 2013.01.10] The current plan is to have the "infrastructure" bits done by I2, so that the tests can be run by people and a potential jenkins job. In I3 and forever more, more tests will be written.

[tnovacik 2014.05.29] Some sort of languge layer should be created because currently there is language specific code although horizon supports changing language. Common web ui regions should be seperated into module, so they can be reused in page objects (tables, sidemenu, forms, etc...). I think that testing horizon plugins should be seperated from general horizon testing, because plugins are not installed by default on gates. I would suggest creating seperate folder for plugin tests and add switch to run_tests.sh script that would start plugin testing (for example sahara integration tests).

[tnovacik 2014.06.23] It might be good idea to create screenshots and download page source in case of error. (Better debugging especially in case of random failures)
[jpichon 2014.06.23] Feel free to open another blueprint to track this work, thanks!

Gerrit topic: https://review.openstack.org/#q,topic:bp/selenium-integration-testing,n,z

Addressed by: https://review.openstack.org/66012
    Integration tests - running the tests

Addressed by: https://review.openstack.org/77425
    Integration tests - page objects pattern

Addressed by: https://review.openstack.org/85607
    Changes in assert message and select dropdown

Addressed by: https://review.openstack.org/85656
    Add user and password parameters to _do_login

Addressed by: https://review.openstack.org/85670
    Add go_to_home_page method and two more properties

Addressed by: https://review.openstack.org/85685
    Integration tests - base class for all tests

Addressed by: https://review.openstack.org/95370
    Settings page ChangePassword page and their tests

Addressed by: https://review.openstack.org/96111
    Adds keypair create,delete tests

Addressed by: https://review.openstack.org/99329
    Adds securitygroup create,delete integration tests

Addressed by: https://review.openstack.org/100493
    Add heading property to BasePage class

Addressed by: https://review.openstack.org/101169
    Adds floatingip allocate,release integration tests

Addressed by: https://review.openstack.org/101177
    Adds container create/delete integration tests

Addressed by: https://review.openstack.org/101951
    Add translation layer from django

Addressed by: https://review.openstack.org/102227
    Add regions module

Addressed by: https://review.openstack.org/102750
    Fix typo in calling is_the_current_page method

Addressed by: https://review.openstack.org/102805
    Edit use heading property for page validation

Addressed by: https://review.openstack.org/106337
    Add new directory structure for page objects

Addressed by: https://review.openstack.org/107281
    Fix typo in select methods definition

Addressed by: https://review.openstack.org/109278
    Add form regions

Addressed by: https://review.openstack.org/109299
    Add menu regions

Addressed by: https://review.openstack.org/109991
    Adds router create/delete integration tests

Addressed by: https://review.openstack.org/110425
    Add navigation among pages

Addressed by: https://review.openstack.org/110896
    Add test skip decorator when service not available

Gerrit topic: https://review.openstack.org/#q,topic:bug/1350408,n,z

Addressed by: https://review.openstack.org/110938
    Verify Help button redirects to the right URL

Addressed by: https://review.openstack.org/110919
    Add integration tests for Admin->Projects page

Addressed by: https://review.openstack.org/112406
    Add wait_till_text_present_in_element method

Gerrit topic: https://review.openstack.org/#q,topic:bug/1354017,n,z

Addressed by: https://review.openstack.org/112579
    Integration tests - BaseWebElement class

Addressed by: https://review.openstack.org/112586
    Add workaround for StaleElementReferenceException

Gerrit topic: https://review.openstack.org/#q,topic:bp/horizon-sahara-module-selenium-integration-tests,n,z

Addressed by: https://review.openstack.org/112631
    Add test_image_register_unregister

Gerrit topic: https://review.openstack.org/#q,topic:bug/1354824,n,z

Addressed by: https://review.openstack.org/113131
    Integration tests - switch_window method

Addressed by: https://review.openstack.org/113830
    Add test_create_delete_job_binary test method

Gerrit topic: https://review.openstack.org/#q,topic:bug/1353334,n,z

Addressed by: https://review.openstack.org/115092
    Add multi user test possibility to integr. tests

Gerrit topic: https://review.openstack.org/#q,topic:my-patch-set-1,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1353628,n,z

Addressed by: https://review.openstack.org/117619
    Updating CSS locators to ID and LINK_TEXT. Partially implements blueprint: selenium-integration-testing Partial-Bug: #1353628

Gerrit topic: https://review.openstack.org/#q,topic:selenium-integration-testing,n,z

Addressed by: https://review.openstack.org/121406
    Fix locator typo in go_to_accesssecurity_page

Addressed by: https://review.openstack.org/121506
    Integration tests - Imagespage and test

Gerrit topic: https://review.openstack.org/#q,topic:menu_fix,n,z

Gerrit topic: https://review.openstack.org/#q,topic:navigation_reb,n,z

Addressed by: https://review.openstack.org/126478
    Add TabbedFormRegion to Integration tests

Addressed by: https://review.openstack.org/133095
    Integration tests - 3 new form fields

Addressed by: https://review.openstack.org/135273
    Integration test for fix the logout issue in basepage

Addressed by: https://review.openstack.org/139028
    Integration tests - networkspage and test

Gerrit topic: https://review.openstack.org/#q,topic:bug/1399268,n,z

Addressed by: https://review.openstack.org/139246
    Fix dropdown menu does not open with firefox 34

Gerrit topic: https://review.openstack.org/#q,topic:bp/launch-instance-selenium,n,z

Addressed by: https://review.openstack.org/140618
    Add "login as admin" for integration tests

Addressed by: https://review.openstack.org/141651
    Integration tests - Enable to test with chrome

Gerrit topic: https://review.openstack.org/#q,topic:bp/volumes-selenium,n,z

Addressed by: https://review.openstack.org/119501
    Integration tests - Volumes tests

Gerrit topic: https://review.openstack.org/#q,topic:bug/1399219,n,z

Addressed by: https://review.openstack.org/141906
    Test's random name gen now uses uuid4

Addressed by: https://review.openstack.org/142417
    Flavors page and basic create flavor test

Addressed by: https://review.openstack.org/143027
    Add integration tests for Creating user

Addressed by: https://review.openstack.org/143295
    Using get_url_current_page wrapper method

Gerrit topic: https://review.openstack.org/#q,topic:bug/1404379,n,z

Addressed by: https://review.openstack.org/143191
    Fix wrong calling of selenium driver function

Gerrit topic: https://review.openstack.org/#q,topic:bug/1404416,n,z

Addressed by: https://review.openstack.org/143224
    Fix method passed to selenium way it suggests

Gerrit topic: https://review.openstack.org/#q,topic:bp/multi-region-integration-test,n,z

Addressed by: https://review.openstack.org/143587
    Integration test - support login multiple regions

Addressed by: https://review.openstack.org/144206
    Add general function for generating random names

Addressed by: https://review.openstack.org/144695
    Decorator for skipping tests hitting known bugs

Addressed by: https://review.openstack.org/145936
    Modified teardown method to handle login and logout failure.

Addressed by: https://review.openstack.org/146139
    Integration_tests - add EmailInputFormFieldRegion

Addressed by: https://review.openstack.org/146143
    Integration_tests - add EmailInputFormFieldRegion

Addressed by: https://review.openstack.org/147119
    Integration tests - Run test case under any user

Gerrit topic: https://review.openstack.org/#q,topic:bug/1411028,n,z

Addressed by: https://review.openstack.org/147339
    Add page forward and refresh wrapper functions

Addressed by: https://review.openstack.org/147426
    Integration tests - Run test with IE

Addressed by: https://review.openstack.org/147557
    Integrtion test - Run test as user just created

Addressed by: https://review.openstack.org/147721
    Integration test - Setup Test case using a new user

Addressed by: https://review.openstack.org/117091
    Integration tests - Instances tests

Gerrit topic: https://review.openstack.org/#q,topic:bug/1412469,n,z

Addressed by: https://review.openstack.org/148300
    Adding -ac and -noreset arguments to Xvfb

Gerrit topic: https://review.openstack.org/#q,topic:bug/1413616,n,z

Addressed by: https://review.openstack.org/149299
    Testing a remote openstack deployment

Gerrit topic: https://review.openstack.org/#q,topic:unused-wait_for_title,n,z

Addressed by: https://review.openstack.org/149565
    Remove unused wait_for_title method

Addressed by: https://review.openstack.org/149303
    Fix editing current project

Addressed by: https://review.openstack.org/151177
    Adds router create/delete integration tests

Gerrit topic: https://review.openstack.org/#q,topic:bp/integration-tests-object-store,n,z

Addressed by: https://review.openstack.org/151187
    Integration tests - Object store

Addressed by: https://review.openstack.org/151608
    Wait till spinner is visible

Addressed by: https://review.openstack.org/151638
    Add project-dropdown

Addressed by: https://review.openstack.org/151796
    Add wait till element methods for selenium Integration tests.

Addressed by: https://review.openstack.org/153084
    Fix typo in calling _is_element_visible method

Addressed by: https://review.openstack.org/154965
    Rename helpers.py to basetestcases.py

Addressed by: https://review.openstack.org/155352
    Use IDs in locators instead of CSS when it is possible

Addressed by: https://review.openstack.org/156604
    Security groups page and test

Gerrit topic: https://review.openstack.org/#q,topic:bug/1424344,n,z

Addressed by: https://review.openstack.org/158114
    Add log lines change to settings page and test

Addressed by: https://review.openstack.org/158461
    Add html source code on test failure for debugging

Addressed by: https://review.openstack.org/158663
    Adds floatingip allocate,release integration tests

Gerrit topic: https://review.openstack.org/#q,topic:bug/1425250,n,z

Addressed by: https://review.openstack.org/158825
    Use _wait_for_spinner()

Gerrit topic: https://review.openstack.org/#q,topic:bug/1425476,n,z

Addressed by: https://review.openstack.org/159084
    Avoid failure when spinner is already gone

Addressed by: https://review.openstack.org/159237
    Fix skip decorator in integration tests

Addressed by: https://review.openstack.org/159869
    New generic _wait_until method which wraps selenium waiting

Addressed by: https://review.openstack.org/162646
    Remove hostname from the url

Addressed by: https://review.openstack.org/164065
    Updated _tab_locator in menus and Added 'submit' method for TabbedForms: In forms.py: In class TabbedFormRegion: * Added submit method for tabbed for forms In menus.py: In TabbedMenuRegion: * changed the _tab_locator to (by.By.CSS_SELECTOR, 'li > a

Gerrit topic: https://review.openstack.org/#q,topic:bug/1433123,n,z

Addressed by: https://review.openstack.org/165102
    Use default values in horizon.conf

Addressed by: https://review.openstack.org/168935
    Add integration tests for Identity->Projects page

Addressed by: https://review.openstack.org/170581
    FIX integration tests

Addressed by: https://review.openstack.org/172858
    Use gen_resource_name in create/delete user test

Addressed by: https://review.openstack.org/177233
    WIP:Integration test - create provider netowrk

Addressed by: https://review.openstack.org/178924
    reuse a working firefox profile

Gerrit topic: https://review.openstack.org/#q,topic:bug/1461118,n,z

Addressed by: https://review.openstack.org/187600
    Limit max length of gen_random_resource_name

Addressed by: https://review.openstack.org/187605
    Limit max length of gen_random_resource_name

Addressed by: https://review.openstack.org/190076
    Refactor of BaseTestCase

Addressed by: https://review.openstack.org/190077
    Refactor of BaseTestCase

Addressed by: https://review.openstack.org/190596
    Fix for test_image_create_delete

Addressed by: https://review.openstack.org/194646
    Add _save_screenshot functionality to helpers.

Addressed by: https://review.openstack.org/204583
    Fix _wait_till_element_disappear

Addressed by: https://review.openstack.org/207491
    Fix table _get_rows() - always return None

Gerrit topic: https://review.openstack.org/#q,topic:bug/1463729,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1491309,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/integration-tests-hardening,n,z

Addressed by: https://review.openstack.org/237342
    Fixed the method name reference in instancespage

Addressed by: https://review.openstack.org/240620
    Handle id_pattern for instances_secondary_row_actions.

Addressed by: https://review.openstack.org/261576
    [wip] select options support

Gerrit topic: https://review.openstack.org/#q,topic:additional-i9n-tests,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/horizon-integration-tests-coverage,n,z

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.