Improve Horizon testing

Registered by Ivan Kolodyazhny on 2018-04-04

Summary
=======
Improve Horizon testing using Selenium and implement cross-project job to test
Horizon changes over plugins.

Motivation
==========
We need better test coverage both for Horizon itself and plugins. We should be
sure that submitted patches won't break anything in Horizon or in any plugin.

Description
===========
We don't have Selenium-based tests now. the selenium-headless job always reports
success. Integration tests are totally broken and we even don't run them on
gates. We need to fix selenium-headless job and integration tests too.

It would be great to have new gate job per each plugin described at [1] per any
Horizon code change to be sure that we don't break anything. The same job with
plugin-specific selenium or integration tests should be executed against each
Horizon plugin's change request.

Action Items
============
* Get a rid of Nose test runner: it's in a maintainance mode for the laset two
  years and doesn't work as expected [2]
* To not break any plugin we need to remove
  ``TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'` from each plugin to
  be sure that it works without nose
* Use django.test.tag to split tests into the following groups: unit, selenium,
  integration. We use environment variables for it now. It leads to having a
  lot of skipped tests per each test run. It makes troubleshooting more complicated.
* Add new gate jobs described in the 'Testing' section of this blueprint.
* Update `Contributing Guide` if needed

Testing
======
* Selenium job should be fixed
* Basic functional tests framework should be implemented
* Add jobs for each supported Horizon plugin to Horizon gates
* Add Horizon integration job to plugins gates

NOTE: All new jobs should be enabled in non-voting mode by default. It's up to
each team to make this job voting once it becomes stable.

Doc Impact
=========
Current `Contributing Guide` [3] looks corret but we need to check it

Links
=====
[1] https://docs.openstack.org/horizon/latest/install/plugin-registry.html
[2] https://github.com/nose-devs/nose/issues/984
[3] https://docs.openstack.org/horizon/latest/contributor/contributing.html
[4] https://etherpad.openstack.org/p/horizon-ptg-rocky

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Ivan Kolodyazhny
Direction:
Needs approval
Assignee:
Ivan Kolodyazhny
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.