Improving crossplatform capabilities of current UI testing

Registered by Gema Gomez on 2012-10-18

With the need of doing UI testing in UE, we are looking for a set of existing test cases that can be used for that, unity test suite is an ideal candidate as a starting point.

Blueprint information

Status:
Complete
Approver:
Pete Graner
Priority:
Essential
Drafter:
Gema Gomez
Direction:
Needs approval
Assignee:
Canonical Platform QA Team
Definition:
Approved
Series goal:
Accepted for raring
Implementation:
Implemented
Milestone target:
None
Started by
Gema Gomez on 2012-11-16
Completed by
Gema Gomez on 2013-05-21

Related branches

Sprints

Whiteboard

There are two aspects of Autopilot, one is the tool and another is the current unity test cases. UE would like to reuse the existing test cases and use them in an integration testing manner. For this, the test cases will need to be reliable crossplatform, so it needs to be discussed if changes will be required for this to happen.

There are two aspects of Autopilot, one is the tool and another is the current unity test cases.

UE would like to reuse the existing test cases and use them in an integration testing manner.

For this, the test cases will need to be reliable crossplatform, so it needs to be discussed if changes will be required for this to happen.

Current status of the UI (unity) testing
358 test cases, they run daily+ on three different HW flavours (different graphic chipsets)

Problems identified (if any!)
- The failure rate is very high (25%-30% fail)
  * Some fail due to genuine bugs (we want to keep these)
    * some of this problems are hidden to users
  * Some of the issues are timing related
    e.g. User pushing super key, verify the dash opens
    eventually feature to tackle the issue, but still not reliable
  * little test isolation between tests (some things happen after the test has finished and cause trouble to the next test)
  * Poorly written test cases

99% of the work should be unit tests

Solutions
Unity tests are to be fixed in the coming 3 weeks. Goal is to use autopilot results as release criteria for unity
* unit tests for code checkins
* autopilot tests for submissions to -proposed

Autopilot + Xpresser can be used together

Profiling support for compiz to be added

Documentation
http://unity.ubuntu.com/autopilot/

(?)

Work Items

Work items for ubuntu-13.04:
[mrazik] to create a team of autopilot reviewers: DONE
[gema] Add a new BP for unity testability: POSTPONED
[thomir-deactivatedaccount] to publish the autopilot documentation: DONE
[nuclearbob] Make sure utah works with the autopilot tests: POSTPONED
[mrazik] To keep the platform QA team in the loop of the fixes, so that we know what to expect: DONE
[thomir-deactivatedaccount] Make the sure the test suite is stable on Nexus 7: DONE