Flavors Quality Assurance

Registered by Nicholas Skaggs

The flavors now run independent development milestones if they wish (alpha, beta) in addition to taking advantage of the expanded packages tracker for manual testing. In addition, autopilot and autopkg tests are strong contenders to supplement testing needs if desired. Let's talk about how to get invovled with using the resources availible to enhance manual and automated flavor testing.

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Automated Testing
Autopilot - flavors
ubuntu-autopilot-tests

Autopkg
https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/
http://developer.ubuntu.com/packaging/html/auto-pkg-test.html
ubiquity

Manual Testing
packages tracker
writing testcases
manpower to test
-all flavors could use further testers
-recruiting is a big priority
-lots of support, but need to convert those into contributors
-focus on social media - increasing social media I can understand - you start focussing on it and you'll end up losing those who don't follow it
-simplify testing, more videos? Videos are a good idea - as long as they are an addition - not everyone works well from videos, some prefer the written word - and if you're testing something it's a lot easier from written in my opinion
-iso test a proper place to start? I think that we're likely to end up with 2 types of people - those who can work through iso testing AND packages, the other being people happy to test packages - might be as well to assume we'll have 2 types of tester and work from that assumption
    knome » ISO and package testing, are from my POV, have very different time commitment levels: ISO testing needs more time at once and heavily emphasizes around milestones, while single tests for packages are quicker and you can pick them up whenever you have time.
-target potential users from flavors, draw more into flavors from say windows xp sunsetting

wiki pages
ubuntu gnome
https://wiki.ubuntu.com/UbuntuGNOME/Testing

Packages tracker:
http://packages.qa.ubuntu.com/qatracker/milestones/300/builds
xubuntu desktop:
http://packages.qa.ubuntu.com/qatracker/milestones/300/builds/50384/testcases
lubuntu desktop:
http://packages.qa.ubuntu.com/qatracker/milestones/300/builds/50439/testcases

writing tests for your default apps
ubuntu-manual-tests
https://launchpad.net/ubuntu-manual-tests
testsuites
http://packages.qa.ubuntu.com
documentation I could take this as a work item - depending on how quickly it's wanted, given that I complain about it a lot :) (Elfy)

Couple of notes from smartboyhw as he can't join in the Hangout

The biggest problem is that some flavours have enough testers, some don't (for example, Ubuntu Studio). For us, we can't even afford to write the manual tests, since most of the team members just have no time. We'd probably be in the same place if I'd not mailed our team list - people responded to that request.
    knome » I don't think anybody can argue any flavor having "more than enough" testers. From my POV, this is an organizational issue: how do you organize testcase writing, and would you be willing to get a bit less testing during a timeframe when most of the testers are actually writing the tests. Furthermore, this is pretty much a one-off commitment, after which the maintaining burden is much smaller. We (Xubuntu) demonstrated it's possible to empower testers (as well as other people) to write testcases and get a huge bunch of them ready during one cycle. While it definitely affected our possibilities to run actual tests, I'm confident it'll pay off.
    smartboyhw >> The problem is that we still can't manage to get a full oraganization of testing working. Most of us are either onto coding (OvenWerks),
leading stuff, maintaining kernels (zequence), still trying to catch on (cub) or writing offline doc and making out how linux-rt works (me). I will try to have a serious discussion within our team.

As discussed in a session on Tuesday I believe it was for default apps of all 'flavours' (notice the correct spelling ;-p ) to be added to the autopilot list, but they will all be run on ubuntu rather than on each flavour. I personally don't know what all the default apps are across the flavours, but it would be handy if someone from each could create a list of apps that we can introspect.

To find this out all thats needed is to launch the app in a terminal with 'autopilot launch -i Gtk <app_name>' and check that the autopilot gtk interface loads and then in another terminal window run 'autopilot vis' and check it says root in the dropdown at the top.

If the interface loads and we can see the root object then we can create a todo bug. Rather than fill up the bug list with apps we can't do/blocked. Maybe use a wiki page much like the Core apps do, but listing what can and can't and maybe look to create autopkgtests for the ones we can't using their upstream testsuites??

(?)

Work Items

Work items:
[dpniel] add todo bugs for introspectable autopilot applications: INPROGRESS
[nskaggs] send post to mailing list for recruitment for more testers: TODO
[nskaggs] help co-ordinate campaign across all flavors for new tester recruitment: TODO
[smartboyhw] : Work with nskaggs and amjjawad and phillw to get promotion working: TODO
[phillw] forward nskagg's emails to lubuntu community: TODO
[phillw] Ensure classroom sessions that will run when Saucy is released are scheduled for about 2 weeks after Saucy is released so as to maximise the interest: TODO
[elfy] Work on documentation for packages.qa.ubuntu.com: TODO
[elfy] Introspectable default apps for Xubuntu, this was started previously -https://wiki.ubuntu.com/Xubuntu/Roadmap/Specifications/Saucy/AutopilotTesting : DONE

This blueprint contains Public information 
Everyone can see this information.