Community exploratory testing

Registered by Nicholas Skaggs on 2013-11-06

For the 14.04 cycle we want to encourage as much exploratory testing as possible. On the phone, tablet and desktop -- everywhere. Let's discuss ideas, plans, needs, etc to ensure this happens.

Review current efforts:
https://wiki.ubuntu.com/QATeam/TouchTesting
https://wiki.ubuntu.com/QATeam/Roles/Tester#Exploratory_Testing

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Undefined
Drafter:
Nicholas Skaggs
Direction:
Approved
Assignee:
Nicholas Skaggs
Definition:
New
Series goal:
Accepted for trusty
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

[amjjawad] If I may share what I have in mind, I think we can agree eventually on some points and set some plans. While it is no secret that Ubuntu and Canonical is focusing their efforts of the Phones and Tables and from the mailing list, I see that very clear, specially the last plan/decision about Ubuntu 14.04 to go for GNOME 3.8. Anyway, I feel there are some important points we should not forget:
1- As much as the phone and table projects are important, let's not forget that Ubuntu is also for PCs and Laptops
2- I strongly suggest to have dedicated group of people for QA/Testing, Bugs, Devs, etc for the i386 and amd64 ISOs. The focus should not be on some aspects and ignore or forget about the others
3- We need a clear plan/roadmap for this cycle (14.04) and we need Plan B so if Plan A will fail, we go automatically to Plan B. I know this is eventually what should happen but I have seen lots of mistakes the previous cycle so hope we can avoid that.

By having dedicated groups for each product, we can ensure successful results in less time as everyone knows what he/she will work on and what to expect etc.

Also, we need to recruit as much new testers as possible. Those who have no experience with testing are the best candidate but then the challenge is who will mentor them or help them to learn? it is a challenge that we need to face and deal with. You need more testers? you should be ready to teach them. Wiki Pages won't be helpful for a newcomer. Videos? yes, that would be helpful a lot.

There are lots of things we can discuss to encourage any kind of test.

---

Exploratory testing

Testing a development release, enourage the culture of exploratory testing, report a bugs.

https://wiki.ubuntu.com/QATeam/TouchTesting
Dogfooding the first phone release:

- flash the device
- look for bugs
- (potentially) write autopilot tests

https://wiki.ubuntu.com/QATeam/Roles/Tester#Exploratory_Testing

Prioritizing between ISO/package testing and exploratory testing (came up on yesterday's meeting)

Install development release, go wild, try to break things, look for bugs and report them.

No specific tests or guidelines, focus on exploratory testing.

Communicate to the community.

Explain what's exploratory testing:
    - anything that isn't part of the testcases !
    - definition: "simultaneous learning, test design and test execution"

Tips and tricks.

Metrics? Reporting?
bug reports aren't necessarily the best metric

    still a metric

number of people who tested and what and when

    how do collect that info?

James Bach (awful), James Whittaker

    Whittaker: taking a tour of an application

video: https://www.youtube.com/watch?v=fNkYz1hB7r0 (large scale exploratory testing - James Whittaker)
book: http://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques/dp/0321636414

exploratory testing is easier for new folks, keeps things fresh, excellent for new contributors

What are the outputs?
Bug reports, testing session log.
Mailing list & IRC channel activity --> ultimately should lead to bug reports?
If you don't take notes, you might not be able to reproduce a bug.
How notes will look like?
Are they useful for other people?
Screencapturing?
Can we use the packages tracker for reporting results (create an "exploratory testing" product, and testcases as needed)?

    sure, the testcase could simply be a description

should we share what we tested? Do we need more feedback on what's being tested besides bugs?

Define a tag for the bugs reported during exploratory testing. If you use the tracker, you automatically get a tag.
Make a summary with the bugs reported.

Implementation Ideas:
use qatracker
    needs a SSO account, requires login
    can link to specific page where the reporting form is -> users do not necessarily need to navigate
    we can modify the tracker to create a simpler page (just bribe stgraber with milk+cookies)

use a spreadsheet
    where? google docs? (requires a google account)

use wiki
    the ubuntu wiki is slow, on some days really slow, so it's not really a good alternative

use pad
    the ubuntu etherpad needs a SSO account as well and a membership on the -etherpad team

use +1 page

    record bugs, notes, time spent

    sounds like the tracker, but maybe a bit simpler

    needs creating

    why reinvent the wheel?

cgoldberg and knome offer to help with tracker :-)

(?)

Work Items

Work items:
[nskaggs] communicate wiki pages, definitions, goals, etc: TODO
[nskaggs] include methodology on wiki pages: TODO
[nskaggs] make a video: TODO
[elopio] make a summary of the exploratory testing chapter on the Agile Testing book.: TODO
[nskaggs] mockup how it could work on the tracker and share: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.