Reporting the state of quality in ubuntu

Registered by Nicholas Skaggs on 2012-10-08

NOTE: This has no relation to the current Weather Report :-)
Create a realtime report that will convene the 'quality' of the ubuntu archive at any point during the development cycle. Summaries from manual and automated testing should be included, as well as pending releases and manually targetted ares to focus.

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Essential
Drafter:
None
Direction:
Approved
Assignee:
Nicholas Skaggs
Definition:
Approved
Series goal:
Accepted for raring
Implementation:
Unknown
Milestone target:
milestone icon ubuntu-13.04

Related branches

Sprints

Whiteboard

[mpt] Why are the QA Dashboard, the QA Weather Report, and the Defects Dashboard three different things? That would make it less likely that people would know about the ones that would help them.

During the course of the cycle it is difficult to understand the quality of the development version of ubuntu. While it can sometimes be easy to know of critical bugs (because you and everyone else running ubuntu+1 are experiencing the error :-) ), in general problems often creep up unnoticed, or are not realized until too late. In addition, it is sometimes difficult to know when and how to foucs testing efforts, both on a coordinated and individual basis.
Potential Data Sources:
QA Dashboard
Reports like:
http://reqorts.qa.ubuntu.com/reports/foundations-bugs/foundations-bugs-recent-bug-ta10sks.html
or launchpad directly
Jenkins
Weekly Release Meeting Summary
Community Feedback (mailing list, forum, IRC, bug reports)
QATracker
errors.ubuntu.com / whoopsie dasiey?
Package status reports:
http://packages.qa.ubuntu.com/qapkgstatus
Lintian (http://lintian.ubuntuwire.org/)
Piuparts (not sure if we're running this ATM)
Debcheck (http://qa.ubuntuwire.com/debcheck/) <-- this is partially mitigated by the new proposed -> release migration
Report Features
Packages needing focused?
confidence level and thumbs up / thumbs down
see: work: https://dev.launchpad.net/DefectsDashboard
test and coverage of critical packages.
how well exercised the packages are - bugs only make sense in context of how exercised the package may be.
Per package level reporting
Per image reporting
correlate package level data to images?
get an initial scale..
include archs
Correllated image - plars, kate.stewart interested in core set of images.
Paul is interested in thresholds rather than absolute.
Manual side to occur on weekly basis, daily for automated.
Use QA Dashboard, not wiki

(?)

Work Items

Work items for ubuntu-13.04-month-6:
[nskaggs] Launch initial version of contributions statistics for manual and automated testing: DONE
[nskaggs] Integrate initial work onto QA dashboard: INPROGRESS

Work items:
[nskaggs] aggregate daily status from jenkins automated tests per image per architecture as a page - red/yellow/green? other? into a prototype.: POSTPONED
[nskaggs] broadcast prototype around and call feedback session early in cycle.: DONE
[nskaggs] review work from Q cycle, and determine what might be applicable.: DONE

This blueprint contains Public information 
Everyone can see this information.