Scalable installation testing

Registered by Matt Zimmerman on 2006-10-24

Currently, installation testing is coordinated by means of a wiki page containing several tables. This approach has a number of scalability problems, including contention on the wiki page, overhead of recording test results, and lack of awareness in the community.

In order to broaden testing participation, anyone who installs Ubuntu should be able to quickly and easily inform us about whether their installation succeeded or failed and which installation parameters they used. The QA team should be able to view reports of test coverage based on this data.

Blueprint information

Status:
Complete
Approver:
Matt Zimmerman
Priority:
Medium
Drafter:
Henrik Nilsen Omma
Direction:
Needs approval
Assignee:
Henrik Nilsen Omma
Definition:
Approved
Series goal:
Accepted for feisty
Implementation:
Implemented
Milestone target:
None
Started by
Matt Zimmerman on 2007-01-31
Completed by
Henrik Nilsen Omma on 2007-04-30

Related branches

Sprints

Whiteboard

review: Please incorporate the whiteboard comments. Did you identify any risks WRT doing this? WHat if there's no Internet connectivity? -- smurf

In order to maximize participation, we should not tie the tests to specific test cases. Any successful installation with network connectivity should be able to give us a report. -mdz, 2006-11-08

pitti: reviewed, whiteboard comments still need to be addressed, back to drafting

sfllaw: Addressed whiteboard comments, back to Review.

pitti, 2006-11-30: talked with mdz about his idea; made spec comments inline, set back to discussion

2007-01-26 kamion: Deferring for now after discussion with Simon since the spec isn't complete and this has interesting dependencies on a remote database somewhere; but if this makes great strides forward soon I'd be willing to un-defer it

2007-01-31 mdz: during the sprint, Henrik began work on an alternative approach which is simpler to implement and achieves many of the same goals. Hijacking this spec to track the status of that instead.

2007-01-31 heno: I've pointed the LP spec URL at the spec I wrote. The howto is here: https://wiki.ubuntu.com/Testing/ReportingResults

2007-02-20 heno: setting status to 'Pending Approval' though the implementation is ahead of the spec(s). Both have evolved over the Herd releases. Anyway, the 'Good progress' status is correct. I hope to trial a new system for Herd 5 and use it more widely for the Beta testing.

2007-02-22 mdz: Setting to Approved, this matches what we discussed in Oslo. We'll fine-tune based on experience with this trial.

2007-04-30: This worked quite well for the Feisty release in the end. We will extend and harden it for Gutsy: iso-test-tracker

(?)

Work Items