Testing in a Cadence

Registered by Nicholas Skaggs on 2012-10-08

Define and identify requirements for participation and reporting of test results on a regular basis during the cycle.

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Essential
Drafter:
None
Direction:
Approved
Assignee:
Nicholas Skaggs
Definition:
Approved
Series goal:
Accepted for raring
Implementation:
Unknown
Milestone target:
milestone icon ubuntu-13.04

Related branches

Sprints

Whiteboard

Quick history of 'cadence testing'

    started in quantal cycle, intended as experiment to help 'assure quality;

    https://lists.ubuntu.com/archives/ubuntu-devel/2012-June/035401.html

Recap of quantal cycle results

    http://www.theorangenotebook.com/2012/08/the-grand-cadence-experiment.html

Blueprints and sessions informing this one:
http://summit.ubuntu.com/uds-r/meeting/21405/foundations-r-schedule/
http://summit.ubuntu.com/uds-r/meeting/21047/foundations-r-prior-release-feedback/
Issues faced last cycle:
visibility -- tools built around milestones, don't show much visibility on a cadence basis
schedule -- confusing, shared calendar
stability -- non-developer centric scheduling and testing; code was caught in an untestable state during planned testing
Discussion:
Review decisions and new schedule for Raring
-- https://wiki.ubuntu.com/RaringRingtail/ReleaseSchedule
Tool needs and discussion

    should be able to use qatracker as-is

    other tools / report needs?

Defining the cadence process

    testing every other week. Once every 2 weeks, for a week long period, manually run testcases and report results. Development will not stop, no freezes of any kind will occur. Images will continue to roll out once a day during the week, and testing will occur independantly throughout the course of the week.

    Should define as Sunday - Saturday?

    Request Friday - Thursday as that gives weekend at the start for in order that most people available are available at the start.

    Should publish calendar seperate to release schedule?

    testing schedule with intending testing?

    Summary of these results should feed into the Quality Report (http://summit.ubuntu.com/uds-r/meeting/21082/qa-r-weather-report/)

    What is needed during the testing week to communicate status or progress for everyone?

ISOs:

    Create a milestone cadence name, with all of the dates filled in for the week. Default to the history view so all the testing and results can be seen. Additionally, a report view will be generated for each cadence week.

    Example:

    http://packages.qa.dev.stgraber.org/qatracker/milestones/227/history

    http://packages.qa.dev.stgraber.org/qatracker/reports/defects

single version during cadence week
Packages:

    Should we target specific packages for testing inclusion as part of cadence weeks?

    Try to align with known landing points?

    Work with team leads as releases happen, testing after landing in archive is 'ok'

    Workflow can mirror iso workflow as above

How does the milestone affect cadence?

    Run milestone the same as before

    Make sure cadence

(?)

Work Items

Work items:
[nskaggs] Create seperate calendar, listing testing details as we are informed. Each cadence week points to wiki page, with specifics.: DONE
[nskaggs] Communicate changes to release schedule: DONE
[nskaggs] Confirm a weeks worth of images remain availible from cadence weeks for debugging: DONE
[nskaggs] ensure installer has testing focus before beta, to ensure it's in good shape for beta milestone (can't fix bugs past freeze): DONE

This blueprint contains Public information 
Everyone can see this information.