Feedback on Precise Release and Improvements

Registered by Kate Stewart on 2012-04-09

Feedback session on what went well and what needs to be tweaked from our experiences in precise. Goal is to keep the experiments that did work, and adjust those parts that were still causing problems.

Blueprint information

Status:
Started
Approver:
Kate Stewart
Priority:
High
Drafter:
None
Direction:
Approved
Assignee:
Ubuntu Release Team
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Beta Available
Milestone target:
milestone icon ubuntu-12.10-beta-2
Started by
Kate Stewart on 2012-05-31

Related branches

Sprints

Whiteboard

Notes: copied in from pad.ubuntu.com on 2012/05/08
http://summit.ubuntu.com/uds-q/meeting/20298/other-q-prior-release-feedback/

What worked well
- new iso tracker (integration with builds, ease of adding images/test cases, localization teamsupport)
- new queue bot (accept/reject notification on queues)
- release note restructuring - better able to handle long term support and flavor specific audience needs. Thank yous to developers, testers, translators, etc.
 - common infrastructure worked well
- improved documentation on why we're respinning.
- unity/kernel didn't break much
- +1 team maintenance effective
- release usable every day. (after beta1 ubiquity issue)
- communication with Linaro worked quite well, with issues mostly raised in time

What needs improvement
- queue audit logs - launchpad bug - in progress background...
   - worked around with pad/wiki experiments... better ideas?
   Launchpad is open source, "somebody" can jfdi this if it's important
AI: discuss if we can find a resource to help (cjwatson over committed already)
 - can we open +1 before we are done the current development release?
  - Launchpad will run into some issues here
  - can put stuff in PPAs and other places instead so we don't stop work
 - can we open backports early for last release?
  - was supposed to happen for precise but didn't (why?)(because no one did the work in lp)
JFDI. who?

- policies need to be standardized on for use of -proposed around milestone release window.
   - discussion to happen in session:
    --> http://summit.ubuntu.com/uds-q/meeting/20290/other-q-freeze-use-of-proposed/

- Generic view of the test coverage we have on top of the daily results would be useful
- had some ubiquity problems after beta 1
- Having a view of what is going to land when, that we can see to start doing risk based testing would help tons, when our capacity to test is not at its best and we cannot cover everything every day
  - iso tracker suggestion to publish daily diff based on manifest.
    --> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-iso-tracker
? check if there's a bug on this, if not add.
2012/05/31 - was handled in iso-tracker session.

- [jbicha] It's confusing that the ReleaseSchedule has multiple days all listed with Thursday's date, such as:
April 12th | NonLanguagePackTranslationDeadline (Tue), FinalFreeze, LanguagePackTestRebuild (Fri)
Some folks thought they had an extra day or two to get the ubuntu-docs upload done before the Language Pack deadline because the format wasn't clear at first look. I propose splitting this events to separate lines
    - AI: go forward with format change [skaet]

- [cjwatson] I propose that we abolish the habit (not rule) of releasing at a UTC time that corresponds to the release version
    - Concerns about press - needs to be revisited.
    - Don't try for specific times, have guideline only.
    [AI] kate.stewart - work with press team on different ideas for embargo, etc. discuss with release team in advance.
    [AI] release team - round table go/no-go on release day with stakeholders.
 [AI] kate.stewart - check in with IS on steps to avoid download surge for 12.10
    - easier to find torrents? in announce/download link?
     - can we encourage press to link to torrents in their articles?
    - easier to mirror the release directories, ...

[AI] [?] make it easier for others to bring up temporary mirrors of release images on CD image

- [skaet] calls for manual QA testing on images (4/6, 4/12) did not get enough significant results and participation to find problems in time for fixes before final freeze.
Need manual testing with higher participation, earlier in cycle.
   - Better planning, and earlier scheduled testing so that we don't hit hard deadlines before testing See blueprint:
   http://summit.ubuntu.com/uds-q/meeting/20465/qa-q-qa-testing-cadence/

- [cjwatson] Some of the images pushed to CloudFront were out of date, because we forgot to tell IS to refresh them. Ideally this would be automatic, but at the very least we need a checklist entry somewhere.
   - [AI]: add this to release checklist. (checksums showed problem) pushmirror of cdimage? specialized script? sync with IS.

- [cjwatson] The cdimage release scripts don't have good handling of the situation where we're releasing different build numbers on different architectures, which is now close to being routine. This was largely due to hacks to make sure that older milestones went away properly, but we need to fix this properly. It led to (a) delays in final publishing because all but the most-recently-published image for some products had vanished, at least in some cases, which fortunately I noticed before we went live; (b) a parallel problem with torrents going missing, which unfortunately I didn't notice before we went live and had incorrectly ascribed to the torrent.ubuntu.com host being old and slow.
  - add work item to fix.

- [pitti] The Chinese Edition had not really been Chinese since February, which was only discovered an hour before 12.04 release. Would be nice if we could integrate the Chinese Edition into the daily jenkins testing, and ensure that the locale is zh_CN and ubuntu-defaults-zh-cn is installed.
  - ?? ownership of testing responsibility needs to be clarified.
  [rickspencer3] Should there not be automated tests for this?
  PES - needs to be represented and participating in release. Need to be following result logging.
[AI] - [jibel] add to automated daily testing
[AI] - [kate.stewart] get owner identified for test sign off.
-[skaet] Attempted to use milestones to denote which images were final, and which were still pre-release, by moving them as teams signed off on them. This didn't work out well in practice. We need another way of indicating team signoff (ideally on the iso tracker), and then only copying to final just before publishing, when all that are going to ship have been signed off.
    --> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-iso-tracker
    (ready to go idea for the iso tracker)
    ?? idea to investigate - logging change by product owner.

 - [gema] Allow only one respin on Monday of the release week. (see blueprint for full discussion)
   --> http://summit.ubuntu.com/uds-q/meeting/20521/qa-q-release-communication-improvements/

 -[skaet] support terms of arm server (armhf+omap4) were changed in the manifest on the day of release.
   - ?? need to get better interlock with PES in manifest discussions.
- [kitterman] (AKA ScottK) The pad is a good idea for maintaining state, but it needs to stay logged in better. There were multiple times I thought I was looking at current state and wasn't. As it was, I ended up having to refresh and re-login every time I looked at it. That mostly defeats the point of using a pad.
   - ?? better suggestions.
   - does this happen when moving around from 3G to wifi and back and such?
   [AlanBell] - new technology emerging? (etherpad-lite)

- [kitterman] Not clear in the ISO tracker which tests have been done in what environments. Some of the automated tests (upgrades in particular) are run without X and so don't fully replicate the user upgrade experience. These should all be run at least once in a full environment and it wasn't clear if this was the case or not.
    --> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-iso-tracker
    Automated testing is great, but want to have manual as well. See distinction between Auto and manual. Linking with hardware profiles? (jibel and stgraber to agree on conventions).

- [kitterman] Need test cases for upgrade notification and upgrade process that end users will use. We had some very late issues with this discovered in Kubuntu. Needs a fake location for changelogs.ubuntu.com to simulate release day and notifications.
    --> agreed need. Flavors can add directly. Magic files need to be written up developer testing. meta list - check list.
- [kitterman] Due to there only being a small number of active armel buildds right before the release(plus the normal limitations on powerpc buildds), a large number of non-virt PPA and private builds were a significant interference with getting the archive stabilized.
   --> https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-buildds-usage
   --> Beyond the above spec, we'll also be getting more buildd hardware for arm* and powerpc this cycle

- [kitterman] Need to look again at when we stop doing unseeded uploads. If the only requirement is that the last build be finished before release, then we could have gone on at least 24 hours longer than we did. Let's not tell people 'pencils down' before we have to.
   --> buildds quiescing was why it was there. Day's grace before release, import archive for security builds. No longer needed. Manual control and funnelling things down.
   Mirror pulse 24 hours before?
   - manually reviewing - communicating. Upload past deadline.
   [refer to SRU criteria?]
   [AI] ubuntu-release mail list discussion.
- [stgraber] With most images now being on cdimage.ubuntu.com (all flavours and some Ubuntu images), the server doesn't stand a chance at release time. I've personally been mirroring Edubuntu and Kubuntu, taking an average of 400Mbps for the few days after release (Thursday -> Sunday), so we definitely have demand for it.
Would be good to either have more DC machines serving cdimage.ubuntu.com at release time or at least making the structure more mirror-friendly so someone can easily mirror all the "released" images without also syncing the dailies.
  --> discussions with IS - how can this be improved?
(Linking to torrents- takes load off of cdimage traffic.)
- [kitterman] It seemed like getting maas into main and on the server CD and horizon into main were pursued past where they should have been. IIRC, maas on the server CD was the cause of at least one set of respins (this is the exception to my comment above that all the respins were for good last minute reasons). I don't understand all the reasons behind why these were so late or why they couldn't be deferred, but they both had a negative impact on release week. I think it's worth looking back at how/when they got FFes (I know maas had a very broadly written one) and not be so expansive in the future.
  --> daviey has a difference of opinion

- [nskaggs] although documenting respins on an etherpad is nice, more visibility would be good and useful at all iso testing phases. None of our community testers have a good idea as to what changed between iso respins and/or why/when it was respun. Something for the iso tracker?
  --> will suggestion to add package diff to iso tracker address?
   https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-iso-tracker
   https://blueprints.launchpad.net/ubuntu/+spec/qa-q-iso-testing-process
- [bilal] We need more and more people to try out the -proposed repo, especially near the end of the cycle. With Precise, we saw many packages coming to -proposed near the end of the cycle before entering release, but still many of them (like Unity 5.10.0-0ubuntu1) had bugs that were detected only after the package hit release.
 --> This may have been discussed in the session ... proposed by default would defeat the major reason for using proposed - to avoid breaking the updates/upgrades due to archive skew. More testing is always better, but proposed by default would be counter productive.
  - How would we fix this? Enable proposed by default on betas/RC and disable after release?
  -- This is best brought up in the '-proposed' sessions http://summit.ubuntu.com/uds-q/meeting/20290/other-q-freeze-use-of-proposed/ , http://summit.ubuntu.com/uds-q/meeting/20465/qa-q-qa-testing-cadence/
[komputes]
-luks in ubiquity/nautilus
-non-PAE LiveCD << very important for laptops running Intel Pentium chips, since currently there's no workaround (mini.iso is not live therefore does not work for recovery/offline usecases. Also I have had bad experiences with mini.iso)
 - [micahg] Use a Lubuntu or Xubuntu 12.04 image
  - but not in 12.10
 - [cjwatson] non-PAE is entirely unsupported in 12.10, so hard to see what to do about this at this point ...
-Removing optional desktop bloat (minimal system install) with less services, less startup items and less I/O
-Facilitating a full installation on USB drive (taking advantage of RAM rather than heavy disk I/O)
[YokoZar]
 - dealing with archive skew for installation/testing of multiarch packages is an increasing issue and I'd like to have a session on it at some point
 session already had apt-improvements, and using

Notes: original from before session at UDS: notes above.

What worked well
- new iso tracker (integration with builds, ease of adding images/test cases, localization teamsupport)
- new queue bot (accept/reject notification on queues)
- release note restructuring - better able to handle long term support and flavor specific audience needs. Thank yous to developers, testers, translators, etc.

What needs improvement
- queue audit logs (LP:#...)
- policies need to be standardized on for use of -proposed around milestone release window.
- Generic view of the test coverage we have on top of the daily results would be useful
- Having a view of what is going to land when, that we can see to start doing risk based testing would help tons, when our capacity to test is not at its best and we cannot cover everything every day
- It's confusing that the ReleaseSchedule has multiple days all listed with Thursday's date, such as:

April 12th | NonLanguagePackTranslationDeadline (Tue), FinalFreeze, LanguagePackTestRebuild (Fri)

Not to single him out, but Matthew East in particular thought he had an extra day or two to get the ubuntu-docs upload done before the Language Pack deadline because the format wasn't clear at first look. I propose splitting this events to separate lines -- jbicha

- [cjwatson] I propose that we abolish the habit (not rule) of releasing at a UTC time that corresponds to the release version. Firstly, the more we do it the more we set an external expectation which I think is overly constrictive. Secondly, the 12.04 release would have gone much more smoothly had we had time to carefully go round all the affected infrastructure teams (IS, web, etc.) and collect go/no-go status, without having set a prior expectation that we would be releasing at a fixed time and thus creating an environment where people felt under pressure to meet that time. The website load issues, the fact that many of the torrents were missing because we didn't have time to check them, and the fact that the CloudFront images were out of date could all have been ameliorated or fixed given more time on the day, and restoring a culture of releasing when we're ready on that day rather than at a fixed time. [kitterman] agrees.

- [skaet] calls for manual QA testing on images (4/6, 4/12) did not get enough significant results and participation to find problems in time for fixes before final freeze.
Need manual testing with higher participation, earlier in cycle.

- [cjwatson] Some of the images pushed to CloudFront were out of date, because we forgot to tell IS to refresh them. Ideally this would be automatic, but at the very least we need a checklist entry somewhere.

- [cjwatson] The cdimage release scripts don't have good handling of the situation where we're releasing different build numbers on different architectures, which is now close to being routine. This was largely due to hacks to make sure that older milestones went away properly, but we need to fix this properly. It led to (a) delays in final publishing because all but the most-recently-published image for some products had vanished, at least in some cases, which fortunately I noticed before we went live; (b) a parallel problem with torrents going missing, which unfortunately I didn't notice before we went live and had incorrectly ascribed to the torrent.ubuntu.com host being old and slow.

- [pitti] The Chinese Edition had not really been Chinese since February, which was only discovered an hour before 12.04 release. Would be nice if we could integrate the Chinese Edition into the daily jenkins testing, and ensure that the locale is zh_CN and ubuntu-defaults-zh-cn is installed.

-[skaet] Attempted to use milestones to denote which images were final, and which were still pre-release, by moving them as teams signed off on them. This didn't work out well in practice. We need another way of indicating team signoff (ideally on the iso tracker), and then only copying to final just before publishing, when all that are going to ship have been signed off.

- [gema] Allow only one respin on Monday of the release week. Any further problem found during testing is release noted and fixed after release, unless that image is broken (i.e. doesn't install). Respin-madness needs to happen the week *before* release week. [kitterman] Can you point to respins that shouldn't have happened? AFAIK, the respins I'm aware of were for ISO doesn't work under some conditions. [gema] I am not saying let's not do respins, just let's do them earlier, release week should be only to tidy up the release notes, not to find problems that break installs. One that could have waited is the server image self verification when you create your usb stick with usb-creator, that didn't break any install, although it looks bad if people verify their images, but I don't think was worth a respin on release week. I think we are not assessing risk of respins carefully enough yet. [adconrad] I'm going to go out on a limb here and say that "doing things eaerlier" solves exactly nothing, except if that "thing" is "testing". This is purely a social fix, both in Canonical QA and with community testers, when Final Freeze lands, everyone needs to be testing, full stop. When the "last respin" happens is "when it's ready", and pushing things back a week so people have nothing to do in the last week changes nothing except making your development cycle one week shorter (and it's already more than short enough, trust me). [nskaggs] Yes, but given the history of so many respins, folks are hesitant to go full-board testing and reporting when there results are simply wiped or outdated by a new respin. Sometimes this happens before they are able to finish reporting. When we spin a final RC, we should have a good feeling (aka, metrics, tests, etc) about it actually being the final iso, and avoid respins (and thus re-work) if at all possible. Gema's idea of re-spinning once a week (or a more frequent, but scheduled time) is likely a nice compromise here. [micahg] So, perhaps, making all the previous test results go away with a respin is the issue? Unless the bug was fixed or likely to have been fixed, maybe the previous reports can be carried forward (and reflected/tagged as belonging to a previous image). Obviously, the final images need to be tested, but the people testing earlier ISOs shouldn't need to think the testing was in vein.

-[skaet] support terms of arm server (armhf+omap4) were changed in the manifest on the day of release.

- [kitterman] (AKA ScottK) The pad is a good idea for maintaining state, but it needs to stay logged in better. There were multiple times I thought I was looking at current state and wasn't. As it was, I ended up having to refresh and re-login every time I looked at it. That mostly defeats the point of using a pad.

- [kitterman] Not clear in the ISO tracker which tests have been done in what environments. Some of the automated tests (upgrades in particular) are run without X and so don't fully replicate the user upgrade experience. These should all be run at least once in a full environment and it wasn't clear if this was the case or not.

- [kitterman] Need test cases for upgrade notification and upgrade process that end users will use. We had some very late issues with this discovered in Kubuntu. Needs a fake location for changelogs.ubuntu.com to simulate release day and notifications.

- [kitterman] Due to there only being a small number of active armel buildds right before the release(plus the normal limitations on powerpc buildds), a large number of non-virt PPA and private builds were a significant intereference with getting the archive stabilized.

- [kitterman] Need to look again at when we stop doing unseeded uploads. If the only requirement is that the last build be finished before release, then we could have gone on at least 24 hours longer than we did. Let's not tell people 'pencils down' before we have to.

- [stgraber] With most images now being on cdimage.ubuntu.com (all flavours and some Ubuntu images), the server doesn't stand a chance at release time. I've personally been mirroring Edubuntu and Kubuntu, taking an average of 400Mbps for the few days after release (Thursday -> Sunday), so we definitely have demand for it.
Would be good to either have more DC machines serving cdimage.ubuntu.com at release time or at least making the structure more mirror-friendly so someone can easily mirror all the "released" images without also syncing the dailies.

- [kitterman] It seemed like getting maas into main and on the server CD and horizon into main were pursued past where they should have been. IIRC, maas on the server CD was the cause of at least one set of respins (this is the exception to my comment above that all the respins were for good last minute reasons). I don't understand all the reasons behind why these were so late or why they couldn't be deferred, but they both had a negative impact on release week. I tihnk it's worth looking back at how/when they got FFes (I know maas had a very broadly written one) and not be so expansive in the future.

- [nskaggs] although documenting respins on an etherpad is nice, more visibility would be good and useful at all iso testing phases. None of our community testers have a good idea as to what changed between iso respins and/or why/when it was respun. Something for the iso tracker?

(?)

Work Items

Work items:
[kate.stewart] get update from launchpad team on queue logging bugs and ETA. If not in time for Beta 1, see if can find others to help. 20121003: pinged repeatedly through cycle - no progress due to launchpad resources not available): DONE
[kate.stewart] when deadline happens on non Thursday in schedule, split the day out into a separate line in ReleaseSchedule.: DONE
[kate.stewart] work with press team on different ideas for embargo, etc. discuss with release team in advance. : INPROGRESS
[kate.stewart] add step in release process checklist to call a round table with stakeholders for a go/no-go survey prior to announce email going out. : DONE
[kate.stewart] check in with IS on steps to avoid download surge for 12.10, investigate capacity prediction based on historical trends. : DONE
[kate.stewart] add to release checklist, tell IS to refresh images prior to push out to CloudFront. : DONE
[ubuntu-release] work with IS to automatically refresh release images prior to push to CloudFront by IS. : POSTPONED
[jibel] add Chinese Images to automated daily testing. : POSTPONED
[kate.stewart] get owner identified for test sign off and manual testing, otherwise consider removing image from builds. : DONE
[kate.stewart] work with PES to get better interlocks on release manifest tracking and announce coordination. : DONE
[alanbell] - email to ubuntu-release, when new technology (etherpad-lite) is available to test out for release coordination. : TODO
[kate.stewart] start discussion off with Ubuntu QA and release team on options to create test cases for upgrade notification and upgrade process that end users will use (Needs a fake location for changelogs.ubuntu.com to simulate release day and notifications.) : POSTPONED
[kate.stewart] start mail list discussion for ubuntu-release to revisit the timing of unseeded universe freeze to closer to release. : DONE
Investigate with IS the Data Center machines serving cdimage.ubuntu.com at release time or at least making the structure more mirror-friendly so someone can easily mirror all the "released" images without also syncing the dailies. : POSTPONED

Dependency tree

* Blueprints in grey have been implemented.