UEC upgrade from 9.10 karmic to Lucid fails

Bug #519513 reported by Boris Devouge
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
eucalyptus (Ubuntu)
Fix Released
High
Dustin Kirkland 
Lucid
Fix Released
High
Dustin Kirkland 

Bug Description

Started with a 9.10 karmic fully updated as of yesterday UEC working install, with one CLC/CC/SC and 2 NCs.

I proceed as follow:

 - ssh in the NCs and stopped eucalyptus related services.
 - ssh in the CLC and did the same

- updated my sources.list to lucid on all machines

- ran on 3 nodes: 'sudo apt-get clean all && sudo apt-get update && sudo apt-get upgrade'

- upgrade went well on all 3 machines, with no issues, and a 'sudo dpkg --configure -a' confirmed all was well with no output.

- on reboot, UEC is not working anymore, any command in the shell time out indefinitely, a look at the logs showed eucalytus services were 'looping' as I saw several times the same thing.

I am attaching the most of that log, my UEC install is logging with 'DEBUG'.

There is a java null pointer error along it. Sorry for the length of that log.

Revision history for this message
Boris Devouge (bdevouge) wrote :
Revision history for this message
Boris Devouge (bdevouge) wrote :

Version info (after upgrade)
=====================
ii eucalyptus-cc 1.6~bzr931-0ubuntu7.4
ii eucalyptus-cloud 1.6~bzr931-0ubuntu7.4
ii eucalyptus-common 1.6~bzr931-0ubuntu7.4
ii eucalyptus-gl 1.6.2~bzr1180-0ubuntu1
ii eucalyptus-java-common 1.6~bzr931-0ubuntu7.4
ii eucalyptus-sc 1.6.2~bzr1180-0ubuntu1
ii eucalyptus-walrus 1.6.2~bzr1180-0ubuntu1
ii libeucalyptus-commons-ext-java 0.4.2-0ubuntu1

Revision history for this message
Boris Devouge (bdevouge) wrote :

As additional comment, we understand Lucid is moving target, so the importance of this bug should be well diminished as the upgrade will need retesting with final lucid release.

Revision history for this message
Torsten Spindler (tspindler) wrote : Re: [Bug 519513] [NEW] UEC upgrade from 9.10 karmic to Lucid fails

On Tue, 2010-02-09 at 21:39 +0000, Boris Devouge wrote:
> - ran on 3 nodes: 'sudo apt-get clean all && sudo apt-get update &&
> sudo
> apt-get upgrade'

Any reason for using this method rather than the recommended
do-release-upgrade -d
procedure?

Revision history for this message
Boris Devouge (bdevouge) wrote :

I did not thought the 'do-release-upgrade -d' method worked until we release Lucid, or does it? Do you think it could have avoided the breakage?

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Thanks Boris, Torsten.

This is a known issue that we discussed at the distro sprint last week. We have a bit of work to do with upstream Eucalyptus to get upgrades from Karmic -> Lucid to work correctly.

I've marked the bug High/Confirmed, and targeted it at Beta1.

Basically, we're going to need some code that ensures that no instances are running during the upgrade, and then do some magic to get the database to upgrade properly.

This bug isn't triaged yet, as we don't yet quite understand all what needs to be done to get this to work. But we're on it ;-)

Changed in eucalyptus (Ubuntu):
status: New → Confirmed
importance: Undecided → High
milestone: none → ubuntu-10.04-beta-1
assignee: nobody → Daniel Nurmi (nurmi)
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Dan-

Assigning this bug to you, for the moment.

Could you please document in a comment, and point to the Eucalyptus documentation which describes upgrading to Eucalyptus 1.6.2? (Once you've done that, you can reassign the bug back to me).

Thanks.

Revision history for this message
ariel (garcia) wrote :

Hi, a comment to Boris' comment #2 first:

> Version info (after upgrade)
> ii eucalyptus-cc 1.6~bzr931-0ubuntu7.4
> ii eucalyptus-cloud 1.6~bzr931-0ubuntu7.4
> ii eucalyptus-common 1.6~bzr931-0ubuntu7.4
> ii eucalyptus-gl 1.6.2~bzr1180-0ubuntu1

you still have some of the older Karmic packages around, the Lucid ones are the 1.6.2.
I don't know about the 'do-release-upgrade -d' method, but you should at least perform an
   apt-get dist-upgrade
to allow apt to upgrade the existing packages even if that implies bigger (dependency) changes.
Check the resulting versions, all the eucalyptus-* packages should be at version 1.6.2-0ubuntu3
(as of today) after the upgrade.

Revision history for this message
ariel (garcia) wrote :

Second comment :-) I am trying to achieve the same (ie, have Eucalyptus 1.6.2) but without doing a full upgrade to Lucid, just upgrading the required packages, in a NC for now.

This combination still doesn't work for me (investigating further) but after i did update eucalyptus-nc with
        apt-get install eucalyptus-nc -t lucid
I had to:

        apt-get install librampart0 -t lucid
(the eucalyptus-common dependency in librampart0 is an unversioned one, but Eucalyptus NC fails to start with the Karmic one... complains that the lib was not found.... tiny issue to file a separate bugreport) At this point the Cloud Controller does see the updated node and can submit instance-requests... but they fail.

Now I've updated kvm and the kernel,
        apt-get install qemu-kvm -t lucid
        apt-get install linux-server -t lucid
trying to get kvm running again, but i still fails,
"warning: could not configure /dev/net/tun: no virtual network emulation"

I guess now qemu drops the CAP_NET_ADMIN capability before starting?? How can i change or solve that?

Thanks!

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Okay, I have received instructions from Dan privately.

I'll work on some test packages that handle the upgrade correctly.

Changed in eucalyptus (Ubuntu Lucid):
assignee: Daniel Nurmi (nurmi) → Dustin Kirkland (kirkland)
status: Confirmed → In Progress
Changed in eucalyptus (Ubuntu Lucid):
status: In Progress → Fix Committed
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

FYI, the fix I committed earlier today isn't quite complete yet. Still working on this. Will require some information from Dan before I can proceed.

Changed in eucalyptus (Ubuntu Lucid):
status: Fix Committed → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu14

---------------
eucalyptus (1.6.2-0ubuntu14) lucid; urgency=low

  * eucalyptus-common, debian/eucalyptus-common.install:
    - install the euca_upgrade script
    - handle upgrades from karmic, LP: #519513
  * debian/rules:
    - hack to make upgrades work
    - upstream code installs scripts into /etc/eucalyptus/cloud.d/scripts,
      but since it goes into /etc, debhelper considers these conffiles,
      which means that they don't get installed until postinst;
      the euca_upgrade runs in eucalyptus-common postinst, and depends
      on this script; thus, move scripts to /usr/lib, and symlink back to
      etc. Ugh. Upstream needs to fix this.
  * tools/euca_upgrade: call the groovy script from the /usr/lib
    location since the /etc/ location won't exist during postinst
 -- Dustin Kirkland <email address hidden> Sat, 13 Mar 2010 16:04:50 -0600

Changed in eucalyptus (Ubuntu Lucid):
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote :

Quoting Dustin:
"euca_upgrade" taking place inside of the postinst isn't quite working as desired.
After the "successful" upgrade, the Eucalyptus services won't run, as the database hasn't been entirely restored.

Reopening bug and targeting to beta2.

Changed in eucalyptus (Ubuntu Lucid):
milestone: ubuntu-10.04-beta-1 → ubuntu-10.04-beta-2
status: Fix Released → In Progress
Changed in eucalyptus (Ubuntu Lucid):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2-0ubuntu20

---------------
eucalyptus (1.6.2-0ubuntu20) lucid; urgency=low

  [ Dustin Kirkland ]
  * Continue fixing karmic -> lucid upgrades, LP: #519513
  * debian/eucalyptus-java-common.postinst: ensure that the db dir exists
    before running euca_upgrade
  * debian/eucalyptus-common.preinst: ensure that the local conf exists
  * tools/euca_upgrade:
    - on upgrade, migrate eucalyptus.conf settings to eucalyptus.local.conf
    - on upgrade, migrated $NODES to nodes.list
  * clc/modules/msgs/conf/scripts/startup.groovy: change m1.small to 192M
    such that we can actually boot the lucid uec image, LP: #544292

  [ Scott Moser ]
  * debian/source_eucalyptus.py: fix AttributeError, LP: #520775
 -- Dustin Kirkland <email address hidden> Mon, 22 Mar 2010 17:22:48 -0700

Changed in eucalyptus (Ubuntu Lucid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.