neutron migrations are not idempotent or invariant to enabled services

Bug #1307344 reported by Jordan Pittier
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Ann Taraday

Bug Description

Neutron is currently failing during grenade upgrade testing. The symptom is that meteringlabels table doesn't exist.

The root cause is that neutron migrations are not idempotent or invariant to enabled services (which must be true for migrations for the system to work as expected). As can be seen here - https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/569e98a8132b_metering.py#L50-L51

Migration content might be skipped based on configuration conditionals. This means it's possible for 2 different installations to report as db schema 569e98a8132b but have different schemas. The logic of db migrations assumes that 569e98a8132b can only result in a single schema, so future migrations depend on the content all being the same here.

The solution here is to probably build a set of "healing migrations" that undo all the past conditional skips to get neutron back in line with a migration id means exactly a single schema. Anything other than that will be largely impossible to support in the field.

(previous description below)
Hello,
I just got a neutron-db-manage failure in the Gate : http://logs.openstack.org/77/83777/1/check/check-tempest-dsvm-neutron-heat-slow/a7ad8c0/

Relevant stack trace is

"
2014-04-13 23:53:15.281 | INFO [alembic.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
2014-04-13 23:53:15.296 | Traceback (most recent call last):
2014-04-13 23:53:15.296 | File "/usr/local/bin/neutron-db-manage", line 10, in <module>
.....................
2014-04-13 23:53:15.340 | sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1146, "Table 'neutron_ml2.meteringlabels' doesn't exist") 'ALTER TABLE meteringlabels CHANGE description description VARCHAR(1024) NULL' ()"
"

Revision history for this message
Kevin Benton (kevinbenton) wrote :

I don't see that in the logs for the link you posted. Looks like the migration worked okay. Wrong link?

http://logs.openstack.org/77/83777/1/check/check-tempest-dsvm-neutron-heat-slow/a7ad8c0/logs/devstacklog.txt.gz

Changed in neutron:
status: New → Confirmed
Revision history for this message
Jordan Pittier (jordan-pittier) wrote :
Sean Dague (sdague)
summary: - neutron-db-manage fails with metering enabled
+ neutron migrations are not idempotent or invariant to enabled services
Sean Dague (sdague)
description: updated
Changed in neutron:
importance: Undecided → Critical
Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

To me, this is more about rethinking a neutron feature (db migrations), and therefore we might need also an accompanying blueprint. A bug is also good to have as it can be used to pin failures in gate/check jobs.

I have shared some thoughts on the mailing list as well. subject: "Neutron Migrations, service plugins and Grenade jobs"

Changed in neutron:
assignee: nobody → Ann Kamyshnikova (akamyshnikova)
Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Henry Gessau (gessau) wrote :

For some reason gerrit did not link the proposed change:
https://review.openstack.org/87935

Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/87935
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c6a8f25f49f84142ad92cf03583047ee3cc21bc3
Submitter: Jenkins
Branch: master

commit c6a8f25f49f84142ad92cf03583047ee3cc21bc3
Author: Ann Kamyshnikova <email address hidden>
Date: Wed Apr 16 15:17:02 2014 +0400

    Fix migration that breaks Grenade jobs

    Migration 33c3db036fe4_set_length_of_description_field_metering
    needs to be changed as it is a reason of Grenade job's fail.

    Closes-bug: #1307344

    Change-Id: Ia57d9a60847e4a34977e40fb8b09181bdbd9c39b

Changed in neutron:
status: In Progress → Fix Committed
Changed in neutron:
milestone: none → juno-1
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.