[SRU] OVN db sync script fails with OVN schema that has label column in ACL table

Bug #1951296 reported by Jakub Libosvar
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
New
Undecided
Unassigned
Ussuri
Fix Released
Undecided
Unassigned
Victoria
Fix Released
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned
Zed
Fix Released
Undecided
Unassigned
neutron
Fix Released
Undecided
Jakub Libosvar
neutron (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
Backport fix to Focal/Ussuri so that neutron-ovn-db-sync-util does not trip up when it finds ovn ACL table entries with a "label" column that does not exist in neutron db.

[Test Plan]
 * Deploy Openstack Ussuri
 * Upgrade OVN to 22.03 (https://charmhub.io/ovn-chassis/configure?channel=22.03/stable#ovn-source)
 * Create a network with security groups
 * Create an instance using this network so that ports get tied to SGs
 * Go to neutron-api unit (neutron-server) and do the following
 * cp /etc/neutron/neutron.conf /etc/neutron/neutron.conf.no_keystone_authtoken
 * remove "auth_section = keystone_authtoken" in the [nova] section of neutron.conf.no_keystone_authtoken
 * run 'neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf.no_keystone_authtoken --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair'
 * the above should not produce any errors like the following:

    RuntimeError: ACL ... already exists

[Regression Potential]
there is no regression potential expected with this patch.

--------------------------------------------------------------------------

OVN introduced a new column in ACL table. The column name is label and when running db-sync script, we compare ACL generated by the ovn mech driver from Neutron DB with the actual ACLs in the OVN DB. Because of the new label column, everything seems like a new ACL because the column differs to what Neutron generated. Thus the script attempts to create a new ACL that already exists.

    b'Traceback (most recent call last):'
    b' File "/usr/local/lib/python3.6/site-packages/neutron/tests/base.py", line 181, in func'
    b' return f(self, *args, **kwargs)'
    b' File "/usr/local/lib/python3.6/site-packages/neutron/tests/base.py", line 181, in func'
    b' return f(self, *args, **kwargs)'
    b' File "/home/cloud-user/networking-ovn/networking_ovn/tests/functional/test_ovn_db_sync.py", line 1547, in test_ovn_nb_sync_repair'
    b" self._test_ovn_nb_sync_helper('repair')"
    b' File "/home/cloud-user/networking-ovn/networking_ovn/tests/functional/test_ovn_db_sync.py", line 1543, in _test_ovn_nb_sync_helper'
    b' self._sync_resources(mode)'
    b' File "/home/cloud-user/networking-ovn/networking_ovn/tests/functional/test_ovn_db_sync.py", line 1523, in _sync_resources'
    b' nb_synchronizer.do_sync()'
    b' File "/home/cloud-user/networking-ovn/networking_ovn/ovn_db_sync.py", line 104, in do_sync'
    b' self.sync_acls(ctx)'
    b' File "/home/cloud-user/networking-ovn/networking_ovn/ovn_db_sync.py", line 288, in sync_acls'
    b' txn.add(self.ovn_api.pg_acl_add(**acla))'
    b' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__'
    b' next(self.gen)'
    b' File "/home/cloud-user/networking-ovn/networking_ovn/ovsdb/impl_idl_ovn.py", line 230, in transaction'
    b' yield t'
    b' File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__'
    b' next(self.gen)'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 110, in transaction'
    b' del self._nested_txns_map[cur_thread_id]'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/api.py", line 61, in __exit__'
    b' self.result = self.commit()'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit'
    b' raise result.ex'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run'
    b' txn.results.put(txn.do_commit())'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit'
    b' command.run_idl(txn)'
    b' File "/usr/local/lib/python3.6/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 124, in run_idl'
    b' self.direction, self.priority, self.match))'
    b'RuntimeError: ACL (from-lport, 1001, inport == @neutron_pg_drop && ip) already exists'

Changed in neutron:
assignee: nobody → Jakub Libosvar (libosvar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/818255

Changed in neutron:
status: New → In Progress
Revision history for this message
Jakub Libosvar (libosvar) wrote : Re: OVN db sync script fails with OVN schema that has label column in ACL table

This issue is already reported at https://bugs.launchpad.net/neutron/+bug/1946666

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818255
Committed: https://opendev.org/openstack/neutron/commit/23b99e2f127731c85f63c88c7144aa0a111c4abf
Submitter: "Zuul (22348)"
Branch: master

commit 23b99e2f127731c85f63c88c7144aa0a111c4abf
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 17 17:29:13 2021 +0000

    ovn: Filter ACL columns when syncing the DB

    The patch filters columns from OVN DB only to those that are used by the
    ovn mechanism driver. It means generated ACLs from Neutron DB and ACLs
    obtained from the OVN DB will always have the same columns. This is
    useful for db sync script when comparing if given security group rule
    has corresponding ACL in the OVN DB.

    Closes-Bug: #1951296
    Signed-off-by: Jakub Libosvar <email address hidden>
    Change-Id: I39e3b987b8546fd970a933b846ed23c8a2588258

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/818471

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/818472

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/818473

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/818474

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818471
Committed: https://opendev.org/openstack/neutron/commit/8cc9ead9198bfb280aaf16de0f529764f345091b
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 8cc9ead9198bfb280aaf16de0f529764f345091b
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 17 17:29:13 2021 +0000

    ovn: Filter ACL columns when syncing the DB

    The patch filters columns from OVN DB only to those that are used by the
    ovn mechanism driver. It means generated ACLs from Neutron DB and ACLs
    obtained from the OVN DB will always have the same columns. This is
    useful for db sync script when comparing if given security group rule
    has corresponding ACL in the OVN DB.

    Closes-Bug: #1951296
    Signed-off-by: Jakub Libosvar <email address hidden>
    Change-Id: I39e3b987b8546fd970a933b846ed23c8a2588258
    (cherry picked from commit 23b99e2f127731c85f63c88c7144aa0a111c4abf)

tags: added: in-stable-xena
tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818472
Committed: https://opendev.org/openstack/neutron/commit/f89df138b5cbc67b3afc2b757b11c6c99eb27e7e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f89df138b5cbc67b3afc2b757b11c6c99eb27e7e
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 17 17:29:13 2021 +0000

    ovn: Filter ACL columns when syncing the DB

    The patch filters columns from OVN DB only to those that are used by the
    ovn mechanism driver. It means generated ACLs from Neutron DB and ACLs
    obtained from the OVN DB will always have the same columns. This is
    useful for db sync script when comparing if given security group rule
    has corresponding ACL in the OVN DB.

    Closes-Bug: #1951296
    Signed-off-by: Jakub Libosvar <email address hidden>
    Change-Id: I39e3b987b8546fd970a933b846ed23c8a2588258
    (cherry picked from commit 23b99e2f127731c85f63c88c7144aa0a111c4abf)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818474
Committed: https://opendev.org/openstack/neutron/commit/1255eded3c7c699e2d2d25e460fc23303a3091ca
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 1255eded3c7c699e2d2d25e460fc23303a3091ca
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 17 17:29:13 2021 +0000

    ovn: Filter ACL columns when syncing the DB

    The patch filters columns from OVN DB only to those that are used by the
    ovn mechanism driver. It means generated ACLs from Neutron DB and ACLs
    obtained from the OVN DB will always have the same columns. This is
    useful for db sync script when comparing if given security group rule
    has corresponding ACL in the OVN DB.

    Closes-Bug: #1951296
    Signed-off-by: Jakub Libosvar <email address hidden>
    Change-Id: I39e3b987b8546fd970a933b846ed23c8a2588258
    (cherry picked from commit 23b99e2f127731c85f63c88c7144aa0a111c4abf)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818473
Committed: https://opendev.org/openstack/neutron/commit/31af5c689bc28f9be2d56846a52384e6084e6fed
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 31af5c689bc28f9be2d56846a52384e6084e6fed
Author: Jakub Libosvar <email address hidden>
Date: Wed Nov 17 17:29:13 2021 +0000

    ovn: Filter ACL columns when syncing the DB

    The patch filters columns from OVN DB only to those that are used by the
    ovn mechanism driver. It means generated ACLs from Neutron DB and ACLs
    obtained from the OVN DB will always have the same columns. This is
    useful for db sync script when comparing if given security group rule
    has corresponding ACL in the OVN DB.

    Closes-Bug: #1951296
    Signed-off-by: Jakub Libosvar <email address hidden>
    Change-Id: I39e3b987b8546fd970a933b846ed23c8a2588258
    (cherry picked from commit 23b99e2f127731c85f63c88c7144aa0a111c4abf)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.1.0

This issue was fixed in the openstack/neutron 19.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0

This issue was fixed in the openstack/neutron 17.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0

This issue was fixed in the openstack/neutron 18.2.0 release.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol

This issue was fixed in the openstack/networking-ovn train-eol release.

description: updated
summary: - OVN db sync script fails with OVN schema that has label column in ACL
- table
+ [SRU] OVN db sync script fails with OVN schema that has label column in
+ ACL table
tags: added: se se-sponsor-dgadomski
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Jakub, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.4.2-0ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Jakub, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Verified focal-proposed using [Test Case] and output:

# neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf.no_keystone_authtoken --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair| grep ACL
/usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
  result = self._query(query)
2023-04-12 14:05:18.810 469229 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-1dbe475d-12e1-4fc2-8893-1e8b2525bbcc - - - - -] ACL-SYNC: started @ 2023-04-12 14:05:18.810604 sync_acls /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:241
2023-04-12 14:05:18.849 469229 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-1dbe475d-12e1-4fc2-8893-1e8b2525bbcc - - - - -] ACL-SYNC: finished @ 2023-04-12 14:05:18.849802 sync_acls /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:301
# apt-cache policy neutron-common
neutron-common:
  Installed: 2:16.4.2-0ubuntu6.1
  Candidate: 2:16.4.2-0ubuntu6.1
  Version table:
 *** 2:16.4.2-0ubuntu6.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:16.4.2-0ubuntu6 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Verified ussuri-proposed using [Test Case] and output:

# neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf.no_keystone_authtoken --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair| grep ACL
2023-04-13 09:17:34.332 1803 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-1e03d71c-8f82-40c3-8bf9-a762b0f0bae9 - - - - -] ACL-SYNC: started @ 2023-04-13 09:17:34.332499 sync_acls /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:241
2023-04-13 09:17:34.355 1803 DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-1e03d71c-8f82-40c3-8bf9-a762b0f0bae9 - - - - -] ACL-SYNC: finished @ 2023-04-13 09:17:34.355450 sync_acls /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:301
# apt-cache policy neutron-common
neutron-common:
  Installed: 2:16.4.2-0ubuntu6.1~cloud0
  Candidate: 2:16.4.2-0ubuntu6.1~cloud0
  Version table:
 *** 2:16.4.2-0ubuntu6.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu8 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

tags: added: verification-done verification-ussuri-done
removed: verification-needed verification-ussuri-needed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:16.4.2-0ubuntu6.1

---------------
neutron (2:16.4.2-0ubuntu6.1) focal; urgency=medium

  * d/p/ovn-Filter-ACL-columns-when-syncing-the-DB.patch: Backport fix for
    neutron-ovn-db-sync-util to ensure that ACL columns are filtered to
    include only those used by the OVN mechanism driver (LP: #1951296).

 -- Edward Hope-Morley <email address hidden> Tue, 28 Mar 2023 16:11:11 +0100

Changed in neutron (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Fix released in the ussuri cloud archive in neutron 2:16.4.2-0ubuntu6.1~cloud0

Changed in neutron (Ubuntu):
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ussuri-eol

This issue was fixed in the openstack/neutron ussuri-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.