Use the new enginefacade from oslo_db

Registered by Armando Migliaccio

[Existing problem]
The oslo.db.sqlalchemy.session.EngineFacade is only a factory, and not a facade. This makes the code inconsistent, low performance, and prone to bugs. The details of its problems are described in [1].
Most intuitive issue is that oslo will report warning at [2]

[Proposal]
Use the new oslo_db.sqlalchemy.enginefacade to replace the current EngineFacade.

[Benefits]
A clean, less problem, higher performance db transaction.

[1] Oslo db spec: http://specs.openstack.org/openstack/oslo-specs/specs/kilo/make-enginefacade-a-facade.html
[2] https://github.com/openstack/oslo.db/blob/60af1042b8123d67ed7c80d1a1720865a4255ad5/oslo_db/sqlalchemy/enginefacade.py#L956

Also see the related RFE bug.

Blueprint information

Status:
Complete
Approver:
Kevin Benton
Priority:
High
Drafter:
Ann Taraday
Direction:
Approved
Assignee:
Miguel Lavalle
Definition:
Approved
Series goal:
Accepted for victoria
Implementation:
Implemented
Milestone target:
milestone icon wallaby-2
Started by
Henry Gessau
Completed by
Slawek Kaplonski

Related branches

Sprints

Whiteboard

Dec-01-2016(armax): Ann actively working on patches:

https://review.openstack.org/#/q/status:open+topic:bp/enginefacade-switch

Implementation Patches:

New engine facade from oslo_db: Step 1 (Import new enginefacade)
https://review.openstack.org/300017

New engine facade from oslo_db: Step 2 (Get rid of lazy init for engine facade)
https://review.openstack.org/312393

New engine facade from oslo_db: Step 3 (Apply read/writer)
https://review.openstack.org/338295

In the drivers meeting on August 11 [1] we decided to forego the reader/writer decorators for existing methods and instead replace transaction contexts with the reader/writer context manager.

[1] http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-08-11-22.14.log.html

Related patches:

Add check that factory started in dispose
https://review.openstack.org/340283

Get rid of get_engine() in db/api.py
https://review.openstack.org/339783

Refactor setting OSprofiler for db calls
https://review.openstack.org/340324

[oslo.db] Add a hook to process newly created engines
https://review.openstack.org/340260

Gerrit topic: https://review.openstack.org/#q,topic:bug/1596075,n,z

Addressed by: https://review.openstack.org/339226
    Always start transactions in quota cleanup methods

Gerrit topic: https://review.openstack.org/#q,topic:391308,n,z

Addressed by: https://review.openstack.org/391308
    Usage of new enginefacade for flavor db

Gerrit topic: https://review.openstack.org/#q,topic:bp/enginefacade-switch,n,z

Addressed by: https://review.openstack.org/392187
    [WIP]Usage new enginefacade for quota db

Addressed by: https://review.openstack.org/392572
    [WIP]Use new enginefacade for address_scope_db

Addressed by: https://review.openstack.org/392619
    [WIP] Use new enginefacade metering db

Addressed by: https://review.openstack.org/392699
    [WIP] Use new enginefacade for qos

Gerrit topic: https://review.openstack.org/#q,topic:bp/push-notifications,n,z

Addressed by: https://review.openstack.org/395972
    Fix ML2, base db plugin update_subnet for transactions

Addressed by: https://review.openstack.org/396013
    Separate floating IP port creation from transaction

Addressed by: https://review.openstack.org/396015
    [WIP]: Use callbacks to create DVR floating GW port

Addressed by: https://review.openstack.org/396055
    [WIP] Use new enginefacade for servicetype_db

Addressed by: https://review.openstack.org/396121
    [WIP] Use new enginefacade for agent db

Addressed by: https://review.openstack.org/396167
    [WIP] Use new enginefacade for tag

Addressed by: https://review.openstack.org/396182
    [WIP] Use new enginefacade for schedulers

Gerrit topic: https://review.openstack.org/#q,topic:bug/1540844,n,z

Addressed by: https://review.openstack.org/397202
    [WIP] Use new enginefacade for provisioning_blocks

Addressed by: https://review.openstack.org/397756
    [WIP] Use new enginefacade for segments

Addressed by: https://review.openstack.org/396014
    Move DVR fip agent gw port create out of transaction

Addressed by: https://review.openstack.org/398873
    [WIP]Change passing session to context in segments db functions

Addressed by: https://review.openstack.org/358845
    OVO for FlatAllocation

Addressed by: https://review.openstack.org/402750
    [WIP]New enginefacade for networks, subnets, rbac

Addressed by: https://review.openstack.org/403679
    [WIP]Use new enginefacade for rbac

Addressed by: https://review.openstack.org/404182
    [WIP] Use new enginefacade for dvr mac db

Addressed by: https://review.openstack.org/404714
    [WIP] Refactor ml2_db to pass context

Addressed by: https://review.openstack.org/404715
    [WIP] New enginefacade for ports

Addressed by: https://review.openstack.org/406275
    Change passing session to context in segments db functions

Addressed by: https://review.openstack.org/385178
    Integrate NetworkSegment OVO

Gerrit topic: https://review.openstack.org/#q,topic:engine/routers,n,z

Addressed by: https://review.openstack.org/367810
    [WIP]Integration of Vlan\VxlanAllocation/Endpoints OVO

Addressed by: https://review.openstack.org/415226
    [WIP] Stop using legacy facade

Gerrit topic: https://review.openstack.org/#q,topic:get_session,n,z

Gerrit topic: https://review.openstack.org/#q,topic:engine/sg,n,z

Addressed by: https://review.openstack.org/418915
    [WIP] Use reader and writer for security groups

Addressed by: https://review.openstack.org/419815
    Use writer for trunk database operations

Addressed by: https://review.openstack.org/403870
    [WIP] New enginefacade for objects

Addressed by: https://review.openstack.org/409807
    Use new enginefacade for routers

Gerrit topic: https://review.openstack.org/#q,topic:final,n,z

Gerrit topic: https://review.openstack.org/#q,topic:engine/obj,n,z

Addressed by: https://review.openstack.org/428779
    Use same session object for ml2 dict extend functions

Addressed by: https://review.openstack.org/434331
    Flavors: move dict extend out of txn

Addressed by: https://review.openstack.org/434454
    Load all relationships on object init

Gerrit topic: https://review.openstack.org/#q,topic:bp/adopt-oslo-versioned-objects-for-db,n,z

Addressed by: https://review.openstack.org/451594
    LOG.exception for mech dict extend failure

Addressed by: https://review.openstack.org/451595
    Fix relationship event handler for flushes and nested

Addressed by: https://review.openstack.org/451696
    Make RBAC entry removal enginefacade friendly

Gerrit topic: https://review.openstack.org/#q,topic:bug/1664643,n,z

Addressed by: https://review.openstack.org/453063
    Add flush_on_subtransaction=True to engine config

Addressed by: https://review.openstack.org/453048
    Fix SQL fixture to preserve engine facade settings

Addressed by: https://review.openstack.org/457038
    move make_port_dict back out of txn

Addressed by: https://review.openstack.org/457170
    ml2: Remove no longer necessary subtransaction workaround

Addressed by: https://review.openstack.org/459662
    [WIP] Use reader and writer for agents

Addressed by: https://review.openstack.org/468883
    Use Neutron new engine-facade

Gerrit topic: https://review.openstack.org/#q,topic:bug/1706750,n,z

Addressed by: https://review.openstack.org/532343
    Revert "Use writer for trunk database operations"

Addressed by: https://review.openstack.org/536679
    Change from new engine facade to old nested transaction for OVO

Addressed by: https://review.openstack.org/545501
    Conclude adoption of new enginefacade in Neutron

Addressed by: https://review.openstack.org/553617
    WIP objects: automatically detect whether engine facade is used

Addressed by: https://review.openstack.org/556667
    Automatically expire obsolete relationships

Addressed by: https://review.openstack.org/559425
    tests: fixed NoSuchGroupError in TestNovaSegmentNotifier

Addressed by: https://review.openstack.org/561066
    Renamed subnet_service_type_db_models module

Addressed by: https://review.openstack.org/561071
    tests: don't configure plugin for tests that don't need it

Addressed by: https://review.openstack.org/563601
    Revert "Automatically expire obsolete relationships"

Addressed by: https://review.openstack.org/563736
    Revert "Revert "Automatically expire obsolete relationships""

Addressed by: https://review.openstack.org/611462
    Remove context guard of Agent OVO

Addressed by: https://review.openstack.org/619473
    Revert "Switch to new engine facade in neutron/db/agents_db.py"

Gerrit topic: https://review.openstack.org/#q,topic:bug/1803745,n,z

Addressed by: https://review.openstack.org/619652
    Switch to new engine facade in test_bgp_dragent_scheduler.py

Gerrit topic: https://review.opendev.org/#/q/topic:bp/enginefacade-switch

Addressed by: https://review.opendev.org/661351
    Switch to new engine facade for Subnet object

Addressed by: https://review.opendev.org/661362
    Switch to new engine facade for L3RpcCallback

Addressed by: https://review.opendev.org/545501
    Conclude adoption of new enginefacade in Neutron

Addressed by: https://review.opendev.org/661695
    Switch to new engine facade for QoS

Addressed by: https://review.opendev.org/661861
    Switch DB _utils to new engine facade

Addressed by: https://review.opendev.org/661909
    Switch to new engine facade for Schedulers

Addressed by: https://review.opendev.org/661916
    Switch to new engine facade for ExtraRoute_dbonly_mixin

Addressed by: https://review.opendev.org/662071
    Switch to new engine facade for StandardAttribute objects

Addressed by: https://review.opendev.org/662270
    Switch to new engine facade for Route objects

Addressed by: https://review.opendev.org/662869
    Switch to new engine facade for IpamPluggableBackend

Addressed by: https://review.opendev.org/663306
    Switch to new engine facade for L3_HA_NAT_db_mixin

Addressed by: https://review.opendev.org/663712
    Switch to new engine facade for Route objects

Addressed by: https://review.opendev.org/663713
    Switch to new engine facade for Route objects

Addressed by: https://review.opendev.org/663714
    Switch to new engine facade for Route objects

Addressed by: https://review.opendev.org/665436
    Switch to new engine facade for DriverController

Addressed by: https://review.opendev.org/666003
    Switch to new engine facade for TrunkPlugin and TrunkSkeleton

Addressed by: https://review.opendev.org/666024
    Switch to new engine facade for BaseResourceFilter

Gerrit topic: https://review.opendev.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db

Addressed by: https://review.opendev.org/704682
    objects: automatically detect whether engine facade is used

Addressed by: https://review.opendev.org/712171
    Switch to new enginefacade in l3_hascheduler_db module

Addressed by: https://review.opendev.org/712446
    Switch to new engine facade in l3_gwmode_db module

Addressed by: https://review.opendev.org/714068
    Switch to new engine facade in test_allowedaddresspairs_db module

Addressed by: https://review.opendev.org/714097
    Finish migration to new engine facade in test_db_base_plugin_v2

Addressed by: https://review.opendev.org/714098
    Switch to new engine facade in test_data_plane_status

Addressed by: https://review.opendev.org/715233
    Switch to new engine facade in ML2 unit tests modules

Addressed by: https://review.opendev.org/715306
    Switch to new engine facade in revision plugin's UT

Addressed by: https://review.opendev.org/715315
    Use new engine facade in L3 DB code

Addressed by: https://review.opendev.org/748529
    Switch to new engine facade in ovn_revision_numbers_db module

Addressed by: https://review.opendev.org/748530
    Switch to new engine facade in L3 router's driver_controller module

Addressed by: https://review.opendev.org/748531
    Switch to new engine facade in OVN trunk driver module

Addressed by: https://review.opendev.org/748532
    Switch to new enginefacade in uplink_status_propagation UT module

Addressed by: https://review.opendev.org/761728
    Remove "autonested_transaction" method

Addressed by: https://review.opendev.org/761729
    Remove Context.session check with new engine facade

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.