Use push style notifications for all server->agent information

Registered by Kevin Benton on 2015-09-21

Our current approach of having lots of functions that can be called at any given time via AMQP by the agents creates stampeding herd scenarios during failures and recoveries of even small amounts of agents in a reasonable scale system. The aim of this blueprint is to fundamentally revisit the pattern we use for getting information to the agents to ensure that the pace is determined by the server delivering data rather than timeouts/retries on the agent side.

Blueprint information

Status:
Started
Approver:
Ihar Hrachyshka
Priority:
High
Drafter:
Kevin Benton
Direction:
Approved
Assignee:
Kevin Benton
Definition:
Approved
Series goal:
Accepted for pike
Implementation:
Slow progress
Milestone target:
milestone icon pike-2
Started by
Armando Migliaccio on 2016-11-01

Related branches

Sprints

Whiteboard

Sep-08-2016(armax): deferred to Ocata

Aug-8-2016(ihrachys): ongoing work, patches started landing, currently waiting for needed ovo objects to land to proceed with remaining RPC bits. Should be on track for N.

Apr-4-2016(armax): to start working on dhcp only, spec to be resume. Needs decoupling from OVO

Mar-3-2106(armax): Moved to Newton. Please ensure you re-submit spec if necessary.

Feb-01-2016(armax): still going through spec review, no approver still.

Jan-24-2016(armax): this needs an 'approver' volunteer for spec/code review. The spec under review needs some love too.

Jan-21-2016(armax): needs an approver for code review; a spec is still going in progress of merging.

Gerrit topic: https://review.openstack.org/#q,topic:bp/push-notifications,n,z

Addressed by: https://review.openstack.org/225995
    Add spec for push notification refactor

Gerrit topic: https://review.openstack.org/#q,topic:push-notifications,n,z

Addressed by: https://review.openstack.org/303966
    WIP

Addressed by: https://review.openstack.org/330327
    Add revision_number to standard attr for OVO

Addressed by: https://review.openstack.org/336729
    Add callbacks for networks and subnets in ML2

Addressed by: https://review.openstack.org/336730
    Add generic object AMQP notifier

Addressed by: https://review.openstack.org/340272
    Add generic object AMQP receiver

Addressed by: https://review.openstack.org/340273
    Load generic AMQP receiver in DHCP agent

Addressed by: https://review.openstack.org/341427
    Rollback port after failed to add it to router

Gerrit topic: https://review.openstack.org/#q,topic:bp/adopt-oslo-versioned-objects-for-db,n,z

Addressed by: https://review.openstack.org/351101
    Introduce ovo objects for security groups

Addressed by: https://review.openstack.org/351368
    Introduce ovo objects for ports

Addressed by: https://review.openstack.org/269658
    Introduce ovo objects for networks

Addressed by: https://review.openstack.org/353571
    Introduce ovo objects for network segments

Addressed by: https://review.openstack.org/357228
    [wip]: add standard attributes to qospolicy

Addressed by: https://review.openstack.org/357730
    Fix init method for HasStandardAttributes

Gerrit topic: https://review.openstack.org/#q,topic:fip_ovo,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/adopt-oslo-versioned-objects-for-db-patch33,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/adopt-oslo-versioned-objects-for-db-patch35,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/adopt-oslo-versioned-objects-for-db-patch37,n,z

Addressed by: https://review.openstack.org/388938
    Simplify resources module in RPC callbacks

Addressed by: https://review.openstack.org/388939
    [WIP]: server-side push notifications for ML2

Addressed by: https://review.openstack.org/392960
    Fix modify_fields_from_db for vif_details empty str

Addressed by: https://review.openstack.org/392931
    Add 'to_primitive' for MACAddress and IPNetwork

Addressed by: https://review.openstack.org/394652
    Get post-subnet actions out of transaction

Addressed by: https://review.openstack.org/395972
    Fix ML2, base db plugin update_subnet for transactions

Addressed by: https://review.openstack.org/396013
    Separate floating IP port creation from transaction

Addressed by: https://review.openstack.org/396014
    [WIP]: move fip agent gw port create out of txn

Addressed by: https://review.openstack.org/396015
    [WIP]: Use callbacks to create DVR floating GW port

Gerrit topic: https://review.openstack.org/#q,topic:bug/1540844,n,z

Addressed by: https://review.openstack.org/404074
    Use revision to discard stale DHCP updates

Addressed by: https://review.openstack.org/409577
    API compare-and-swap updates based on revision_number

Gerrit topic: https://review.openstack.org/#q,topic:bug/1493714,n,z

Addressed by: https://review.openstack.org/417520
    Don't create default SG in transaction

Addressed by: https://review.openstack.org/417552
    Don't emit SG rule AFTER events until TX closed

Addressed by: https://review.openstack.org/404265
    Agent-side receiver cache for ML2 OVO

Addressed by: https://review.openstack.org/410422
    Use push notification for security groups

Addressed by: https://review.openstack.org/410385
    Use push-notificates for OVSPluginAPI

Addressed by: https://review.openstack.org/410418
    Decompose SG RPC API DB methods

Addressed by: https://review.openstack.org/410417
    Fixes to allow OVO deserializion of ports/networks

Addressed by: https://review.openstack.org/399964
    Add missing port UPDATE event to ML2

Addressed by: https://review.openstack.org/412613
    Add bulk pull OVO interface

Addressed by: https://review.openstack.org/417672
    Add missing module-level SUBNET ref

Addressed by: https://review.openstack.org/425332
    Change _VALID_CLS to store obj names

Addressed by: https://review.openstack.org/435748
    Avoid loading network and all rels for subnet query

Addressed by: https://review.openstack.org/435904
    Cleanup _find_related_obj

Addressed by: https://review.openstack.org/458464
    Use dirname in object recursive import

Addressed by: https://review.openstack.org/461099
    Avoid loading network and all rels for subnet query

Addressed by: https://review.openstack.org/464904
    Notify L2pop driver from update_device_(up|down)

Addressed by: https://review.openstack.org/465423
    [wip]: Fix trace in make_compatiable for port

Addressed by: https://review.openstack.org/471357
    DNM Revert "Use push-notificates for OVSPluginAPI"

Addressed by: https://review.openstack.org/474058
    Move retry decorator to DB methods

Addressed by: https://review.openstack.org/474059
    Move db methods to bottom of SG RPC class

Addressed by: https://review.openstack.org/474060
    Move info retrieval methods below notifier

Addressed by: https://review.openstack.org/474146
    Lazy load of resources in resource cache

Addressed by: https://review.openstack.org/474383
    Manually increment revision numbers in revision plugin

Addressed by: https://review.openstack.org/475112
    Add support for list querying in resource cache

Addressed by: https://review.openstack.org/475283
    OVO: Allow port queries based on security_group_ids

Addressed by: https://review.openstack.org/475646
    Add revises_on_change to Binding DB models

Addressed by: https://review.openstack.org/475650
    Use objects instead of SQLA deep copies in PortContext

Addressed by: https://review.openstack.org/485829
    Ignore duplicate record_resource_delete calls

Addressed by: https://review.openstack.org/489697
    Revert "Use push notification for security groups"

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.