Transaction Log - Communication Protocol Definition

Registered by Jay Pipes

This blueprint has been superseded. See the newer blueprint "Native Replication" for updated plans.

Define the Protobuffer messages to be used in communication protocol between subscriber and publisher for the transaction log's reference implementations of those plugins.

At a minimum, the protocol must support the following communication requests:

From Subscriber to Publisher:

Simple PING/heartbeat
Ask what is last transaction ID applied in a given channel
Ask for all transaction messages applied after a given transaction ID
Ask for continuous checksum for a table/channel/transaction ID combination

Blueprint information

Status:
Complete
Approver:
Jay Pipes
Priority:
Medium
Drafter:
Jay Pipes
Direction:
Approved
Assignee:
David Shrewsbury
Definition:
Superseded
Series goal:
None
Implementation:
Deferred
Milestone target:
None
Completed by
David Shrewsbury

Related branches

Sprints

Whiteboard

Got protocol messaging pretty much v1 done:

/**
 * The communication protocol between a subscriber
 * and a publisher
 */
message ReplicationCommunicationPacket
{
  enum Command
  {
    REQUEST_PING = 1; /* ping/heartbeat */
    RESPONSE_PING = 2;
    REQUEST_CHANNEL_APPLIED = 2; /* get last transaction ID applied on a channel */
    RESPONSE_CHANNEL_APPLIED = 3; /* response the last transaction ID a publisher applied */
    REQUEST_TRANSACTION = 4; /* sent by subscriber to request a stream of transactions */
    RESPONSE_TRANSACTION_HEADER = 5; /* information about transaction messages to be sent */
  }
  message RequestChannelApplied
  {
    required uint32 channel_id = 1; /* the channel in question */
  }
  message ResponseChannelApplied
  {
    required uint32 channel_id = 1; /* the channel in question */
    required uint64 transaction_id = 2; /* the transaction ID last applied on the publisher */
  }
  message RequestTransaction
  {
    required uint32 channel_id = 1; /* the channel in question */
    required uint64 transactions_after_id = 2; /* subscriber is requesting transactions AFTER this ID. */
    required uint32 max_number_transactions = 3; /* response no more than this number of transaction messages in response */
  }
  message ResponseTransactionHeader
  {
    required uint32 channel_id = 1; /* the channel in question */
    required uint64 min_transaction_id = 2; /* first transaction ID of transaction messages to be sent */
    required uint64 max_transaction_id = 3; /* last transaction ID of transaction messages to be sent */
    required uint32 num_transactions = 4; /* total number of transactions to be sent */
  }
  required Command command = 1; /* the type of command in this packet */
  optional RequestChannelApplied request_channel_applied = 2;
  optional ResponseChannelApplied response_channel_applied = 3;
  optional RequestTransaction request_transaction = 4;
  optional ResponseTransactionHeader response_transaction_header = 5;
}

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.