Security Logging Object (SLO)

Requirement
1. Security events including traffic session ACCEPTs and DROPs due to enforcement of policy and security groups have to be logged to Analytics. Configuration control is required to selectively enable logging for sessions matching specified policy rules or security groups.
2. Currently, flows are logged independently. To reduce the amount of data sent from vRouter Agent to Analytics, the logging will henceforth be done on a session basis combining the data for both the forward and reverse flows in a single message.
3. Even with reduced information flowing into analytics, the flows should be debuggable at all entry and exit points (no information loss)

Each session record will have three parts
 1. Record key
 2. Forward and Reverse Flow logging data
 3. Forward and Reverse Flow sampling data

The flow sampling algorithm continues as is with the change that session records are sent to Analytics when a flow is selected by the sampling algorithm. The flow sampling rate can be configured as is done currently (in R3.x releases) to control the export of flow data export from vRouter Agent to Analytics.

Based on provisioned configuration, the flow log records may go to local files on the compute node or are sent to Analytics. Local log files created for this are managed by rotating regularly.

In the analytics node, changes will be needed to handle the new session log data. Additionally the session log data will be indexed only against the source and destination index table to reduce the writes (currently each flow record is indexed against 5 tables). The schema for the Source and Destination index tables have to be changed, so that they can be queried against all possible cases and should not result in any information loss. The query process/ write process has to change to accommodate the new index tables and removal of the other index tables viz., flowtableprotdpver2,flowtableprotspver2, flowtablevrouterver2. All the queries against the sessions which are logged should be specified explicitly while querying (if left unspecified, query will be performed against the sampled data). This will change the implementation of where query processing which looks up the respective index tables against which queries are issued. With new index table, processing of query involving FlowSeriesTables have to change; necessary logic to be added to fetch values even if the FlowSeriesQueries are to be queried against the secondary or tertiary indices viz., port no, protocol,..

Configuration impact
A new Security-logging object (SLO) will be introduced.
¥ SLO can be created at global level or at tenant level
¥ Tenant quota for SLO
¥ SLO can be attached to
        ¥ Network (log hits that have matching rule in SLO for all interfaces in this network)
        ¥ Interface (log all hits that have matching rule in SLO, for this interface.)
¥ Enabled globally at tenant level or global level depending where the object was created.
¥ Fields of SLO
        ¥ List of {(security-group, rule-uuid, rate)}
        ¥ List of {(network-policy, rule-uuid, rate)}
                ¥ Rule-uuid can be *, which implies all rules from the policy or security group
                ¥ Rate controls how many flows are logged. The first session in every R (rate) number of sessions matching the SLO will be logged. When the rate is set to 1, all sessions are logged.
        ¥ Enable/disable this SLO
        ¥ SLO rate, which is the logging rate if rate is not specified in the rule list
¥ Knob to enable/disable security logging at global level.

Analytics-api impact:
For querying flows, an option to specify sampled session record or logged session record should be added.