Anonymous statistics performance testing

Registered by Igor Shishkin on 2014-10-13

It's required to measure performance for anonymous statistics server side application:

BP: https://blueprints.launchpad.net/fuel/+spec/send-anon-usage

Load tests should be performed for Collector service which is used for gathering and storing raw Fuel usage statistics information:

https://github.com/stackforge/fuel-stats

Yandex-tank will be used as testing tool for that. It requires third-party utility for creating random requests data (e.g. JSON schema with Fuel action logs or installation info).

Load test will be performed manually using specially created parameterized Jenkins' job:

 http://jenkins-product.srt.mirantis.net:8080/view/All/job/fuel_stats_performance_testing_collector/

RPS (requests per second) will be used as the metric. This measures the approximate number of Fuel installations from that collector service can receive statistics.

There are three stages of performance testing that should be performed:

1) Get threshold RPS value:
 - produce load on collector which growths linearly from 1 to some huge unreal RPS value (e.g. 10,000) with +1 step;
 - get the RPS max value (Threshold) which collector can handle correctly: less than 0,1% of 502 or connection timeout errors;
 - assume that collector service passed this stage of tests if 50% Threshold value is higher than expected max RPS produced by real Fuel installations (for example we can use 100,000 as greatest possible Fuel installations number, so expect up to 48 requests per seconds to collector from them according to normal (Gaussian) probability distribution, because Fuel will send statistics not more than once per hour).

2) Run long-term load test using the following test plan:
 - produce constant load using RPS = 50% of Threshold value;
 - periodically linearly change RPS value for +/- 25% for short period of time (< 5 minutes);
 - consider that collector service passed this stage of tests if not more than 0,01% requests to it failed.

3) Run stress tests:
 - perform 2nd step again using huge or invalid requests (JSON data);
 - measure response latency after adding 1, 5 and 10 millions of records to the collector database;
 - assume that collector service passed this stage of tests if not more than 1% of valid requests failed and average latency is less that timeout value.

Blueprint information

Status:
Complete
Approver:
Alexander Kislitsky
Priority:
High
Drafter:
Igor Shishkin
Direction:
Approved
Assignee:
Andrey Sledzinskiy
Definition:
Review
Series goal:
Accepted for 6.1.x
Implementation:
Implemented
Milestone target:
milestone icon 6.1
Started by
Artem Panchenko on 2014-12-10
Completed by
Andrey Sledzinskiy on 2015-04-16

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/anonymous-statistics-performance-testing,n,z

Addressed by: https://review.openstack.org/130468
    Add requests templates for performance testing

Addressed by: https://review.openstack.org/152014
    Add request for OSWL collector

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.