Nailgun performance testing, emulate hundreds of agents

Registered by Evgeniy L

When user have a cluster with hundreds of nodes nailgun start to work very slow because agents on this nodes make requests to nailgun. As a result not all of nodes can be served by nailgun and part of them go to offline.

Need to emulate an arbitrary amount of agents and to calculate request statistics.

Example of agent's request you can find in log file /var/log/nailgun/api.log
Firstly agent sends POST request and then PUT.
http://paste.openstack.org/show/61239/

Blueprint information

Status:
Started
Approver:
Evgeniy L
Priority:
Medium
Drafter:
Evgeniy L
Direction:
Approved
Assignee:
Igor Shishkin
Definition:
Approved
Series goal:
Accepted for future
Implementation:
Started
Milestone target:
milestone icon next
Started by
Evgeniy L

Related branches

Sprints

Whiteboard

ishishkin:
Hello, we just tested nailgun for performance.

Test description:
POST /api/nodes/ with unique data of node.

Load testing results:
Tested two configurations:
a) Python HTTP server - currently used in nailgun
b) uWSGI application under nginx frontend written by Nikolay M.

Linear load line(1, 1000, 10m)
a: disorder at 80rps
b: disorder at 117rps
Disordered because of end of CPU idle. In configuration “a” csw level was higher. Configuration b disordered because of internal nailgun operations.

Combined load line(N, N+10, 30s), const(N+10, 2m) and go on from 1 to 1000rps
a: disorder at 18rps, response time over 3s. disordered because of csw level at 140-150k. CPU idle at 82%.
b: disorder at 50rps. Load at 10rps with peaks up to 100ms response time. 30% of 502’s at 50rps because of no CPU idle left. 50% of 502’s at 60rps. 60% of 502’s at 70rps.

NOTES:
 * Postgres metadata dir moved to tmpfs
 * Stop test conditions:
   * response time > 3s for last 10s
   * network errors
   * 30% of 5xx errors for last 1s

evgeniyl:
Hi Igor, we have merged new patch for agent handler
https://github.com/stackforge/fuel-web/commit/0a75052b3c504cdd015f217f6b4c4226ad44fb3c
Could you please test new version of handler (/nodes/agent/)? With uWSGI.

ishishkin:
Hello again,
PUT /api/nodes/agent linear http://loadosophia.org/examples/121070/
PUT /api/nodes/agent stairs http://loadosophia.org/examples/121081/

evgeniyl:

Could you please add documentation and add scripts in some fuel repo?

ishishkin:
Nikolay M. wanted to do this. He has all of necessary stuff.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.