neutron

Pure python driven linux network configuration

Registered by Li Ma

Currently, Linux network configuration in Neutron heavily relies on shell commands, like ip, brctl, etc. Shell commands makes Neutron agents inefficient and really hard to operate in high load environment.

There is a blueprint that switch openflow operations from shell to ryu-based pure python library, but it is not sufficient. I'd like to introduce a pure Python netlink library: pyroute2. It supports network configuration including ip-link, ip-route, ip-netns, tc, ipset and iptables in the roadmap, and is also compatible with python3. It only requires standard library which is also awesome, because you don't need to rely on other unstable third-party libraries that makes dependency hard to maintain. Moreover, it supports transactional local DB operations for network configuration called IPDB.

I strongly suggest to switch all the Linux-based network configuration commands to this pure python library.

Doc Link: http://docs.pyroute2.org/general.html
Pypi Link: https://pypi.python.org/pypi/pyroute2

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Li Ma
Direction:
Needs approval
Assignee:
Li Ma
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
Armando Migliaccio

Related branches

Sprints

Whiteboard

Nov-30-2015(nick-ma-z): OK. Actually I think it may need a bp or devref. One the one hand, there should be a place to put all the outputs of performance and switching from external commands to library calls. On the other hand, it may greatly affect the debugging and operations. A related documentation is needed here.

By the way, I'll start to work on performance evaluation. However, this feature relies on oslo.privsep to be released. I'm also watching the progress.

Nov-24-2015(armax): no need for a bp.
----------
Can you provide some data using cProfile on the runtime of the fork-exec-wait for shell commands like ip ? I measured them to be roughly 0.05 seconds per command which limits the overall performance/scalability of the agent to less than 20 such operations per second. Such data would help strongly motivate this work.

Answer:
Here's the discussion link: https://bugs.launchpad.net/neutron/+bug/1492714

Thanks for your reviewing, you can follow the link above to join discussion. Generally, the problems are got from our production environments which heavily relies on auto-orchestration and auto-scaling. I cannot touch such an environment without permission from our clients. I just reads the monitoring system and message queues. However, as far as I know, single command based on subprocess is always OK, but many many concurrent external commands with external lock mechanism is not always OK for production. It consumes much more memory, it gives kernel much more pressure to manage processes, etc. if we can improve it for sure, from subprocess to python library call, it is better.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.