Parallel DNS lookups

Registered by Morten Brekkevold

Several pages in NAV's web interface perform potentially large numbers of DNS reverse lookups in serial; among them IP Device Info and Machine Tracker. This does not scale well, and causes inordinately slow page loads. NAV should be able to performs DNS lookups asynchronously (in parallel). This can either be implemented using the GNU adns library (python-adns) or Twisted's Names library (by driving the reactor manually, since we're in a synchronous environment). ipdevinfo and machinetracker should have access to a simple API which hides the details of selecting and using either a parallel or serial method of making DNS queries.

Blueprint information

Morten Brekkevold
Morten Brekkevold
Christian Strand Young
Series goal:
Accepted for 3.10
Milestone target:
milestone icon 3.10.0
Started by
Christian Strand Young
Completed by
Morten Brekkevold

Related branches



adns does not support IPv6, reverse lookups work, but that is just luck. The only sulution as I see it is to use this: which adds ipv6 support to adns, then we would have to fork adns-python and rewrite the binding to support the changes in adns. , this now has support for looking up quad A. I have also talked with the original author, which might be releasing a new version.

Turns out it was more than good enough to use twisted for the async lookups. This works like a charm and scales very well.


The whole thing might fail with twisted 11.0 ref.
To use twisted in something that is not a twisted process, is kind of bad. The implementation will also block everything until every deferred is resolved. Starting and stopping the reactor in the same process should not work, and is a bug before 11.0. If the same process is used for multiple requests from the API, the whole thing can crash.

The implementation now with deferredList works and is better than sequential lookups, but the use of twisted is not quite right after a chat with some "gurus".

- Inline callbacks should be used for better readability.
- We could nudge the reactor. Not sure how with the deferredlist.


The API fails when implemented in ipdevinfo. This is because ipdevinfo does a forward-lookup first, then a reverse-lookup. They are called from the same twisted process, and therefore the reactor is asked to start twice in the same process, which is not possible.

A quick solution is to use getaddrinfo() for the forward lookups and asyncdns for the reverse, since reverse is the slowest one.

A test with both forward and reverse lookups on with the methodes getaddrinfo() and socket.gethostbyaddr() gives the following time results
real 0m0.122s
user 0m0.016s
sys 0m0.012s

Pageload timings:

navdev with async

navdev without async

Maybe something else seems to be the issue with ipdevinfo?


The latest code from Christian starts and stops the reactor, which cannot be done multiple times during the same process. This is why the original suggestion was to drive the reactor loop manually using reactor.iterate. Yes, we know it's a dirty hack, but it works. I've refactored the entire asyncdns module and tested it manually and with the Machine Tracker changes and it seems to be working fine now.

So the only thing missing now is an implementation for ipdevinfo (which I'm guessing you've already tried working on, according to the whiteboard)


Work Items

This blueprint contains Public information 
Everyone can see this information.


No subscribers.