lava-dashboard fails with can't adapt type 'Decimal' deserialization error

I cannot reproduce this locally on natty (although I may have made a mistake so I'm checking again). From my reading of the code the only suspect is a bit of magic Django _may_ (I'm not certain this is true) be putting between us and postgresql in the adapter classes. From my naive checking postgresql adapter supports decimals just fine so something else must be at play.

The bundle that caused this is https://bugs.launchpad.net/lava-dashboard/+bug/850770. One can download it with the following command:

$ lava-dashboard-tool get --dashboard-url=http://validation.linaro.org/lava-server/RPC2/ 492c3fd101d748816d949b85cd458dc99becbf2d --output bug-850770.json

Bumped up the priority as this is causing lots of deserialization errors on the daily runs

Still trying to nail this one. It seems to work all the time. I did manage to reproduce it only once, the very first time I tried, then it succeeded.

Well, I think I have _some_ idea what's going on, although not much. Basically, the problem is that the _psycopg module has a process global dictionary of adapters, including the adapter for Decimal types. mod_wsgi runs all the various WSGI apps it is hosting in one process (or at least only a small number of processes) and so if one WSGI app messes with this adapter registry somehow, all the other hosted WSGI apps will break. I don't know which app we're hosting could do this, but _possibly_ it's the edge site. I notice that the edge .wsgi file uses the "activate_this" script from the virtualenv, but this is a bad idea with mod_wsgi:

http://code.google.com/p/modwsgi/wiki/VirtualEnvironments

It's possible that some of the manipulations activate_this does mean that importing the decimal module will fail, and if another app is unlucky enough to import psycopg2 module at this moment psycopg2 for that process forever more will not support decimals. This seems moderately unlikely, but it's the best guess I have.

Fixes that suggest themselves, in increasing order of invasiveness but also perhaps desirability:

1) Change the edge .wsgi script to not use activate_this.py
2) Switch to using mod_wsgi in daemon mode
3) Stop using mod_wsgi at all and ProxyPass to gunicorn or something.

Oh, this also suggests a workaround: if this starts happening a lot, restart apache, and the race probably won't happen on next startup. Any bundles that failed to deserialize should deserialize fine if you just try again after a restart.

Ah! the error does not seem to appear any more. The lava test execution for beagle board now completes.
This can be closed.

Thanks!!!
Deepti.

For the record: I removed our edge site so that currently apache runs only one python interpreter