Sentry logging and monitoring for LAVA

Registered by Zygmunt Krynicki on 2012-02-21

We want to know about application failures, errors, warnings and crashes throughout the LAVA stack. We have identified sentry http://sentry.readthedocs.org/en/latest/index.html as the solution to deploy and integrate with our codebase.

Blueprint information

Status:
Started
Approver:
Paul Larson
Priority:
Medium
Drafter:
Zygmunt Krynicki
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Blocked
Milestone target:
milestone icon 2012.03
Started by
Zygmunt Krynicki on 2012-02-21

Whiteboard

[pwlars, 2012.02.21] Marking milestone for 2012.03 there are just a few things left to do but this is mostly complete
[zkrynicki, 2012-02-21] Everything is setup on http://sentry.validation.linaro.org/ Because each event has request data (including cookies) we have limited access to closed circle of trusted people. If you need access please contact either me (Zygmunt Krynicki) or Paul Larson.
[zkrynicki, 2012-03-07] I've managed to hose the old sqlite database by accident. I've setup fresh DB for sentry and updated production to use that.
[zkrynicki, 2012-03-07] I've ran across a bug in sentry. https://github.com/dcramer/sentry/pull/375 this is now reported upstream.
[zkrynicki, 2012-03-22] I've postponed the remaining tasks. I'll have another look at the startup scripts shortly after 12.04 is released. Setting implementation to deployment.
[fboudra, 2012-08-07] Set implementation deployment->blocked , affected by LP: #962766

Meta:
Headline: LAVA Stack can be monitored with Sentry, a real-time event logging and aggregation platform
Acceptance: Events from all server side components, apart from lava-test, can be seen on one aggregated page
Roadmap id: LAVA2012-LAB-MONITORING

(?)

Work Items

Work items:
Deploy sentry: DONE
Ensure sentry starts up after reboot: POSTPONED
Request a new DNS entry via RT: DONE
[fboudra] Create a private mailing list for sentry notifications: DONE
Configure sentry to send even notifications to the private mailing list: POSTPONED
Make sentry private: DONE
Setup production and staging projects in sentry: DONE
Create LAVA plugin for sentry client, raven (https://launchpad.net/lava-raven): DONE
Deploy lava-raven in production: DONE
Setup SENTRY_DSN for production: DONE
Ensure that dispatcher issues are logged to sentry: DONE
Ensure that scheduler issues are logged to sentry: DONE
Upgrade sentry to 3.x: DONE
Move sentry to postgresq for storage: DONE
Create sentry user/vhost on rabbitmq: DONE
Move sentry to celery for processing: DONE
Discover a bug in celery: DONE
Open merge request upstream: DONE
Re-create accounts for users that ought to have access here: POSTPONED

This blueprint contains Public information 
Everyone can see this information.