Continued development of the Ubuntu error tracker

Registered by Evan

Integration with a hardware database.
Pig or Hive (Hadoop) support.
Backporting the reporting client (whoopsie) to previous Ubuntu versions.
Discuss the state of Upstart's inotify event support.
Further changes to the statistics website.
API requirements.
Sort out our story on mean time between failures (http://paste.ubuntu.com/957235/, http://instagr.am/p/KDe4WPhRxl).
Additional problem types (desktop application hangs, package installation failures, debconf dialogs during package installation, ...)
Providing an apport API for applications handling recoverable errors.
Surfacing architecture (and other data points) only when relevant
Should we continue to process and retrace crashes from derivatives?
Can we set /proc/sys/fs/suid/dumpable to 2? (programs not normally dumped are dumped, but only readable by root). We need some way of ensuring we never ever send cores from ssh-agent, among others.
...

Other crash database blueprints:
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-metrics
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-updates-from-crash-reports
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-fix-ddebs
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-bucketing-improvements
https://blueprints.launchpad.net/ubuntu/+spec/foundations-q-phased-updates

Additional background information:
http://paste.ubuntu.com/958213/

Specification (read this first):
http://wiki.ubuntu.com/ErrorTracker

Blueprint information

Status:
Started
Approver:
Steve Langasek
Priority:
High
Drafter:
Evan
Direction:
Approved
Assignee:
Evan
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Started
Milestone target:
milestone icon ubuntu-12.10
Started by
Kate Stewart

Whiteboard

(?)

Work Items

Work items:
[mpt] Design privacy settings for previous Ubuntu releases <https://wiki.ubuntu.com/ErrorTracker#settings>: DONE
[bryce] file a bug about whoopsie-daisy about Xorg not showing up in errors.ubuntu.com: DONE
[james-w] work with ev to determine why kernel crashes aren't showing up (for what it is worth brian-murray filtered out netdev watchdog messages from kernel oops reporting and it might be worth sending those to the error tracker): TODO
[bryce] work with ev to determine why GPU hangs are not being sent (appears to be working now with our latest xserver fault handling work): DONE
[brian-murray] work with ev to determine why package installation failures are not being reported: POSTPONED
[ev] Teach Daisy to handle package failure error reports: POSTPONED
[brian-murray] file a bug about whoopise to include the LiveMediaBuild key value pair in the interesting data for the error tracker: DONE
[kees] decide whether we can set /proc/sys/kernel/setuid_dumpable to 2 by default: TODO
[ev] Deploy Hadoop on Cassandra with Pig/Hive: POSTPONED
[ev] move to a copy-on-write approach for the retracer caches: POSTPONED
[ev] teach daisy's retracers to write to launchpad (synthesizing a bug) and phase out the launchpad retracers (crashdb.update_bug exists): DONE
[ev] Investigate hardware database options: POSTPONED
[ev] Implement a hardware database: POSTPONED
[ev] Create a basic "what's interesting about this problem?" section: POSTPONED
[ev] Feed hardware database information into the "what's interesting?" section: POSTPONED
[ev] Feed package fields into the "what's interesting?" section (low on disk space, arch, date, etc): POSTPONED
[ev] Work with the Landscape team to evaluate providing an option for automatic crash reporting: POSTPONED
[ev] Move whoopsie from long-running to using upstart's inotify events: BLOCKED
[ev] Implement "Last seen" field, with highlighting based on whether it's the most current version, or split apart fixed and still open problems into two tables: DONE
[ev] Backport whoopsie: POSTPONED
[ev] Develop end-to-end nagios checks that insert sample data (and then remove it): POSTPONED
[ev] Implement end-to-end system testing using juju and the canonistack/ec2: INPROGRESS
[ev] Write tests for the juju system testing framework: POSTPONED
[ev] Provide a mechanism for developers to signal they want to retain a core file of a future crash for further inspection: POSTPONED
[ev] Provide realtime updates to errors.ubuntu.com using txlongpoll: POSTPONED
[ev] Import developer to package and developer to team mappings and filter the default view on errors.ubuntu.com to problems the developer and their team are responsible for: POSTPONED
[ev] Ensure we're getting errors from package failures: DONE
[mpt] Design UI for application-specified errors <https://wiki.ubuntu.com/ErrorTracker#app-requested>: DONE
[ev] Provide a API for creating reports from recoverable errors: DONE
[mpt] Design UI for Debconf errors <https://wiki.ubuntu.com/ErrorTracker#debconf>: DONE
[ev] Support errors from debconf dialogs: INPROGRESS
[ev] Support errors from hangs using compiz: INPROGRESS
[ev] Replace mean time between failures with crashes for today divided by users for today: DONE
[ev] Speed up most common errors calculation by using a frequently calculated intermediary CF for the subset: INPROGRESS
[mpt] Design "Errors I've submitted" (link from Privacy settings, page on errors.ubuntu.com) <https://wiki.ubuntu.com/ErrorTracker#previous>: DONE
[ev] Implement "Errors I've submitted", using the dbus service with polkit to grab the sha512 of the system UUID: DONE
[ev] Change the default release filter to "All non-EOL Ubuntu releases": POSTPONED
[ev] Add fragment identifiers to URLs on errors.ubuntu.com that are updated when each page changes: POSTPONED
[mpt] Design date range selection on errors.ubuntu.com (bug 1023920): DONE
[ev] Implement date range selection on errors.ubuntu.com: DONE
[ev] Move to tastypie for the errors.ubuntu.com API: DONE
[ev] Create graphs for and build out operational API data (retracer counts, oops per day, etc): DONE
[ev] Evaluate moving to Celery from amqplib: POSTPONED
[ev] Clean up individual problem and instance pages: DONE

Dependency tree

* Blueprints in grey have been implemented.