manage.jujucharms site stability and scalability improvements
[GOAL]
Bring stability, scalability, trustworthiness to manage.
[RATIONALE]
manage.
Blueprint information
- Status:
- Complete
- Approver:
- Gary Poster
- Priority:
- Essential
- Drafter:
- None
- Direction:
- Approved
- Assignee:
- None
- Definition:
- Obsolete
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
- Curtis Hovey
Whiteboard
* manage.
- Ingest must only process chams that are genuinely in the store.
- Ingest must reconcile its past collection of charms with the store.
- Ingest must not overwrite the good copy of a charm when there are processing errors.
- Ingest must recognise promulgated charms.
- Ingest may record the last known good copy of a charm if the charm is in the store,
but the charm is broken.
- Ingest must not store Jenkins test data that will not be used.
- Ingest should be observable, ie errors should get recorded and should have a admin ui
view to see what the errors are. (oopt tools?)
- Charm branches should not be processed if they haven’t changed.. which goes along with...
- Ingest should use the modified parameter to getBranchTips
- Ingest should use the series tip parameter on the LP result to determine branch tips. The
charm existance should be verified against the charm store still as a branch tip may still be
an invalid charm.
- Be efficient about pulling only data from the store that has not already been pulled. Ie. only
pull stats for new dates on charms.
* manage.
- model charm revisions as separate entity to properly support the API, at the moment
just returns current for historical information, ie lies.
- split out bzr/branch revision history into separate table. We’re generating reports on charm
activity from these, to be accurate we need the full history instead of the current behavior
of the last 10.
- we also want to correlate charm revision to bzr revision. this info is the charm store response.
- proper promulgation support for non ~charmers owners.
* manage.
- The app must not fail with bad charms. If the charm is in the store, the UI/API must support it.
(eg maintainer is required by proof, but not the store).
* manage.
- The charm must have nagios checks to very all crucial functions are up.
- MongoDB must install from a dump file.
- MongoDB log at a lower level. (It currently spews debug-level).
* Performance
- Do scaling tests and respond to performance concerns
- Kapil is concerned that serving file content does not scale. ie, serving raw large
binaries through the api could be a ddos, plus many redundant versions of the
file possible.
[USER STORIES]
(The ~jujugui teams's juju-gui charm is not shown as the recommended version)
As a dev-op,
I want to see the reviewed charms owned by non-charmers
so that I know which charms are recommend to deploy.
(the charms stat is wrong when the charm is unpromulgated then deleted from Lp)
As a dev-op,
I don't want to see unrecommended charms as reviewed
so that I am informed when charms are unpromulgated.
(Charms temporarily disappear when ingest cannot talk to other services)
As a dev-op,
I want to see the last known versions of charms
because I need to work even when Lp, the store, or jenkins are unavailable.
(manage.
As a member of webops
I want reliably update the charms in the manage.
So that I do not need to manually start or add relations to bring the site up.
Nb. ElasticSearch times out during restarts/upgrades.
(The mongodb charm collection is bloated, 99% of the data is never read)
As a member of webops
I don't want to providing storage for unused data
because storage is expensive and backup/restores must be fast.
(Show information about older revisions of charms)
As a dev-op
I want to see the data about an older version of charm
So that I can learn about the version I have in my local environment.
(Ingest is slow, delaying the time to see updated charms)
As a charm author
I want my charm to discovered quickly after it is published
so that I can see it in the charm browser.
(manage.
As a member of webops
I want the charmworld app to respond to many requests quickly
because proxies only help when the app can provide enough information to cache.
(Ingest errors should be observable to developers and admin)
As a developer of manage.
I want to see the errors ingest encounters
So that I can produce reports about quality, and plan fixes to ingest.
[ASSUMPTIONS]
[RISKS]
[IN SCOPE]
[OUT OF SCOPE]
[USER ACCEPTANCE]
[RELEASE NOTE/BLOG]
Work Items
Work items:
[ce-orange-squad] The ~jujugui teams's juju-gui charm is not shown as the recommended version: DONE
[ce-orange-squad] the charms stat is wrong when the charm is unpromulgated then deleted from Lp: DONE
[ce-orange-squad] Charms temporarily disappear when ingest cannot talk to other services: DONE
[ce-orange-squad] manage.
[ce-orange-squad] The mangodb charm collection is bloated, 99% of the data is never read: DONE
[ce-orange-squad] Ingest is slow, delaying the time to see updated charms: DONE
[ce-orange-squad] Show information about older revisions of charms: INPROGRESS
[ce-orange-squad] manage.
[ce-orange-squad] Ingest errors should be observable to developers and admin: TODO
