Store bundles in a compressed form

Registered by Michael Hudson-Doyle on 2011-12-08

Result bundles are currently just dumped on disk, but JSON is highly redundant and even gzip with default settings achieves a compression factor >90%. If we store them compressed, having them around on developer machines becomes less of a burden and if we don't have to compress as part of backing up, backups will be faster and more intelligent strategies than "tar up the whole media" directory become possible.

Blueprint information

Status:
Started
Approver:
Paul Larson
Priority:
Medium
Drafter:
None
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Blocked
Milestone target:
None
Started by
Paul Larson on 2012-01-31

Related branches

Sprints

Whiteboard

[2012-12-20, mwhudson] after casually asserting that django had a built-in compressing file storage class a while ago, I found out that I was wrong. So we'll have to write one -- it shouldn't be too hard though.
[2012-02-14, mwhudson] setting status to blocked until celery is deployed.

(?)

Work Items

Work items:
write a storage class that gzip's the content: DONE
add compressed filefield to model: DONE
change code to look for compressed bundle first: DONE
write celery job to compress a bundle: TODO
run job to compress existing bundles: TODO
change bundle-receiving code to kick off compressing job: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.