Streamline glance images upload via Horizon

Registered by Timur Sufiev on 2015-09-02

Summary
=======
To upload Glance images directly to Glance, bypassing Horizon's Django views and provide status of upload in UI. There are 2 implementations of this feature in Horizon: for Django [legacy] content and for new Angular content.

Motivation
=========
Piping large Glance images through Horizon incurs the risk of exhausting space on the Controller node, because usually it's more limited the the space of the Storage node used by Glance.

Also from an operator perspective the service of uploading images has different characteristics than serving the dashboard. The dashboard can live and thrive on 1Gbps links possible on the controllers nodes themselves. Glance on the other hand may have to be deployed on service nodes possibly with multiple 10Gbps links to handle the user data, storage etc.

Finally, transfer of gigabyte-sized images takes a significant time and in a modern web it seems a waste of time to upload it in a synchronous way thus blocking all other operations in dashboard. Even if the user could open another tab/window to continue doing some other things in Horizon, that isn't a good pattern. Hence, it's better to upload images asynchronously and provide the progress of upload in the UI, so the user is aware that in order to upload to finish he should not close the page/tab.

Description
=========
[Django] First we need to expunge the 'Image File' field from the list of fields managed by Django, so the contents of file field doesn't even get to Django. Then using some cleverly written JS code, attached to the 'Create Image' form, an asynchronous request directly to Glance API should be initiated as soon as form is considered valid by Django. In order for request to succeed both Horizon and Glance must play nice around CORS headers.

[Angular] No need to do all the magic specific to Django, since we could upload the image directly to Glance service using ng-file-upload library.

UX
===
[Django] A standard spinner modal element which is shown on form submit needs to be enhanced with the progress bar.

[Angular] Create Image modal doesn't hide until the upload is complete, with upload progress meter is being updated constantly and shown instead of the 'Select File...' button.

Wireframes, Mocks, Videos and UI Markup
---------------------------------------------------------------
TBD

Testing
======
Should be covered both with unit and integration tests.

Outside Dependencies
==================
The client-side code relies on CORS being enabled for Glance service (otherwise browser would forbid the PUT request to a location different from the one form content came from). In a Devstack setup you'll need to edit [cors] section of glance-api.conf file, setting `allowed_origin` setting to the full hostname of the web server (say, http://<HOST_IP>/dashboard) and restart glance-api process.

Requirements Update Required
========================
N/A

Doc Impact
=========
[Django] HORIZON_IMAGES_UPLOAD_MODE new setting being equal to 'direct' defines if the new behavior for Django is enabled. If it's equal to 'legacy', then the old approach is used. If it's equal to 'off', then upload from a local file is disabled alltogether.

[Angular] Angular Create Image file upload uses the same setting HORIZON_IMAGES_UPLOAD_MODE, the semantics is the same.

Blueprint information

Status:
Complete
Approver:
Rob Cresswell
Priority:
Medium
Drafter:
Timur Sufiev
Direction:
Approved
Assignee:
Timur Sufiev
Definition:
Approved
Series goal:
Accepted for 10.0.0-newton
Implementation:
Implemented
Milestone target:
milestone icon newton-3
Started by
Rob Cresswell on 2016-03-30
Completed by
Rob Cresswell on 2016-08-17

Related branches

Sprints

Whiteboard

Noted by david-lyle during team meeting the new glance tasks API may be useful in this scenario https://wiki.openstack.org/wiki/Glance-tasks-import and avoid token re-issue complexity -- Sep 23, 2015

[tsufiev, Sep 23, 2015] According to https://wiki.openstack.org/wiki/Glance-tasks-import section Formerly Open Questions (with Answers!), question 1 there is no possibility of direct upload using tasks-import API. Thus ruling out tasks-import for our goals.

Also I've spoken with mfedosin, he confirmed that fixing long-living image operations to not interrupt due to Keystone token expiration is planned for Mitaka, Keystone trusts will be used. And yes, that's a Glance responsibility in that case.

Gerrit topic: https://review.openstack.org/#q,topic:bug/1403129,n,z

Addressed by: https://review.openstack.org/230434
    [WIP] Glance images async upload

[tsufiev, Oct 16, 2015] One more thing to consider here: Glance API expects the image contents being uploaded to consist entirely of raw binary data, while using combination of ajax + FormData() results in having binary data prepended with 'multipart/form-data' header and related stuff. I found this out after comparing md5sums of the local image and the image again downloaded from Glance on controller node (after successful upload from browser). So the sums were different, then examining contents of uploaded image revealed extra header.

So the next idea here is to slice large images into chunks and send each chunk (as a binary string) with a separate ajax request. For this to happen, Glance should support partial uploads - not a big deal, I suppose (could be done with HTTP headers, like Content-Length etc). So I'm going first to discuss this first with Glance folks.

Again, the current limitation as I see it: either you use FormData() and get it encoded as multipart/form-data (but it handles uploading large files for us), or you should read binary content into browser's memory and send it without extra headers. Please correct me if I missed something.

[tsufiev, Nov 24, 2015] Waiting for multipart/form-data parser be implemented by Glance team. Resumable upload was considered more complicated solution (not preferred one).

Gerrit topic: https://review.openstack.org/#q,topic:bug/1403129-patch9,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1403129-patch15,n,z

Addressed by: https://review.openstack.org/317365
    Embed support for external data sinks into api.glance

Gerrit topic: https://review.openstack.org/#q,topic:create-image-file-upload,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/horizon-glance-large-image-upload,n,z

Addressed by: https://review.openstack.org/317456
    [NG] Support local file upload in Create Image workflow

Addressed by: https://review.openstack.org/330392
    [WIP] See how CORS works for Image Create + upload

Addressed by: https://review.openstack.org/317457
    [NG] Enhance Create Image workflow with upload tracking

Gerrit topic: https://review.openstack.org/#q,topic:bug/1467890,n,z

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.