Custom Language Packs

Registered by Devdatta Kulkarni on 2014-05-23

High-level description:
-------------------------------

This blueprint considers requirements, implementation, and changes to Solum CLI and API to support custom language packs.

This BP discusses custom language packs that are VMs.

Details:
----------
A language pack in Solum is essentially a base image (VM) with appropriate libraries, compilers, runtimes, and so on installed on it.

Custom language pack is a base image that has libraries and packages specified by the user (cloud operator and/or an application developer) installed on it.

Towards building such custom language packs Solum needs:
- the ability to specify the required base image type and version
- the ability to specify the required libraries and packages to be installed on the base image
- the ability to register the language pack with Glance

One way to implement this feature is to require that a user provide two things to Solum -
(a) base image name and type
(b) link to github repository with a script with a predefined name (such as 'prepare') in it, which
     contains installation instructions to install the required libraries and packages

Solum would provide a 'language pack builder' API to build the language pack and register it in glance.

At a high-level the builder API will do following actions in a secure environment:
- mount the specified base image
- clone the repository
- run the 'prepare' script
- snapshot the new image
- upload the image to glance

Implementation options, issues, and recommendation:
-------------------------------------------------------------------------

Option 1: Use disk-image-builder (dib) as the mechanism to build the language pack

The issues with this option are:
- Figuring out how do the contents of 'prepare' translate into dib element (pre-install.d, install.d, post-install.d, finalize.d, etc.)

- Figuring out how to run 'prepare' in a sandboxed environment. We need this as the contents
  of 'prepare' can be anything.

Option 2: Don't use disk-image-builder, but do similar to what we are currently doing as part of 'build-app'.

One of the advantages of Option 1 is that the images that it creates are compatible with glance.
One of the disadvantages is that Solum would need to build a translator to translate contents of 'prepare' in the dib's dsl.

One of the advantage of Option 2 is that no translation is required of the contents of the 'prepare' script. The disadvantage is that Solum would need to build mechanisms to create a Glance compatible image.

One approach to achieve advantages of both options without incurring their disadvantages is to
use dibs but avoid converting 'prepare' into various elements. Just convert it into install.d element.

Overtime Solum can support 'hints' which can be used to indicate how should 'prepare' be broken up to be used within different elements. It is possible that such hints are provided as separate scripts that are supported by Solum. For example, Solum may start supporting 'pre_install, prepare, post_install' scripts.

(devkulkarni, June 30, 2014):
Option 3: Provide Dockerfile
Advantages outlined in the Whiteboard section below:

Recommendation:
Start with Option 1 (dib). Add functionality in Solum to convert 'prepare' to 'install.d' element.

(devkulkarni, June 30, 2014):
recommendation is to use option 3

Proposed changes:
---------------------------

1) CLI changes:
     - Create a new command 'lp create' which will take in a 'lp description file'. It will be used as follows: solum lp create <lp description file>

    Contents of the <lp description file>:
    - Base image name and type
    - link to a github repository which has the 'prepare' script

    - Create a new command 'lp base-image show' which will list the supported base images in Solum.
    It will be used as follows:
    - solum lp base-image show

    Its output will be of following nature:
    [{Ubuntu:12.04},{Ubuntu:12.10},{Ubuntu:13.04},{Fedora:f19}]

2) API changes:
    - Start with our builder API. Modify it as required.

    - Add API to support 'base-image show'. It could be something like:
      GET /v1/solum/base-images/

QE requirements:
-------------------------

1) Test that the created language pack was installed with all the packages and libraries listed in the 'prepare' script.

References:
-----------------

Some discussions in this regard are available on:

https://etherpad.openstack.org/p/custom-language-packs

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Devdatta Kulkarni
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Implemented
Milestone target:
None
Started by
Adrian Otto on 2015-06-11
Completed by
Adrian Otto on 2015-06-11

Whiteboard

Roshan: Custom Language Pack should also be a mechanism to specify the source code build process (i.e. how to perform build of an F# or C++ source code to generate the binary). We should verify if this can be achieved with the current implementation approach.

After discussing with Adrian and PaulCzar we have come up with option 3 for implementing custom language packs.

Option 3:

User provides a Dockerfile instead of a 'prepare' script in their language pack repository. Solum builds the language pack via 'docker build'.

Note that the language pack created using this approach is going to be a Docker container and not a VM image.

There are several advantages of this option. It uses the mechanism of Dockerfile which has become standard in the Docker-based container systems for describing the installation requirements.
Second it avoids the problem of having to figure out how to map the contents of the 'prepare' script onto various elements needed by DiBs.

Towards using such a language pack in creating a DU (deployment unit), we have to consider the following.

If the operator has configured Solum to use 'docker' as the SOLUM_IMAGE_FORMAT then the Docker based LP will work without any issues.

However, if SOLUM_IMAGE_FORMAT is set to 'vm' then Heat/Nova will need to be configured to provision VMs that have a Docker driver installed on them.

(PaulCzar: could you please add a line or two here about the changes that need to be made to Devstack Heat for enabling this).

Additional considerations:
-----------------------------------
The question of why to separate out language pack creation step and not have the language pack Dockerfile listed in users' application code repository came up. While this is certainly possible there are several advantages of separating language pack creation from application building. First, language pack creation is a one time process. Clubbing it with application building will lead to build performance overhead. Second, by requiring application developers to define Dockerfiles in their application repositories Solum will be binding them in a contract that their code would work only when the operator has configured Solum to use Docker as the image format.

Recommendation:
-------------------------
Pursue option 3

Terms:
---------
1) LP (Language Pack): OS image (VM or container) with the required build and runtime libraries and packages installed on it.

2) DU (Deployment Unit): LP + application code

--------
aratim: What if the user does not have a git repo with the 'prepare' script in it ? What if he just needs standard java/python libraries in this language pack, how would he specify them? I think we need to think about this use case as well.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.