Store configuration in ini/conf format compatible with other projects
Summary
=======
Change Horizon's and OpenStack Dashboard's configuration to use INI files instead of Python code.
Motivation
========
There is a number of problems with the current approach. While the
code-based configuration gives us a lot of elasticity and freedom,
that comes at a cost:
- It's impossible to process the configuration files with any
automated tools, such as migration scripts, validators, Puppet
scripts, etc. without executing them inside the Horizon application.
It's also practically impossible to load them and write them back in
the same form, without losing all the code in them.
- The syntax of Python is much more fragile than the syntax of
``.ini`` files, the parsing errors are very confusing for the end
users.
- Since the tests use a separate set of setting files, the code
contained in our settings is never automatically tested.
- Since the setting file has the ability to import Python modules,
there are several instances where we use Python objects as values for
settings. This doesn't work well when the settings need to be printed,
for example when debugging, and importing inside the settings file can
lead do all sorts of hard to debug problems, such as cyclic
dependencies, use of not fully initialized parts of the application,
the need for deferred/lazy evaluation.
- We have some gettext-translated strings in there, which makes little
sense, as they will not be translated anymore when the users change
them, and they need to be translated to whatever language the UI user
is using, not to the default language used when the configuration is
being loaded.
- The Django settings are freely mixed with Horizon and OpenStack
Dashboard settings, without any indication which is which.
- There is no schema, and no validation. A misspelled option name,
incorrect type for the option's value, or another simple mistake can
be very hard to find, and can actually stay unnoticed in a production
environment for years.
- It's harder than it should be to include other files, add to
existing values, etc. We have to rely on Python's import mechanisms,
which are nontrivial to understand for someone who is not a Python
programmer.
Description
========
Ironically, the way to avoid code in our settings files is by putting
code in our settings file. Specifically, we need to put some code in
the Django's ``settings.py`` file that would parse our ``.ini`` files
using ``oslo.config`` and pretend that all the values were defined in
it.
The ``oslo.config`` library needs a schema for every section that it
is going to parse. We will have the following sections:
- ``[django]`` for all configuration options specific to Django or its
extensions. Since the full Django schema changes from version to
version, and we need to support quite a few different versions of
Django, we will only include the options that we are actually using in
Horizon here.
- ``[horizon]`` for the ``HORIZON_CONFIG`` setting.
- ``[openstack]`` for all the other settings from
``local_
- ``[dashboard:
a particular dashboard. Those are the ones that now live in the
``enabled`` directory.
- ``[type-
to support customizations for more elements of the dashboards, such as
table columns and their sort order, widgets to be displayed on
particular views, action buttons available in tables, etc.
Extensions to Horizon will be able to add their own sections to the
configuration, with their own schemas.
The current content of ``settings.py`` will be moved into the schema's
default values for all the setting options. It will then read the file
``local/
``local/
defined in a later file overwrite settings defined in an earlier one.
The ``oslo.config`` files allow interpolation of variables in the
option values. By default, the following variables will be defined:
- ``root_path``, the path to the base directory of OpenStack
Dashboard,
- ``local_path``, the directory where ``local_
``local_
Additional variables can be defined at the top of the setting files,
outside of any section. It's not possible to refer to the values of
options inside sections.
Schema And Documentation
-------
A schema describing all the options that we need has to be created.
Options that are not registered in the schema cannot be accessed. So
we need to compile a complete list of all options. Each option can
also have a default value and a help text. Our settings currently
don't have any help text, and the comments that sometimes are added to
them in the settings file need to be rewritten to make sense out of
context.
Complex Value Types
-------------------
Some of the currently used options take Python objects as values (such
as exception classes, functions, object instances). To support that,
we need custom option parsers, and a text format for describing them.
For instance, Horizon adds an ``Importable`` type, that takes a module
path to the Python object to be imported. Ideally, those settings
should be refactored to take a string with the module path directly.
Other options take just dictionaries or lists, but the internal
structure of those dictionaries or lists is quite complex. Since it's
not possible to express such a complex structure with the list and
dictionary formats provided by ``oslo.config``, Horizon defines two
types, ``JSONList`` and ``JSONDict`` that can be used in those cases.
Again, ideally, a simpler structure would be desirable, maybe using
more subsections, and failing that, a reference to an external JSON
file maybe?
Translatable Strings
-------
Some options use Gettext to translate their values. Obviously, Gettext
can only translate strings that it knows about, so that only makes
sense for defaults and has been preserved. The problems that Django
doesn't know what language is going to be used by the UI while loading
the config files is alleviated by Django's lazy evaluation of
translated strings. There is currently no way for the users to
translate the strings that they customize themselves inside the
settings files.
Backward Compatibility
-------
As mentioned in the rationale, there is currently no easy way to
automatically process Horizon's settings files, so there is no way to
migrate from the old configuration automatically. The users will have
to do it manually.
We can, however, leave the old settings code in place, at least for
one version, and use it to load settings if the
``local_
users to switch to the new format.
Extensions And Plugins
-------
Any Horizon extensions that require their own configuration, will need
to register a schema for their options, preferably as a separate
section, and then to copy the options from the parser to the Django
settings, possibly doing some processing of their own (or just access
the ``oslo.config`` parser directly).
This means, that any extension would need some way of executing Python
code before, or while, the settings are loaded. It's not clear right
now how this should be accomplished.
An alternative would be for them to have their own configuration
files.
UX
===
This does not affect the user interface.
Outside Dependencies
=================
We will use oslo.config library for parsing the configuration files.
All projects that generate, modify, read or otherwise depend on the Horizon's configuration files will be affected.
Doc Impact
=========
The new settings format has to be documented, obviously. But there is
only one version of documentation online, and it's for all released
versions of OpenStack at once, just saying in which version an option
was added or removed. This probably won't work in this case, and we
will need two separate sections of documentation, one for the old
configuration, and one for the new one.
Blueprint information
- Status:
- Started
- Approver:
- Rob Cresswell
- Priority:
- Medium
- Drafter:
- Akihiro Motoki
- Direction:
- Approved
- Assignee:
- Akihiro Motoki
- Definition:
- Approved
- Series goal:
- Accepted for 16.0.0-train
- Implementation:
-
Needs Code Review
- Milestone target:
-
stein-3
- Started by
- Radomir Dopieralski
- Completed by
Related branches
Sprints
Whiteboard
We can either use oslo.config and write the shim ourselves, including the schemas for all versions of django that we support, or we can use a ready solution such as django-configglue. More research needed. -- Radomir Dopieralski, 2014-06-06
Gerrit topic: https:/
It seems that configglue isn't too good for our purposes after all. Trying with oslo.config now. -- Radomir Dopieralski, 2014-06-11
Addressed by: https:/
Use oslo.config for Horizon and Django configuration
[david-lyle | 2015-03-17] Moving out of kilo.
Addressed by: https:/
Specify POLICY_
Addressed by: https:/
Move warnings from settings.py to a separate function
Addressed by: https:/
Move the execution of local_settings.d snippets into a function
Addressed by: https:/
Move SECRET_KEY and LOCAL_PATH code out of settings.py
Addressed by: https:/
Move all Django configuration into .conf file
Implementation notes and planning: https:/
-- Radomir Dopieralski, 2017-03-14
Addressed by: https:/
WIP
[yingzuo 2017-10-25] Radomir, are you still working on this?
[rdopiera 2017-10-26] I would love to keep working on this, though I'm blocked until the most recent patch merges.
[yingzuo 2017-10-26] Thanks Radomir. I think you are referring to https:/
Addressed by: https:/
WIP: logging configuration using YAML file
Gerrit topic: https:/
Addressed by: https:/
Define default settings explicitly (openstack_
Addressed by: https:/
Define default settings explicitly (openstack_
Addressed by: https:/
Define default settings explicitly (openstack_
Addressed by: https:/
Define default settings explicitly (openstack_
Addressed by: https:/
Define default settings explicitly (openstack_
Addressed by: https:/
Define default settings explicitly (openstack_auth)
Addressed by: https:/
Move descriptions of openstack_auth settings
Addressed by: https:/
Remove unused settings in openstack_dashboard
Addressed by: https:/
Define remaining openstack_dashboard settings
Addressed by: https:/
Move openstack_dashboard specific settings from horizon
Addressed by: https:/
Define default settings explicitly (horizon)
Addressed by: https:/
Move horizon settings in openstack_
Addressed by: https:/
Move default values defined in settings.py to defaults.py
Addressed by: https:/
Document the plan of ini-based-
Addressed by: https:/
Enable OPENSTACK_
Work Items
Work items:
Remove code from settings: TODO
Separate Django settings: TODO
Generate defaults for Django settings: TODO
Use oslo-config for Django settings: TODO
Use oslo-config for plugin settings: TODO
Use oslo-config for local settings: TODO