Store configuration in ini/conf format compatible with other projects

Registered by Radomir Dopieralski on 2014-06-06

Summary
=======

Change Horizon's and OpenStack Dashboard's configuration to use INI files instead of Python code.

Motivation
========

There is a number of problems with the current approach. While the
code-based configuration gives us a lot of elasticity and freedom,
that comes at a cost:
- It's impossible to process the configuration files with any
  automated tools, such as migration scripts, validators, Puppet
  scripts, etc. without executing them inside the Horizon application.
  It's also practically impossible to load them and write them back in
  the same form, without losing all the code in them.
- The syntax of Python is much more fragile than the syntax of
  ``.ini`` files, the parsing errors are very confusing for the end
  users.
- Since the tests use a separate set of setting files, the code
  contained in our settings is never automatically tested.
- Since the setting file has the ability to import Python modules,
  there are several instances where we use Python objects as values for
  settings. This doesn't work well when the settings need to be printed,
  for example when debugging, and importing inside the settings file can
  lead do all sorts of hard to debug problems, such as cyclic
  dependencies, use of not fully initialized parts of the application,
  the need for deferred/lazy evaluation.
- We have some gettext-translated strings in there, which makes little
  sense, as they will not be translated anymore when the users change
  them, and they need to be translated to whatever language the UI user
  is using, not to the default language used when the configuration is
  being loaded.
- The Django settings are freely mixed with Horizon and OpenStack
  Dashboard settings, without any indication which is which.
- There is no schema, and no validation. A misspelled option name,
  incorrect type for the option's value, or another simple mistake can
  be very hard to find, and can actually stay unnoticed in a production
  environment for years.
- It's harder than it should be to include other files, add to
  existing values, etc. We have to rely on Python's import mechanisms,
  which are nontrivial to understand for someone who is not a Python
  programmer.

Description
========

Ironically, the way to avoid code in our settings files is by putting
code in our settings file. Specifically, we need to put some code in
the Django's ``settings.py`` file that would parse our ``.ini`` files
using ``oslo.config`` and pretend that all the values were defined in
it.
The ``oslo.config`` library needs a schema for every section that it
is going to parse. We will have the following sections:
- ``[django]`` for all configuration options specific to Django or its
  extensions. Since the full Django schema changes from version to
  version, and we need to support quite a few different versions of
  Django, we will only include the options that we are actually using in
  Horizon here.
- ``[horizon]`` for the ``HORIZON_CONFIG`` setting.
- ``[openstack]`` for all the other settings from
  ``local_settings.py``.
- ``[dashboard:name-of-the-dashboard]`` for the settings specific to
  a particular dashboard. Those are the ones that now live in the
  ``enabled`` directory.
- ``[type-of-extension:name-of-extension]`` in the future, we may need
  to support customizations for more elements of the dashboards, such as
  table columns and their sort order, widgets to be displayed on
  particular views, action buttons available in tables, etc.
Extensions to Horizon will be able to add their own sections to the
configuration, with their own schemas.
The current content of ``settings.py`` will be moved into the schema's
default values for all the setting options. It will then read the file
``local/local_settings.conf`` and then all the files matching
``local/local_settings.d/*.conf`` in alphabetical order. Settings
defined in a later file overwrite settings defined in an earlier one.
The ``oslo.config`` files allow interpolation of variables in the
option values. By default, the following variables will be defined:
- ``root_path``, the path to the base directory of OpenStack
  Dashboard,
- ``local_path``, the directory where ``local_settings.conf`` and
  ``local_settings.d`` are located.
Additional variables can be defined at the top of the setting files,
outside of any section. It's not possible to refer to the values of
options inside sections.

Schema And Documentation
------------------------

A schema describing all the options that we need has to be created.
Options that are not registered in the schema cannot be accessed. So
we need to compile a complete list of all options. Each option can
also have a default value and a help text. Our settings currently
don't have any help text, and the comments that sometimes are added to
them in the settings file need to be rewritten to make sense out of
context.

Complex Value Types
-------------------

Some of the currently used options take Python objects as values (such
as exception classes, functions, object instances). To support that,
we need custom option parsers, and a text format for describing them.
For instance, Horizon adds an ``Importable`` type, that takes a module
path to the Python object to be imported. Ideally, those settings
should be refactored to take a string with the module path directly.
Other options take just dictionaries or lists, but the internal
structure of those dictionaries or lists is quite complex. Since it's
not possible to express such a complex structure with the list and
dictionary formats provided by ``oslo.config``, Horizon defines two
types, ``JSONList`` and ``JSONDict`` that can be used in those cases.
Again, ideally, a simpler structure would be desirable, maybe using
more subsections, and failing that, a reference to an external JSON
file maybe?

Translatable Strings
--------------------

Some options use Gettext to translate their values. Obviously, Gettext
can only translate strings that it knows about, so that only makes
sense for defaults and has been preserved. The problems that Django
doesn't know what language is going to be used by the UI while loading
the config files is alleviated by Django's lazy evaluation of
translated strings. There is currently no way for the users to
translate the strings that they customize themselves inside the
settings files.

Backward Compatibility
----------------------

As mentioned in the rationale, there is currently no easy way to
automatically process Horizon's settings files, so there is no way to
migrate from the old configuration automatically. The users will have
to do it manually.
We can, however, leave the old settings code in place, at least for
one version, and use it to load settings if the
``local_settings.conf`` file is missing, issuing a warning asking the
users to switch to the new format.

Extensions And Plugins
----------------------

Any Horizon extensions that require their own configuration, will need
to register a schema for their options, preferably as a separate
section, and then to copy the options from the parser to the Django
settings, possibly doing some processing of their own (or just access
the ``oslo.config`` parser directly).
This means, that any extension would need some way of executing Python
code before, or while, the settings are loaded. It's not clear right
now how this should be accomplished.
An alternative would be for them to have their own configuration
files.

UX
===
This does not affect the user interface.

Outside Dependencies
=================

We will use oslo.config library for parsing the configuration files.

All projects that generate, modify, read or otherwise depend on the Horizon's configuration files will be affected.

Doc Impact
=========

The new settings format has to be documented, obviously. But there is
only one version of documentation online, and it's for all released
versions of OpenStack at once, just saying in which version an option
was added or removed. This probably won't work in this case, and we
will need two separate sections of documentation, one for the old
configuration, and one for the new one.

Blueprint information

Status:
Started
Approver:
Rob Cresswell
Priority:
Medium
Drafter:
Radomir Dopieralski
Direction:
Approved
Assignee:
Radomir Dopieralski
Definition:
Approved
Series goal:
Accepted for 14.0.0-rocky
Implementation:
Needs Code Review
Milestone target:
milestone icon rocky-2
Started by
Radomir Dopieralski on 2014-08-13

Related branches

Sprints

Whiteboard

We can either use oslo.config and write the shim ourselves, including the schemas for all versions of django that we support, or we can use a ready solution such as django-configglue. More research needed. -- Radomir Dopieralski, 2014-06-06

Gerrit topic: https://review.openstack.org/#q,topic:bp/ini-based-configuration,n,z

It seems that configglue isn't too good for our purposes after all. Trying with oslo.config now. -- Radomir Dopieralski, 2014-06-11

Addressed by: https://review.openstack.org/100521 ABANDONED
    Use oslo.config for Horizon and Django configuration

[david-lyle | 2015-03-17] Moving out of kilo.

Addressed by: https://review.openstack.org/402448 - Merged
    Specify POLICY_CHECK_FUNCTION as a string

Addressed by: https://review.openstack.org/403063
    Move warnings from settings.py to a separate function

Addressed by: https://review.openstack.org/403100
    Move the execution of local_settings.d snippets into a function

Addressed by: https://review.openstack.org/404117
    Move SECRET_KEY and LOCAL_PATH code out of settings.py

Addressed by: https://review.openstack.org/404735
    Move all Django configuration into .conf file

Implementation notes and planning: https://etherpad.openstack.org/p/horizon-ini
-- Radomir Dopieralski, 2017-03-14

Addressed by: https://review.openstack.org/462228 - Abandoned
    WIP

[yingzuo 2017-10-25] Radomir, are you still working on this?

[rdopiera 2017-10-26] I would love to keep working on this, though I'm blocked until the most recent patch merges.

[yingzuo 2017-10-26] Thanks Radomir. I think you are referring to https://review.openstack.org/#/c/404735. The test failure seems to be a false positive, so probably just need a recheck.

Addressed by: https://review.openstack.org/523416
    WIP: logging configuration using YAML file

Gerrit topic: https://review.opendev.org/#/q/topic:bp/ini-based-configuration

Addressed by: https://review.opendev.org/655206
    Define default settings explicitly (openstack_dashboard 1/5)

Addressed by: https://review.opendev.org/655207
    Define default settings explicitly (openstack_dashboard 2/5)

Addressed by: https://review.opendev.org/655208
    Define default settings explicitly (openstack_dashboard 3/5)

Addressed by: https://review.opendev.org/655209
    Define default settings explicitly (openstack_dashboard 4/5)

Addressed by: https://review.opendev.org/655210
    Define default settings explicitly (openstack_dashboard 5/5)

Addressed by: https://review.opendev.org/655211
    Define default settings explicitly (openstack_auth)

Addressed by: https://review.opendev.org/655212
    Move descriptions of openstack_auth settings

Addressed by: https://review.opendev.org/655213
    Remove unused settings in openstack_dashboard

Addressed by: https://review.opendev.org/655214
    Define remaining openstack_dashboard settings

Addressed by: https://review.opendev.org/655215
    Move openstack_dashboard specific settings from horizon

Addressed by: https://review.opendev.org/655216
    Define default settings explicitly (horizon)

Addressed by: https://review.opendev.org/655217
    Move horizon settings in openstack_dashboard.settings

Addressed by: https://review.opendev.org/655218
    Move default values defined in settings.py to defaults.py

Addressed by: https://review.opendev.org/656181
    Document the plan of ini-based-configuration

(?)

Work Items

Work items:
Remove code from settings: TODO
Separate Django settings: TODO
Generate defaults for Django settings: TODO
Use oslo-config for Django settings: TODO
Use oslo-config for plugin settings: TODO
Use oslo-config for local settings: TODO

This blueprint contains Public information 
Everyone can see this information.