Implement apt package pinning

Overview
########

Package pinning helps to provide a 'stable' environment, which is defined as an environment in which changes to the base infrastructure are as controlled as possible.

Note that this blueprint complements the concept of providing a 'frozen repository'. A 'frozen repository' aims to provide a repository which is updated in a controlled fashion. Providing a frozen repository will be handled in a separate blueprint.

Package pinning is a complementary control layer on top of a frozen repository which provides the capability to keep packages at the same level even when the repository has been updated. Package pinning provides the capability to do controlled updates of packages from a (frozen or not frozen) repository, enabling the deployment of updated selected packages from the repository as and when required to selected groups of servers.

Problem Description
-------------------

In the past it has been discovered that unexpected updates to apt packages have resulted in a destabilised environment.

When a deployer has executed a playbook in order to change configuration or deploy a new host, the result has unexpectedly been that packages get updated. This introduces new problems in the environment that are unrelated to the intended change.

There is a need to allow specified key packages to be kept at a certain version and only updated when specifically intended. This need also includes ensuring that any new hosts added to the environment have the same versions of the same key packages as the rest of the environment.

Proposed Change
---------------

I propose that we make use of the debops.apt_preferences Ansible Galaxy role (https://galaxy.ansible.com/list#/roles/1552) in order to handle the actual apt configuration to enable pinning. Documentation for the role is available here: http://docs.debops.org/en/latest/ansible/roles/debops.apt_preferences.html

The advantages of making use of this role are:
1 - It's already written, so we'll be able to join a broader community of developers that share needs and development skills instead of building a role in isolation and carrying the development ourselves.
2 - The role is already in an Ansible Galaxy format.
3 - The role is tested by a test suite for every commit, and therefore is likely to be of high quality.
4 - The debops collection of roles may also be useful for other infrastructure services. It has, for instance, already been suggested as a solution for spec/improved-network-generation.

Playbooks need to be provided which do updates of packages on designated host groups. These updates should respect the package pinning rules.

Playbook Impact
---------------

Playbooks will need to be added which provide the capability to execute apt upgrades to packages which are already deployed on each host. It must be possible to deliberately execute these updates separately from any installation of configuration plays.

Alternatives
------------

1 - Continue to do without pinning in any way. This would continue to cause frustrating delays in development as gate checks fail due to unrelated changes. It would also continue to cause frustration in production environments
2 - Only implement infrastructure to provide a 'frozen repository'. While this would have the effect of 'pinning' packages at a specific version, it would not provide the capability to
3 - Implement controlled image-based deployment for all containers, hosts, etc. This is something that would be great, but is a far more complex implementation.

Security Impact
---------------

If the package pinning list is not actively maintained by the deployers and the upgrade plays are not executed on a regular schedule then the deployed environment will have the risk of being compromised due to known security issues. This will need to be highlighted in documentation.

Performance Impact
------------------

None.

End User Impact
---------------

None.

Deployer Impact
---------------

There would need to be documentation produced which covers how to:
 - review and change the package pinning list
 - how to run the upgrade configuration plays independently from the install/configuration plays
 - how to upgrade from the method used in icehouse/juno to this new method

Gate checks would need to include:
 - running the upgrade playbooks
 - checking that the pinning preferences were respected
 - completing functional tests after the upgrade

Developer Impact
----------------

Developers would need to routinely update any pinning preferences set in the code repository to ensure that they stay current enough without impacting the development process.