journal is broken in unprivileged LXC and nspawn containers

Upstream discussion: http://lists.freedesktop.org/archives/systemd-devel/2015-May/032113.html

Status changed to 'Confirmed' because the bug affects multiple users.

Fixed upstream:
  http://cgit.freedesktop.org/systemd/systemd/commit/?id=417a7fdc418
  http://cgit.freedesktop.org/systemd/systemd/commit/?id=01906c76c

However, there's one more detail to fix in unprivileged containers:

root@v:/# getpcaps $$
Capabilities for `608': = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,37+ep

The cap_audit_* are a lie, the audit subsystem in current kernels isn't namespace aware and thus unprivileged containers can't have these caps. The failed systemd-journald-audit.socket unit there isn't a big deal, but this should be fixed in LXC.

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-vivid&id=8db397eef12d6

Quoting Martin Pitt (<email address hidden>):
> The cap_audit_* are a lie, the audit subsystem in current kernels isn't

To be pedantic, it is not a lie - you have that capability against your
own user namespace, but the only check for that capability is explicitly
against the initial user namespace.

But it certainly seems the easiest (short-term) workaround is to drop
that capability. Unfortunately that will be tough coordinate with the
(soon-coming) namespaced audit. If we drop it now in container configs,
how do we tell userspace to re-enable it when available. The cleaner
way from our pov would be for systemd to check using bind() whether it
has the access. Then as soon as the kernel provided the ability to
do that in a non-init userns, containers could use it.

To put it another way, the check for capability bounding set is always
explitily a check for capabilities against your user namespace. If the
question is "can I read audit logs", then "do I have CAP_AUDIT_READ in
my bounding set" is simply the wrong check.

> To be pedantic, it is not a lie - you have that capability against your own user namespace,

Ah, so that says "you can do it", but it's never actually going to work? I guess that's just another expression of audit not working in namespaces then..

> Unfortunately that will be tough coordinate with the (soon-coming) namespaced audit.

Ooh, is that coming? Then I guess we shouldn't bother much, it's not an important problem. For the most part unpriv containers work fine now.

This bug was fixed in the package systemd - 219-10ubuntu1

---------------
systemd (219-10ubuntu1) wily; urgency=medium

  * Merge with Debian experimental branch. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Add debian/udev.lvm2.service to avoid running the dummy lvm2 init
      script.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also run ifup in the background during boot,
      to avoid blocking network.target. (LP: #1425376)
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Drop hwdb-update dependency from udev-trigger.service, which got
      introduced in v219-stable. This causes udev and plymouth to start too
      late and isn't really needed in Ubuntu yet as we don't support stateless
      systems yet and handle hwdb.bin updates through dpkg triggers. This can
      be dropped again with initramfs-tools 0.117.
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor dep to the Ubuntu version where it moved to /lib.
    - Lower apparmor Breaks: to the Ubuntu version that dropped $remote_fs.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being triggered by Jenkins, but is totally
     ...

Hello Martin, or anyone else affected,

Accepted systemd into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/219-7ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I can confirm that unprivileged vivid containers now start properly with the package from proposed.

This bug was fixed in the package systemd - 219-7ubuntu6

---------------
systemd (219-7ubuntu6) vivid; urgency=medium

  * Fix assertion crash with empty Exec*= paths. (LP: #1454173)
  * systemd-fsckd autopkgtest: Stop assuming that
    /etc/default/grub.d/90-autopkgtest.cfg exists.
  * systemd-fsckd autopkgtest: Add missing plymouth test dependency.
  * debian/tests/boot-smoke: Allow 10 seconds for systemd jobs to settle down.
  * Fix "tentative" state of devices which are not in /dev (mostly in
    containers), and avoid overzealous cleanup unmounting of mounts from them.
    (LP: #1444402)
  * journal: Gracefully handle failure to bind to audit socket, which is known
    to fail in namespaces (containers) with current kernels. Also
    conditionalize systemd-journald-audit.socket on CAP_AUDIT_READ.
    (LP: #1457054)
  * Add sigpwr-container-shutdown.service: Power off when receiving SIGPWR in
    a container. This makes lxc-stop work for systemd containers.
    (LP: #1457321)

 -- Martin Pitt <email address hidden> Thu, 21 May 2015 14:47:46 +0200

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.