cloud-init-local.service messages not written to /var/log/cloud-init.log in systemd
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| cloud-init |
Fix Released
|
Medium
|
Unassigned | ||
| cloud-init (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
| Xenial |
Fix Released
|
Medium
|
Unassigned | ||
| Yakkety |
Fix Released
|
Medium
|
Scott Moser | ||
Bug Description
=== Begin SRU Template ===
[Impact]
Cloud-init's logging is inconsistent due to availability of syslog during
boot.
Cloud-init logs to /var/log/
a way that was originally designed to prefer to use syslog if it was
available, and then fall back to writing directly to that file.
Over time this has been shown to be problematic.
a.) it relied on syslog during boot, and on some distros it wasn't
present.
b.) sometimes it would not be available during cloud-init-
and then would be during cloud-init.service. The result was that
the log would have two different time stamp formats (one written
by rsyslog and one by python logging).
c.) if rsyslog was used, micro seconds were not included in the log.
d.) since the move to systemd, there has even been times when cloud-init's
attempt to determine if syslog was available would false-positive.
that would result logging not being written to the file at all.
Over all, the complexity was just not found to worth the benefit.
[Test Case]
* Launch an instance.
* Look at /var/log/
on start, the cloud-int process logs a message like
'Cloud-init v 0.7.8 running'
Look at those messages specifically. In the example here (lxd), neither
cloud-
# grep Cloud-init /var/log/
Dec 2 18:06:56 y2 [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.8 running 'modules:config' at Fri, 02 Dec 2016 18:06:56 +0000. Up 5.0 seconds.
Dec 2 18:06:58 y2 [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.8 running 'modules:final' at Fri, 02 Dec 2016 18:06:58 +0000. Up 7.0 seconds.
Dec 2 18:06:58 y2 [CLOUDINIT] util.py[DEBUG]: Cloud-init v. 0.7.8 finished at Fri, 02 Dec 2016 18:06:58 +0000. Datasource DataSourceNoCloud [seed=/
* update to proposed, cleanup reboot
# enable propose and update
# cleanup
sudo rm -Rf /var/log/
sudo reboot
* login again and look.
This time, all messages will have the format:
2016-12-02 17:58:43,175 - util.py[DEBUG]: Cloud-init v. 0.7.8 running 'init-local' at Fri, 02 Dec 2016 17:58:43 +0000. Up 13.73 seconds.
And you will have one for each 'init-local', 'init', 'modules:config' and
modules:final.
[Regression Potential]
Users relying on cloud-init writing entries to syslog will lose that.
[Other Info]
=== End SRU Template ===
output of cloud-init-
The result is that there is no output in /var/log/
There is some information in https:/
about how this occurrs and how it used to work.
copying part of that here:
Cloud-init's logging basically employed a "try syslog and fallback to direct log to file".
The proposed "just log to a file" is definitely dramatically simpler and advantageous in some cases.
The way the "try syslog and fallback" works (or worked) on Ubuntu up until systemd was:
a.) cloud-init init --local
1. read logging config,
2. attempt to log to syslog ([ *log_base, *log_syslog ])
3. that fail, so it log to file directly
b.) cloud-init init
1.) rsyslog would have /dev/log up functional at this point
2.) cloud-init logging config read and ends up logging to syslog
Systemd changed some things in teh way /dev/log was handled, and the above no longer worked well.
Additionally, cloud-init installs a file /etc/rsyslog.
The value of doing this in this way was that we use syslog, so if the user had configured the system to log remotely, cloud-init's logs would go to that remote system as they desired.
If we directly log to a file, then cloud-init's log messages will not without further configuration go to syslog.
One other thing to be aware of is that cloud-init can itself configure rsyslog through cloudinit/
Related branches
- cloud-init Commiters: Pending requested
-
Diff: 38 lines (+12/-4)1 file modifiedconfig/cloud.cfg.d/05_logging.cfg (+12/-4)
| Scott Moser (smoser) wrote | #1 |
| Changed in cloud-init: | |
| status: | New → Confirmed |
| importance: | Undecided → Medium |
| status: | Confirmed → Fix Committed |
| Changed in cloud-init (Ubuntu): | |
| status: | New → Confirmed |
| status: | Confirmed → Fix Released |
| importance: | Undecided → Medium |
| Changed in cloud-init (Ubuntu Yakkety): | |
| status: | New → In Progress |
| importance: | Undecided → Medium |
| assignee: | nobody → Scott Moser (smoser) |
| Changed in cloud-init (Ubuntu Xenial): | |
| status: | New → Confirmed |
| importance: | Undecided → Medium |
| description: | updated |
| Changed in cloud-init (Ubuntu Xenial): | |
| status: | Confirmed → In Progress |
| status: | In Progress → Triaged |
| Scott Moser (smoser) wrote | #2 |
| Changed in cloud-init: | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Please test proposed package | #3 |
| Changed in cloud-init (Ubuntu Yakkety): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed |
| Scott Moser (smoser) wrote | #4 |
| tags: |
added: verification-done removed: verification-needed |
| Andy Whitcroft (apw) wrote Update Released | #5 |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in cloud-init (Ubuntu Yakkety): | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Please test proposed package | #7 |
| Changed in cloud-init (Ubuntu Xenial): | |
| status: | Triaged → Fix Committed |
| tags: | removed: verification-done |
| tags: | added: verification-needed |
| Brian Murray (brian-murray) wrote | #8 |
| Scott Moser (smoser) wrote | #9 |
| tags: |
added: verification-done removed: verification-needed |
| Launchpad Janitor (janitor) wrote | #10 |
| Changed in cloud-init (Ubuntu Xenial): | |
| status: | Fix Committed → Fix Released |
| James Falcon (falcojr) wrote | #11 |
I'm going to merge https:/
That will mean that by default cloud-init.service does nto send output to rsyslog, but we'll get consistent timestamps in /var/log/