tail -0f /var/log/kern.log never prints anything (livecd cow overlayfs)

Tuomas Heino wrote:
> Public bug reported:
>
> Doing tail -f on /var/log/kern.log (which gets appended every 10 seconds) does not print anyting.
> May be an issue with the livecd / how overlayfs works. In any case tail should at least complain if underlaying fs does not properly support whatever -f needs to work (inotify below?):
>
> open("/var/log/kern.log", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0640, st_size=10956666, ...}) = 0
> lseek(3, 0, SEEK_CUR) = 0
> lseek(3, 0, SEEK_END) = 10956666
> fstat(3, {st_mode=S_IFREG|0640, st_size=10956666, ...}) = 0
> fstatfs(3, {f_type=0x1021994, f_bsize=4096, f_blocks=2042790, f_bfree=2003669, f_bavail=2003669, f_files=2042790, f_ffree=2041029, f_fsid={0, 0}, f_namelen=255, f_frsize=4096}) = 0
> inotify_init() = 4
> inotify_add_watch(4, "/var/log/kern.log", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1
> fstat(3, {st_mode=S_IFREG|0640, st_size=10956666, ...}) = 0
> read(4,

Thanks for the report.
In upstream coreutils development I've just made a change to fix that:
if the type of FS is unknown, it uses polling rather than inotify.

What does this print for that system?

    $ stat -f --format %t:%T /var/log

If it is a new non-local file system type, then we should add it
to stat.c's table.

In the mean time, you can use tail's deliberately undocumented
---disable-inotify option.

ubuntu@ubuntu:~$ stat -f --format %t:%T /var/log
1021994:tmpfs

[ following up to this report,
  http://bugs.launchpad.net/+source/coreutils/+bug/908354
  ]

Tuomas Heino wrote:
> ubuntu@ubuntu:~$ stat -f --format %t:%T /var/log
> 1021994:tmpfs

Thanks.

Unlike most file system types, overlayFS appears to have no magic number.
Now we're seeing how using files on such a file system can cause trouble.
I see no direct way to distinguish this file (for which inotify does not
work) from any other on a tmpfs file system, for which inotify works
just fine.

Ugly work-around: tail -f could try using both inotify and polling,
and, eventually, if polling spots a change for which there was no
inotify event, it would give up on using inotify for that file.
As soon as tail sees an inotify event for a file, however, it could
not stop polling: for most remote FS types, inotify works with changes
generated locally, but not with those generated remotely. So polling
can still be useful when using inotify.

We'll see...

Marked Triaged/Medium; this is reported upstream as http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10376. Thanks to Jim for the quick answer.

for reference, related overlayfs report: bug 882147