Kernel 6.8 + zfs-2.2.2: copy_file_range Operation Not Supported

Status changed to 'Confirmed' because the bug affects multiple users.

I ran into the same issue here.

It's the same symptom. I'm running Noble with ZFS on root and kernel 6.8.

Setting up tomcat10-common (10.1.16-1) ...
Setting up tomcat10 (10.1.16-1) ...
Creating group 'tomcat' with GID 994.
Creating user 'tomcat' (Apache Tomcat) with UID 994 and GID 994.
Failed to backup /etc/group: Operation not supported
dpkg: error processing package tomcat10 (--configure):
 installed tomcat10 package post-installation script subprocess returned error exit status 1
Processing triggers for rsyslog (8.2312.0-3ubuntu3) ...
Processing triggers for libc-bin (2.39-0ubuntu2) ...
Errors were encountered while processing:
 tomcat10
E: Sub-process /usr/bin/dpkg returned an error code (1)

I can confirm that patch (https://github.com/openzfs/zfs/issues/15930) works as is with noble-level (just little fuzz in one hunk). I applied it to the noble-level at the end of patching and built dkms package and it works smoothly with that module.

https://git.launchpad.net/~antst/ubuntu/+source/zfs-linux/log/?h=linux_6.8_compat_splice_copy_file_range_fallback

this is branch with the backported patch (little change to avoid fuzz).
based on noble-devel branch.

Not sure if I need to open PR and how :)

Anybody who wants fix right now can do
git clone -b linux_6.8_compat_splice_copy_file_range_fallback https://git.launchpad.net/~antst/ubuntu/+source/zfs-linux
cd zfs-linux
dpkg-buildpackage -rfakeroot --no-sign -b

it will build all related packages, then install locally built zfs-dkms package as
apt install ./zfs-dkms_2.2.2-0ubuntu8_all.deb

Fix for systemd to work around this: https://github.com/systemd/systemd/pull/32341.

Think it is easier to fix ZFS with simple patch, than to fix dozens of packages :)

Also of note, ZFS 2.2.2 is not explicitly compatible with Linux 6.8. ZFS 2.2.3 is explicitly compatible with kernel 6.7, and preliminary support for 6.8.

I've tested an initial version with the upstream patch and attached the debdiff here. Will work to get this uploaded.

and a build is available at

https://launchpad.net/~john-cabaj/+archive/ubuntu/zfs-splice-fix/+packages

feel free to test before it is available in noble

oh, it needs the kernel module to be rebuilt for the fix.. don't think we have time for that anymore before the release.

if fixed zfs packages are pushed to the release, then user will have an option to use DKMS at least (as I do now)

Since kernel rebuilds are required for the ZFS fix to be readily available, I am uploading the systemd workaround to help mitigate the problem.

Could someone please advise how these issues can float a little higher before kernel freeze things happen? I tried to get this on the books quite early by doing a Ubuntu Root on ZFS setup and giving it a test. I know it's not directly related to the issue, but I tried to raise awareness of this quite a while ago with no interest. Is this the right place for reports like this? I knew this was going to affect an LTS release...so it would suck if this didn't work well just out of the box....if that makes sense. Especially ZFS -- users (I think) assume some amount of....integrity with that particular filesystem. If there's a newer version of the package, that is likely to work better with an LTS kernel........that seems like a better option? I am new at all of this, so I'm happy to take some criticism.

Yep, I agree that it would be important to push some number of backports like this one to kernel module before release. Ideally, 2.2.3 would be better for 6.8....

This bug was fixed in the package zfs-linux - 2.2.2-0ubuntu9

---------------
zfs-linux (2.2.2-0ubuntu9) noble; urgency=medium

  * debian/patches/0007-Linux-6.8-compat-use-splice_copy_file_range-for-fallback.patch
    - Linux 6.8 support(LP: #2058179)
  * debian/control:
    - add libtirpc-dev as build dependency

 -- John Cabaj <email address hidden> Thu, 18 Apr 2024 23:00:06 -0500

This might be user error, but I seem to still be running into issues with docker and this. I think I'm on the latest kernel and zfs versions, but I also see that one of them is older.

zfs --version:

zfs-2.2.2-0ubuntu9
zfs-kmod-2.2.2-0ubuntu7

uname -r:
6.8.0-28-generic

[00:41:58 ERROR]: Encountered an unexpected exception
org.spongepowered.configurate.ConfigurateException: []: java.io.IOException: Operation not supported
 at org.spongepowered.configurate.loader.AbstractConfigurationLoader.save(AbstractConfigurationLoader.java:205) ~[configurate-core-4.2.0-SNAPSHOT.jar:?]
 at io.papermc.paper.configuration.Configurations.trySaveFileNode(Configurations.java:111) ~[paper-1.20.4.jar:git-Paper-488]
 at io.papermc.paper.configuration.Configurations.initializeGlobalConfiguration(Configurations.java:136) ~[paper-1.20.4.jar:git-Paper-488]
 at io.papermc.paper.configuration.Configurations.initializeGlobalConfiguration(Configurations.java:106) ~[paper-1.20.4.jar:git-Paper-488]
 at io.papermc.paper.configuration.PaperConfigurations.initializeGlobalConfiguration(PaperConfigurations.java:198) ~[paper-1.20.4.jar:git-Paper-488]
 at net.minecraft.server.dedicated.DedicatedServer.initServer(DedicatedServer.java:209) ~[paper-1.20.4.jar:git-Paper-488]
 at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1131) ~[paper-1.20.4.jar:git-Paper-488]
 at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:319) ~[paper-1.20.4.jar:git-Paper-488]
 at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.io.IOException: Operation not supported
 at sun.nio.fs.LinuxNativeDispatcher.directCopy0(Native Method) ~[?:?]
 at sun.nio.fs.LinuxFileSystem.directCopy(LinuxFileSystem.java:159) ~[?:?]
 at sun.nio.fs.UnixFileSystem.copyFile(UnixFileSystem.java:682) ~[?:?]
 at sun.nio.fs.UnixFileSystem.copy(UnixFileSystem.java:1060) ~[?:?]
 at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:300) ~[?:?]
 at java.nio.file.Files.copy(Files.java:1304) ~[?:?]
 at org.spongepowered.configurate.loader.AtomicFiles.atomicBufferedWriter(AtomicFiles.java:85) ~[configurate-core-4.2.0-SNAPSHOT.jar:?]
 at org.spongepowered.configurate.loader.AtomicFiles.lambda$atomicWriterFactory$0(AtomicFiles.java:58) ~[configurate-core-4.2.0-SNAPSHOT.jar:?]
 at org.spongepowered.configurate.loader.AbstractConfigurationLoader.save(AbstractConfigurationLoader.java:188) ~[configurate-core-4.2.0-SNAPSHOT.jar:?]
 ... 8 more

I just updated and rebooted but I'm also still seeing the same problem: Failed to backup /etc/group: Operation not supported. It's reproducible with "apt install tomcat10".

Oh the new kernel wasn't built with the new zfs-dkms =/

II: dkms-build downloading zfs (zfs-dkms_2.2.2-0ubuntu7_all.deb)
II: fetching https://launchpad.net/ubuntu/+archive/primary/+files/zfs-dkms_2.2.2-0ubuntu7_all.deb
II: fixing https://launchpadlibrarian.net/718392332/zfs-dkms_2.2.2-0ubuntu7_all.deb
II: following http://launchpadlibrarian.net/718392332/zfs-dkms_2.2.2-0ubuntu7_all.deb
II: dkms-build-configure dkms-build-configure--zfs found, executing

Per https://irclogs.ubuntu.com/2024/04/20/%23ubuntu-release.html#t18:58 the required kernel is -31, which is currently in proposed

This bug was fixed in the package systemd - 255.4-1ubuntu8

---------------
systemd (255.4-1ubuntu8) noble; urgency=medium

  * copy: ignore -EOPNOTSUPP from copy_file_range() (LP: #2058179)
    File: debian/patches/lp2058179-copy-ignore-EOPNOTSUPP-from-copy_file_range.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=fc81bb680f5d5c8c5995fcf6ed7f55f739dcd6a7

 -- Nick Rosbrook <email address hidden> Fri, 19 Apr 2024 10:24:36 -0400

Kernel -31 fixed the problem for me. Thanks!

with 6.8-31 I was able to install 24.04 ubuntu-server-minimal with zfs boot & root using the OpenZFS tutorial for 22.04 with minimal change.

Since there's no ZFS root install script in subiquity yet the OpenZFS instructions are going to be the go-to for many people again.

https://github.com/openzfs/zfs/discussions/16120