Ubuntu
zfs-linux package

zfs diff: Unable to determine path or stats for object

Bug #1849665 reported by Andreas Hasenack
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
High
Colin Ian King
Eoan
Fix Released
Undecided
Unassigned

Bug Description

== SRU Justification, Eoan ==

Using zfs diff on an encrypted dataset with large objects one can hit an error such as follows:

# zfs diff nsnx/trusty-2a@snap1 nsnx/trusty-2a
+ /nsnx/trusty-2a/bin
Unable to determine path or stats for object 5 in nsnx/trusty-2a@zfs-diff-32359-000000010001f165: File exists

== Fix ==

Upstream commit d359e99c38f667 ("diff_cb() does not handle large dnodes") as addressed in ZFS bug fix: https://github.com/zfsonlinux/zfs/pull/9343

== Testcase ==

# mkdir /zfs-test
# cd /zfs-test
# truncate -s 10G file.img
# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,304365 s, 138 MB/s
# zfs snapshot tank/d1@s1
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,305324 s, 137 MB/s

Without the fix, one hits an error such as:

# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists

With the fix, we get:
+ /tank/d1/somedata2.bin
M /tank/d1/

== Regression Potential ==

This is a minor change in module/zfs/dmu_diff.c and it only affects the zfs diff component, so this should not affect ZFS in terms of file system corruption/data loss. This has also been upstream regression tested and passes the Ubuntu ZFS regressions tests too. So the risk is limited.

---------------------

Eoan 19.10
zfsutils-linux 0.8.1-1ubuntu14
kernel 5.3.0-19-generic #20-Ubuntu

When using zfs diff on an encrypted dataset, I frequently encounter this error:

# zfs diff nsnx/trusty-2a@snap1 nsnx/trusty-2a
+ /nsnx/trusty-2a/bin
Unable to determine path or stats for object 5 in nsnx/trusty-2a@zfs-diff-32359-000000010001f165: File exists

I believe this to be upstream bug https://github.com/zfsonlinux/zfs/issues/7678, fixed with https://github.com/zfsonlinux/zfs/pull/9343

Here is one way to reproduce it:

# mkdir /zfs-test
# cd /zfs-test
# truncate -s 10G file.img
# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,304365 s, 138 MB/s
# zfs snapshot tank/d1@s1
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,305324 s, 137 MB/s
# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists

There may be a simpler way to test this, but this should be enough to start with.

See original description
Andreas Hasenack (ahasenack)
description: updated
Colin Ian King (colin-king)
Changed in zfs-linux (Ubuntu):
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
status: New → In Progress
Revision history for this message
Colin Ian King (colin-king) wrote :

Confirming that upstream commit https://github.com/zfsonlinux/zfs/commit/d359e99c38f66732d42278c32d52cfcf1839aa4f fixes this issue.

Revision history for this message
Colin Ian King (colin-king) wrote :

With the fix:

root@eoan-amd64-efi:/home/cking# mkdir /zfs-test
root@eoan-amd64-efi:/home/cking# cd /zfs-test
root@eoan-amd64-efi:/zfs-test# truncate -s 10G file.img
root@eoan-amd64-efi:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@eoan-amd64-efi:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@eoan-amd64-efi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.238499 s, 176 MB/s
root@eoan-amd64-efi:/zfs-test# zfs snapshot tank/d1@s1
root@eoan-amd64-efi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.228746 s, 183 MB/s
root@eoan-amd64-efi:/zfs-test# zfs diff tank/d1@s1 tank/d1
+ /tank/d1/somedata2.bin
M /tank/d1/

Colin Ian King (colin-king)
description: updated
Revision history for this message
Colin Ian King (colin-king) wrote :

I'll get this uploaded into -proposed once the current SRU backlog is out.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Thanks!

Revision history for this message
Colin Ian King (colin-king) wrote :

uploaded zfs-linux (0.8.1-1ubuntu14.1) eoan (will land in -proposed sometime soon)
uploaded zfs-linux (0.0.1.1ubuntu16) focal

Once the packages are uploaded the dkms driver component will be sync'd into the next kernel and then once this is in -proposed it can be fully tested.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.1-1ubuntu16

---------------
zfs-linux (0.8.1-1ubuntu16) focal; urgency=medium

  * 4600-diff_cb-does-not-handle-large-dnodes.patch
    - fix zfs diff on snapshots with large dnodes (LP: #1849665)
      Minor backport of upstream ZFS commit d359e99c38f667
      ("diff_cb() does not handle large dnodes")

 -- Colin Ian King <email address hidden> Wed, 24 Oct 2019 16:07:23 +0100

Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Andreas, or anyone else affected,

Accepted zfs-linux into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.1-1ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Eoan):
status: New → Fix Committed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Eoan verification

First, reproducing the bug:

root@nsn7:~# mkdir /zfs-test
root@nsn7:~# cd /zfs-test/
root@nsn7:/zfs-test# truncate -s 10G file.img
root@nsn7:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@nsn7:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,425336 s, 98,6 MB/s
root@nsn7:/zfs-test# zfs snapshot tank/d1@s1
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,260445 s, 161 MB/s
root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Now installing the updated kernel and rebooting into it.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Hm, at first glance this didn't work with the new kernel and zfsutils-linux package:

root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-21-generic #22-Ubuntu SMP Tue Oct 29 22:55:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@nsn7:/zfs-test# dpkg -l|grep linux-image
ii linux-image-5.3.0-18-generic 5.3.0-18.19+1 amd64 Signed kernel image generic
ii linux-image-5.3.0-19-generic 5.3.0-19.20 amd64 Signed kernel image generic
ii linux-image-5.3.0-21-generic 5.3.0-21.22 amd64 Signed kernel image generic
ii linux-image-generic 5.3.0.21.24 amd64 Generic Linux kernel image
root@nsn7:/zfs-test#

This was with the existing pool and dataset from the previous test.

Let me try restarting it from scratch:
root@nsn7:/zfs-test# cd
root@nsn7:~# zpool destroy tank
root@nsn7:~# rm -rf /zfs-test/
root@nsn7:~# mkdir /zfs-test
root@nsn7:~# cd /zfs-test
root@nsn7:/zfs-test# truncate -s 10G file.img
root@nsn7:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@nsn7:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,39306 s, 107 MB/s
root@nsn7:/zfs-test# zfs snapshot tank/d1@s1
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,265228 s, 158 MB/s
root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-21-generic #22-Ubuntu SMP Tue Oct 29 22:55:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

root@nsn7:/zfs-test# apt-cache policy linux-image-5.3.0-21-generic zfsutils-linux
linux-image-5.3.0-21-generic:
  Installed: 5.3.0-21.22
  Candidate: 5.3.0-21.22
  Version table:
 *** 5.3.0-21.22 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
zfsutils-linux:
  Installed: 0.8.1-1ubuntu14.1
  Candidate: 0.8.1-1ubuntu14.1
  Version table:
 *** 0.8.1-1ubuntu14.1 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.1-1ubuntu14 500
        500 http://br.archive.ubuntu.com/ubuntu eoan/main amd64 Packages

Same thing.

Revision history for this message
Colin Ian King (colin-king) wrote :

So this is a 2-phase fix. The dkms package is updated, then we test this, then this gets sync'd into the kernel. I'm testing it right now, let me sanity check the zfs-dkms part first and get that updated as step #1.

Revision history for this message
Colin Ian King (colin-king) wrote :

Installed the new zfsutils + zfs dkms to sanity check the kernel driver part of the fix:

dmesg | grep ZFS
[ 22.420188] ZFS: Loaded module v0.8.1-1ubuntu14.1, ZFS pool version 5000, ZFS filesystem version 5

And now the test:

root@eoan-amd64-uefi:~# mkdir /zfs-test
root@eoan-amd64-uefi:~# cd /zfs-test
root@eoan-amd64-uefi:/zfs-test# truncate -s 10G file.img
root@eoan-amd64-uefi:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@eoan-amd64-uefi:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@eoan-amd64-uefi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.319657 s, 131 MB/s
root@eoan-amd64-uefi:/zfs-test# zfs snapshot tank/d1@s1
root@eoan-amd64-uefi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.312195 s, 134 MB/s
root@eoan-amd64-uefi:/zfs-test# zfs diff tank/d1@s1 tank/d1
M /tank/d1/
+ /tank/d1/somedata2.bin

The zfsutils + dkms package has the fix. Once this lands we can then sync this into the next kernel release for the complete fix.

tags: added: verification-done-eoan
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.1-1ubuntu14.1

---------------
zfs-linux (0.8.1-1ubuntu14.1) eoan; urgency=medium

  * 4600-diff_cb-does-not-handle-large-dnodes.patch
    - fix zfs diff on snapshots with large dnodes (LP: #1849665)
      Minor backport of upstream ZFS commit d359e99c38f667
      ("diff_cb() does not handle large dnodes")

 -- Colin Ian King <email address hidden> Wed, 24 Oct 2019 16:07:23 +0100

Changed in zfs-linux (Ubuntu Eoan):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.