zfs mount -a: double free / memory corruption / segfault when mountpoint of dataset is not empty

Bug #1902588 reported by Lurf Jurv
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
High
Andrea Righi
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
Hirsute
Fix Released
High
Andrea Righi

Bug Description

== SRU Justification Focal ==

zfs mount -a when run on a nonempty mountpoint causes a double free, memory corruption, and a segfault.

== Impact ==

Double free and memory corruption in ZFS when run as root and attempting to mount all. While running this I observed other ZFS volumes randomly unmounting, and mount points owner being spuriously zeroed (set to root).

== Fix ==

https://github.com/openzfs/zfs/commit/d1b84da8c1a69c084f04b504beefe804591bca07

== Test ==

Steps are laid out in the ZFS issue: https://github.com/openzfs/zfs/issues/9560

== Regression Potential ==

Limited to the behavior of zfs mount when a previous attempt to mount has failed, or is still in progress. Changes the behavior in that case to failure, instead of double-free.

Example case of running into this bug, with dmesg: https://pastebin.com/YRXW8WgM

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal

$ apt-cache policy zfsutils-linux
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.4
  Candidate: 0.8.3-1ubuntu12.4
  Version table:
 *** 0.8.3-1ubuntu12.4 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

Revision history for this message
Andrea Righi (arighi) wrote :

debdiff for focal reported in attach (built and tested).

Revision history for this message
Andrea Righi (arighi) wrote :

debdiff for groovy in attach (all zfs autotests completed correctly).

tags: added: patch
Revision history for this message
Andrea Righi (arighi) wrote :

debdiff for hirsute in attach.

Revision history for this message
Andrea Righi (arighi) wrote :

New debdiff for focal on top of 0.8.3-1ubuntu12.7.

Revision history for this message
Andrea Righi (arighi) wrote :
Changed in zfs-linux (Ubuntu Hirsute):
assignee: nobody → Andrea Righi (arighi)
importance: Undecided → High
Changed in zfs-linux (Ubuntu Groovy):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Focal):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Bionic):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Xenial):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Hirsute):
status: New → In Progress
Changed in zfs-linux (Ubuntu Groovy):
status: New → In Progress
Changed in zfs-linux (Ubuntu Focal):
status: New → In Progress
Changed in zfs-linux (Ubuntu Bionic):
status: New → In Progress
Changed in zfs-linux (Ubuntu Xenial):
status: New → In Progress
Andrea Righi (arighi)
Changed in zfs-linux (Ubuntu Xenial):
status: In Progress → Invalid
Changed in zfs-linux (Ubuntu Bionic):
status: In Progress → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.4-1ubuntu16

---------------
zfs-linux (0.8.4-1ubuntu16) hirsute; urgency=medium

  * fix potential user-space double free when running "zfs mount -a"
    (LP: #1902588)
   - 4702-Revert-Let-zfs-mount-all-tolerate-in-progress-mounts.patch

 -- Andrea Righi <email address hidden> Fri, 27 Nov 2020 18:40:01 +0000

Changed in zfs-linux (Ubuntu Hirsute):
status: In Progress → Fix Released
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Lurf, or anyone else affected,

Accepted zfs-linux into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.4-1ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Groovy):
status: In Progress → Fix Committed
Changed in zfs-linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Lurf, or anyone else affected,

Accepted zfs-linux into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.3-1ubuntu12.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-failed-groovy verification-needed-focal
tags: added: verification-needed-groovy
removed: verification-failed-groovy
Andrea Righi (arighi)
tags: added: verification-done-focal
removed: verification-needed-focal
tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Andrea Righi (arighi) wrote :

Steps used to verify the fix (both in focal and groovy):

$ fallocate -l 8G /home/ubuntu/disk.img
$ sudo zpool create pool /home/ubuntu/disk.img
$ for i in {1..20}; do sudo zfs create pool/ds$i; done
$ sudo zfs unmount pool/ds20
$ sudo zfs mount -a & sudo zfs mount -a &
[1] 1964
[2] 1965
ubuntu@groovy:~$ filesystem 'pool/ds20' is already mounted
cannot mount 'pool/ds20': mountpoint or dataset is busy

[1]- Done sudo zfs mount -a
[2]+ Exit 1 sudo zfs mount -a

With the fix applied we should be able to see the error "mountpoint or dataset is busy", due to the concurrent "zfs mount -a" running. That means the old behavior has been restored, since it's not a problem anymore for systemd (and the applied workaround could cause the segfault).

Revision history for this message
Mathew Hodson (mhodson) wrote :

This commit that introduced the segfault was applied o the 0.8-release branch with https://github.com/openzfs/zfs/commit/95fcb04215015950b3388ba0a6edad8e1b463415 so Focal and Groovy are affected.

That change was never applied to the 0.6.5-release or 0.7-release branches, so Bionic and Xenial aren't affected.

no longer affects: zfs-linux (Ubuntu Xenial)
no longer affects: zfs-linux (Ubuntu Bionic)
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.4-1ubuntu11.1

---------------
zfs-linux (0.8.4-1ubuntu11.1) groovy; urgency=medium

  [ Didier Roche ]
  [ Jean-Baptiste Lallement ]
  * Generate clone uuid without dd which is flagged as having an executable
    stack. Thanks Usarin Heininga for the patch (LP: #1894329)

  [ Andrea Righi ]
  * fix potential user-space double free when running "zfs mount -a"
    (LP: #1902588)
    - 4702-Revert-Let-zfs-mount-all-tolerate-in-progress-mounts.patch

 -- Colin Ian King <email address hidden> Mon, 30 Nov 2020 19:00:00 +0000

Changed in zfs-linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.3-1ubuntu12.6

---------------
zfs-linux (0.8.3-1ubuntu12.6) focal; urgency=medium

  [ Didier Roche ]
  [ Jean-Baptiste Lallement ]
  * Generate clone uuid without dd which is flagged as having an executable
    stack. Thanks Usarin Heininga for the patch (LP: #1894329)

  [ Andrea Righi ]
  * fix potential user-space double free when running "zfs mount -a"
    (LP: #1902588)
    - 4702-Revert-Let-zfs-mount-all-tolerate-in-progress-mounts.patch

 -- Colin Ian King <email address hidden> Mon, 30 Nov 2020 19:00:00 +0000

Changed in zfs-linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.