ZFS revert from grub menu not working.

Bug #1894329 reported by Usarin Heininga
54
This bug affects 6 people
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
High
Jean-Baptiste Lallement
Focal
Fix Released
High
Jean-Baptiste Lallement
Groovy
Fix Released
High
Jean-Baptiste Lallement

Bug Description

[Impact]

 * Users can’t revert to previous snapshots when enabling the hw enablement stack kernel on focal or using any more recent version.
 * The option is available on grub and will let you with a broken system, partially cloned.

[Test Case]

 * Boot on a system, using ZFS and ZSys.
 * In grub, select "History" entry
 * Select one of the "Revert" option: the system should boot after being reverted with an older version.

[Where problems could occur]
 * The code is in the initramfs, where the generated id suffix for all our ZFS datasets was empty due to new coreutils/kernels.
 * We replace dd with another way (more robust and simple) for generating this ID.

---------------------

@coreutils maintainers, any idea why dd is being flagged as having an executable stack?

--------------------

When I try to revert to a previous state from the grub menu, the boot fails. The system drops me to a repair modus.

zfs-mount-generator fails with the message:
couldn't ensure boot: Mounted clone bootFS dataset created by initramfs doesn't have a valid _suffix (at least .*_<onechar>): \"rpool/ROOT/ubuntu_\"".

After a reboot I have an extra clone called "rpool/ROOT/ubuntu_", indeed without a suffix.
After a little investigation I found the problem in /usr/share/initramfs-tools/scripts/zfs at the end in function
uid()
{
   dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6
}, the dd command fails during boot with the message "process 'dd' started with executable stack.
After this an empty uid is returned which explains the dataset without a proper suffix.
Replacing the function with:
uid()
{
   grep -a -m10 -E "\*" /dev/urandom 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6
}

fixes the problem.

Ubuntu version is:
Description: Ubuntu Groovy Gorilla (development branch)
Release: 20.10

zfs-initramfs version is:
0.8.4-1ubuntu11

With regards,

Usarin Heininga

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: zfs-initramfs 0.8.4-1ubuntu11
ProcVersionSignature: Ubuntu 5.8.0-18.19-generic 5.8.4
Uname: Linux 5.8.0-18-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu45
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: KDE
Date: Fri Sep 4 20:23:44 2020
InstallationDate: Installed on 2020-09-02 (2 days ago)
InstallationMedia: Ubuntu 20.10 "Groovy Gorilla" - Alpha amd64 (20200831)
ProcEnviron:
 LANGUAGE=
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=nl_NL.UTF-8
 SHELL=/bin/bash
SourcePackage: zfs-linux
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Usarin Heininga (usarinheininga) wrote :
Revision history for this message
Usarin Heininga (usarinheininga) wrote :
Revision history for this message
Usarin Heininga (usarinheininga) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "zfs_patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Colin Ian King (colin-king) wrote :

This is a useful workaround, the fundamental issue is a coreutils issue with dd. I wonder if the coreutils maintainers can figure out why dd has an executable stack.

Changed in coreutils (Ubuntu):
importance: Undecided → High
description: updated
Changed in zfs-linux (Ubuntu):
status: New → Confirmed
status: Confirmed → Triaged
Changed in coreutils (Ubuntu):
status: New → Incomplete
Changed in zsys (Ubuntu):
importance: Undecided → High
Revision history for this message
Usarin Heininga (usarinheininga) wrote :

When I extract my initramfs I see that the size of the dd command is not equal to the size of the dd command provided by coreutils. It is the dd command provided by klibc-utils.
So you might want to tag them as well.

Regards.

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

I'm closing the zsys task because it is an issue with dd not zsys. For lack of fix in dd we will implement the workaround as a last resort.

Changed in zsys (Ubuntu):
status: New → Invalid
Changed in zfs-linux (Ubuntu):
importance: Undecided → High
status: Triaged → In Progress
assignee: nobody → Jean-Baptiste Lallement (jibel)
Changed in zfs-linux (Ubuntu Focal):
status: New → Triaged
Changed in zfs-linux (Ubuntu Groovy):
status: New → Triaged
Changed in zfs-linux (Ubuntu Focal):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Groovy):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Focal):
assignee: nobody → Jean-Baptiste Lallement (jibel)
Changed in zfs-linux (Ubuntu Groovy):
assignee: nobody → Jean-Baptiste Lallement (jibel)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in coreutils (Ubuntu Focal):
status: New → Confirmed
Changed in coreutils (Ubuntu Groovy):
status: New → Confirmed
Changed in zsys (Ubuntu Focal):
status: New → Confirmed
Changed in zsys (Ubuntu Groovy):
status: New → Confirmed
Changed in zsys (Ubuntu Focal):
status: Confirmed → Invalid
Changed in zsys (Ubuntu Groovy):
status: Confirmed → Invalid
Changed in coreutils (Ubuntu Focal):
status: Confirmed → Incomplete
Changed in coreutils (Ubuntu Groovy):
status: Confirmed → Incomplete
no longer affects: zsys (Ubuntu)
no longer affects: zsys (Ubuntu Groovy)
no longer affects: zsys (Ubuntu Focal)
Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :
Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

This is the patch from Usarin Heininga applied against Hirsute.

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

Patch for 20.10.

Revision history for this message
Jean-Baptiste Lallement (jibel) wrote :

Patch for Focal.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.4-1ubuntu14

---------------
zfs-linux (0.8.4-1ubuntu14) hirsute; urgency=medium

  [ Didier Roche ]
  [ Jean-Baptiste Lallement ]
  * Generate clone uuid without dd which is flagged as having an executable
    stack. Thanks Usarin Heininga for the patch (LP: #1894329)

 -- Jean-Baptiste Lallement <email address hidden> Mon, 09 Nov 2020 10:25:57 +0100

Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Usarin Heininga (usarinheininga) wrote :

I have tested the new package on Hirsute. Everything is working now.
Thanks.

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

Thanks for the confirmation :)

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

We will backport your patch to previous releases soon.

description: updated
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello Usarin, or anyone else affected,

Accepted zfs-linux into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.4-1ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Groovy):
status: Triaged → Fix Committed
Changed in zfs-linux (Ubuntu Focal):
status: Triaged → Fix Committed
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

Hello Usarin, or anyone else affected,

Accepted zfs-linux into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.3-1ubuntu12.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Arvydas Ramanauskas (sttagent) wrote :

This fixed the bug for me on 20.10 after I updated relevant zfs packages to 0.8.4-1ubuntu11.1 from groovy-proposed.

After the update I installed a random package. Tried to revert to the snapshot created by apt. The package was not on my system. It worked. Then I reverted back the revert. The package was back on my system. Everything seems to work now.

Thank you to everyone involved for the fix!

Revision history for this message
olivier (ofthesun) wrote :

I installed: zfs-initramfs 0.8.4-1ubuntu11.1 amd64 from groovy-proposed
olivier@bison:~$ sudo apt install zfs-initramfs

I created a new snapshot:
olivier@bison:~$ sudo zsysctl state save -s xfer01

I restarted the laptop and did a revert from xfer01, and the boot was successful

I confirm this fixes the issue for groovy

Revision history for this message
Nosehair (sbi-gaijin) wrote :

Yes!

Used the Software Updater to add the groovy-proposed repository.
Selected and installed the ZFS packages (from Software Updater).
Installed a small package (using "apt install") to force a grub update.
Rebooted.

"df -h" showed root mounted on the correct filesystem (instead of the previous, truncated "rpool/ROOT/ubuntu_").

Thank folks!!

Mathew Hodson (mhodson)
tags: added: verification-done-groovy verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.4-1ubuntu11.1

---------------
zfs-linux (0.8.4-1ubuntu11.1) groovy; urgency=medium

  [ Didier Roche ]
  [ Jean-Baptiste Lallement ]
  * Generate clone uuid without dd which is flagged as having an executable
    stack. Thanks Usarin Heininga for the patch (LP: #1894329)

  [ Andrea Righi ]
  * fix potential user-space double free when running "zfs mount -a"
    (LP: #1902588)
    - 4702-Revert-Let-zfs-mount-all-tolerate-in-progress-mounts.patch

 -- Colin Ian King <email address hidden> Mon, 30 Nov 2020 19:00:00 +0000

Changed in zfs-linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Colin Ian King (colin-king) wrote :

I tested this out with focal -proposed and don't see any issues. Looks OK to me.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.3-1ubuntu12.6

---------------
zfs-linux (0.8.3-1ubuntu12.6) focal; urgency=medium

  [ Didier Roche ]
  [ Jean-Baptiste Lallement ]
  * Generate clone uuid without dd which is flagged as having an executable
    stack. Thanks Usarin Heininga for the patch (LP: #1894329)

  [ Andrea Righi ]
  * fix potential user-space double free when running "zfs mount -a"
    (LP: #1902588)
    - 4702-Revert-Let-zfs-mount-all-tolerate-in-progress-mounts.patch

 -- Colin Ian King <email address hidden> Mon, 30 Nov 2020 19:00:00 +0000

Changed in zfs-linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Mathew Hodson (mhodson)
no longer affects: coreutils (Ubuntu Groovy)
no longer affects: coreutils (Ubuntu Focal)
no longer affects: coreutils (Ubuntu)
Revision history for this message
Marc Neumann (netwizard.x1) wrote :

Does this fix also work for Impish Indri (21.10) because I noticed the same issue but I was not able to fix the issue by manually changing the line within /usr/share/initramfs-tools/scripts/zfs.

Revision history for this message
Nicholas Harvey (mobile-harvey) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.