zpools fail to import after reboot on fresh install of eoan

Bug #1850130 reported by Miguel de Salas
66
This bug affects 10 people
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
High
Jean-Baptiste Lallement
Focal
Fix Released
High
Jean-Baptiste Lallement

Bug Description

Fresh installation of stock Ubuntu 19.10 Eoan with experimental root on ZFS.
System has existing zpools with data.

Installation is uneventful. First boot with no problems. Updates applied. No other changes from fresh installation. Reboot.

External pool 'tank' imports with no errors. Reboot.

External pool has failed to import on boot. In contrast bpool and rpool are ok. Manually re-import 'tank' with no issues. I can see both 'tank' and its path in /dev/disk/by-id/ in /etc/zfs/zpool.cache. Reboot.

'tank' has failed to import on boot. It is also missing from /etc/zfs/zpool.cache. Is it possible that the cache is being re-generated on reboot, and the newly imported pools are getting erased from it? I can re-import the pools again manually with no issues, but they don't persist between re-boots.

Installing normally on ext4 this is not an issue and data pools import automatically on boot with no further effort.

Miguel de Salas (mfdes)
affects: canonical-identity-provider → zfs-linux (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in zfs-linux (Ubuntu):
status: New → Confirmed
Revision history for this message
BertN45 (lammert-nijhof) wrote :

I have the same problem with my custom install, inherited and upgraded from my Ubuntu 19.04 system. After each boot I run a small script to export and import my three pools again. zsys is not installed anymore on this manual install from early 2019. The system worked without problems on Ubuntu 19.04.

Revision history for this message
MasterCATZ (mastercatz) wrote :

same worked perfectly 18.04 gone 19.10 this week and now i have to keep manually importing then restarting mysql service because it could not access its data ,,

Revision history for this message
MasterCATZ (mastercatz) wrote :

from what I can see is that its trying to do importing before the disks have been mapped because /etc/multipath.conf is being read just after zfs failed

Dec 11 23:04:57 aio systemd[1]: Started Mount ZFS filesystems.
Dec 11 23:04:57 aio systemd[1]: Starting Mount ZFS filesystems...
Dec 11 23:04:57 aio systemd[1]: Reached target ZFS pool import target.
Dec 11 23:04:57 aio systemd[1]: Failed to start Import ZFS pools by cache file.
Dec 11 23:04:57 aio systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Dec 11 23:04:57 aio systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Dec 11 23:04:57 aio zpool[3622]: a backup source.
Dec 11 23:04:57 aio zpool[3622]: Destroy and re-create the pool from
Dec 11 23:04:57 aio zpool[3622]: cannot import 'ZFS3WAY': no such pool or dataset
Dec 11 23:04:57 aio multipathd[3627]: path checkers start up
Dec 11 23:04:57 aio multipathd[3627]: read /etc/multipath.conf
Dec 11 23:04:57 aio multipathd[3627]: --------start up--------
Dec 11 23:04:57 aio kernel: rdac: device handler registered
Dec 11 23:04:57 aio kernel: emc: device handler registered
Dec 11 23:04:57 aio kernel: alua: device handler registered
Dec 11 23:04:57 aio systemd[1]: Starting Import ZFS pools by cache file...
Dec 11 23:04:57 aio systemd[1]: Started Install ZFS kernel module.
Dec 11 23:04:57 aio systemd[1]: Starting Device-Mapper Multipath Device Controller...
Dec 11 23:04:57 aio systemd[1]: Starting Install ZFS kernel module...
Dec 11 23:04:57 aio systemd[1]: Started udev Wait for Complete Device Initialization.
Dec 11 23:04:57 aio kernel: ZFS: Loaded module v0.8.2-1, ZFS pool version 5000, ZFS filesystem version 5
Dec 11 23:04:57 aio systemd[1]: Condition check resulted in Forward Password Requests to Plymouth Directory Watch being skipped.
Dec 11 23:04:57 aio systemd[1]: Condition check resulted in Show Plymouth Boot Screen being skipped.
Dec 11 23:04:57 aio systemd[1]: Starting LVM event activation on device 9:0...
Dec 11 23:04:57 aio systemd[1]: Created slice system-lvm2\x2dpvscan.slice.

Revision history for this message
Phil Merricks (seffyroff) wrote :

I'm seeing the same thing here. Eoan with external mpath array, I have to manually import the pools on that array whilst other pools more conventionally presented as physical disks import without issue.

Revision history for this message
Richard Laager (rlaager) wrote :

Try adding "After=multipathd.service" to zfs-import-cache.service and zfs-import-pool.service. If that fixes it, then we should probably add that upstream.

Revision history for this message
Gustaf (gustypants) wrote :

Tried adding "After=multipathd.service" to zfs-import-cache.service. Do not have a zfs-import-pool.service file. Still no mount after reboot.

Revision history for this message
Richard Laager (rlaager) wrote :

@gustypants: Sorry, the other one is scan, not pool. Are you using a multipath setup? Does the pool import fine if you do it manually once booted?

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

I have a local data mirror zpool, and need to import it manually like others. The system was installed with eoan and zfs on root, the data mirror was added afterwards.

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

And I'm on focal now.

Revision history for this message
Richard Laager (rlaager) wrote :

I think there are multiple issues here. If it's just multipath, that issue should be resolved by adding After=multipathd.service to zfs-import-{cache,scan}.service.

For other issues, I wonder if this is cache file related. I'd suggest checking that the cache file exists (I expect it would), and then looking at the cache file (e.g. strings /etc/zfs/zpool.cache | less). I suspect the issue is that the cache file has only the rpool. I'm not entirely sure why that is happening.

Revision history for this message
Phil Ritter (paritter) wrote :

Same problem, but pool did was not attached when system was initialized.

Fresh install of 19.10 using Ubuntu installer with ZFS root. Pool exported from another system (also running 19.10, but with a "normal" ext4 boot drive). Attached to new host, imported just fine. Reboot and pool is not imported.

Refreshed the zpool cache. Same behavior.
Added requirements for mpath per comment #6. Same behavior.

As a workaround, tried setting ZPOOL_IMPORT_ALL_VISIBLE='yes' in /etc/defaults/zfs. No change. Even if this had worked it is at best a workaround due to possibly importing unwanted pools.

Revision history for this message
Miguel de Salas (mfdes) wrote :

This bug also affects the daily builds of focal. Should I start a new bug report?

Changed in zfs-linux (Ubuntu Focal):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Focal):
assignee: nobody → Jean-Baptiste Lallement (jibel)
Revision history for this message
Timo Aaltonen (tjaalton) wrote :

so I've looked at the cache file, and indeed it looks like it only has entries for both bpool and rpool after booting up, and after 'zpool import data' it gets updated to include that as well

my guess is that it gets overwritten during bootup

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

or something, the pools don't seem to be ready for import?

maalis 11 12:31:59 bryant systemd[1]: Starting Import ZFS pools by cache file...
-- Subject: A start job for unit zfs-import-cache.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit zfs-import-cache.service has begun execution.
--
-- The job identifier is 15.
maalis 11 12:31:59 bryant zpool[2141]: no pools available to import
maalis 11 12:31:59 bryant systemd[1]: Started Import ZFS pools by cache file.
-- Subject: A start job for unit zfs-import-cache.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit zfs-import-cache.service has finished successfully.
--
-- The job identifier is 15.
maalis 11 12:31:59 bryant systemd[1]: Reached target ZFS pool import target.
-- Subject: A start job for unit zfs-import.target has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit zfs-import.target has finished successfully.
--
-- The job identifier is 14.
maalis 11 12:31:59 bryant systemd[1]: Mounting /home/tjaalton...
-- Subject: A start job for unit home-tjaalton.mount has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit home-tjaalton.mount has begun execution.
--
-- The job identifier is 61.
maalis 11 12:31:59 bryant systemd[1]: Mounting /root...
-- Subject: A start job for unit root.mount has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit root.mount has begun execution.
.
.
.

and so on, it mounts everything else during zfs-import but not my /data pool..

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Did you run "zpool export <poolname>" before rebooting? IIRC that removes it from the cache file.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

In other words, boot, import the pool, and reboot without exporting it. Sorry if you did that already, I just skimmed the bug and jumped to the last comment.

Revision history for this message
Timo Aaltonen (tjaalton) wrote :

I've never done that manually. Besides it's clear that the pools aren't imported by zfs-import-cache.service anyway.

Revision history for this message
Miguel de Salas (mfdes) wrote :

@ahasenack: The disappearing pools were not exported. To reproduce:

* Install with zfs on root option
* Import a third pool
* Reboot

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Good to know, thanks. Maybe something in the reboot process is exporting these extra pools, causing them to be removed from the cache.

Have you guys also tried editing /etc/default/zfs and listing your extra pool(s) there? Specifically:

# List of pools that SHOULD be imported at boot by the initramfs
# instead of trying to import all available pools. If this is set
# then ZFS_POOL_EXCEPTIONS is ignored.
# Only applicable for Debian GNU/Linux {dkms,initramfs}.
# This is a semi-colon separated list.
#ZFS_POOL_IMPORT="pool1;pool2"

There is also ZPOOL_IMPORT_ALL_VISIBLE='no' with a big disclaimer on top of it.

Not saying that the current behaviour isn't a bug: just investigating options.

Might need an initramfs update after changing anything in that file, tough.

Revision history for this message
Miguel de Salas (mfdes) wrote :

Hi @ahasenack, I just had a chance to try these.

Not great news.

Firstly with ZFS_POOL_IMPORT="bpool;rpool;extpool"

No effect at all. extpool is still not imported on boot.

Secondly with ZPOOL_IMPORT_ALL_VISIBLE='yes'

Same thing. The third pool does not re-import automatically after a re-boot.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.3-1ubuntu9

---------------
zfs-linux (0.8.3-1ubuntu9) focal; urgency=medium

  * debian/patches/4000-zsys-support.patch:
    - Fix import of external pools by preserving original cache file on zpool
      detection imports (LP: #1850130)

 -- Didier Roche <email address hidden> Fri, 27 Mar 2020 14:56:47 +0100

Changed in zfs-linux (Ubuntu Focal):
status: Confirmed → Fix Released
Revision history for this message
fardok@yandex.ru (fardok) wrote :
Revision history for this message
Gustaf (gustypants) wrote :

fardok:

Page not found
This question was removed from Super User for reasons of moderation. Please refer to the help center for possible explanations why a question might be removed.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub2 (Ubuntu Focal):
status: New → Confirmed
Changed in grub2 (Ubuntu):
status: New → Confirmed
Revision history for this message
fardok@yandex.ru (fardok) wrote :

Howtofixit

1) replace “Exec zfs mount -a” to “Exec zpool import -a” in systemd service file for zfs

2) uncomment “#zfs” to “zfs” in /lib/modules-load/somefile

3) make snapshot for pool with problems, attach snapshot as pool to required path

Now I'm not near my laptop so I don't remember exactly. I'll write to you tomorrow.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Removed spurious grub task.

no longer affects: grub2 (Ubuntu)
no longer affects: grub2 (Ubuntu Focal)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.