ISST-LTE: not all mpath device files are symbol link

Bug #1486370 reported by bugproxy
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Fix Released
High
Mathieu Trudel-Lapierre

Bug Description

[Impact]
Minimal impact, this is limited to aesthetic issues; devices should if possible all be actual block device nodes rather than symlinks.

[Test case]
1) ls -l /dev/mapper
2) Reboot, try again.

Also:
while true; do kpartx -d -p -part /dev/dm-2;udevadm settle; kpartx -a -p -part /dev/dm-2; udevadm settle;done

Or:
while true; do multipath -F; multipath; done

[Regression potential]
Timing issues may cause delays in generating device nodes or symlinks due to the need for multipath-tools, device-mapper and udev to synchronize.

---

== Comment: #0 - Ping Tian Han <email address hidden> - 2015-07-21 23:38:48 ==
---Problem Description---
Some mpath devices in /dev/mapper are symbol links to /dev/dm-<n>, but there are exceptions:

% ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Jul 21 20:46 control
lrwxrwxrwx 1 root root 7 Jul 21 20:46 mpath0 -> ../dm-0
brw------- 1 root root 252, 4 Jul 21 20:46 mpath0-part1
brw------- 1 root root 252, 6 Jul 21 20:46 mpath0-part2
brw------- 1 root root 252, 7 Jul 21 20:46 mpath0-part3
lrwxrwxrwx 1 root root 7 Jul 21 20:46 mpath1 -> ../dm-1
brw-rw---- 1 root disk 252, 2 Jul 21 20:46 mpath2
lrwxrwxrwx 1 root root 7 Jul 21 20:46 mpath3 -> ../dm-3
lrwxrwxrwx 1 root root 7 Jul 21 20:46 mpath4 -> ../dm-5

mpath2 isn't a symbol link.

% sudo dmestup info mpath2
sudo: dmestup: command not found
% sudo dmsetup info mpath2
Name: mpath2
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 4
Major, minor: 252, 2
Number of targets: 1
UUID: mpath-36005076304ffc69f0000000000000469

% sudo dmsetup info /dev/dm-2
Name: mpath2
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 4
Major, minor: 252, 2
Number of targets: 1
UUID: mpath-36005076304ffc69f0000000000000469

Contact Information = Ping Tian <email address hidden>, Chanh H <email address hidden>

---uname output---
Linux onyxlp2 3.19.0-22-generic #22~14.04.1-Ubuntu SMP Wed Jun 17 10:03:39 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = na

---Steps to Reproduce---
 1. install ubuntu on onyxlp2, which has mpath device
2. ls /dev/mapper, see some mpath device file isn't symbol link

Userspace tool common name: na

The userspace tool has the following bit modes: 64-bit

Userspace rpm: na

Userspace tool obtained from project website: na

== Comment: #6 - Mauricio Faria De Oliveira <email address hidden> - 2015-08-18 12:33:17 ==
This shouldn't be a problem at all.
Either symlinks to the dm devices or block devices which refer to them directly (via major:minor number) will work.

It isn't ideal, though.

== Comment: #7 - Mauricio Faria De Oliveira <email address hidden> - 2015-08-18 12:35:34 ==
BTW, that expectation is per the SRU policy.

And the problem cause is likely a race involving udev rules (multipath & kpartx), and possibly multipathd.

Revision history for this message
bugproxy (bugproxy) wrote : dmesg

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-127890 severity-medium targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : multipath -ll

Default Comment by Bridge

affects: ubuntu → udev (Ubuntu)
Martin Pitt (pitti)
affects: udev (Ubuntu) → multipath-tools (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-09-24 14:08 EDT-------
Hello Canonical,

Are there any updates on this? Do you need any more details from us?

thanks!

Luciano Chavez (lnx1138)
Changed in multipath-tools (Ubuntu):
assignee: nobody → Taco Screen team (taco-screen-team)
bugproxy (bugproxy)
tags: added: targetmilestone-inin14044
removed: targetmilestone-inin---
Steve Langasek (vorlon)
Changed in multipath-tools (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I think this is http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=9a632fff65df3bf82a3ef70f62319a1c45f3c028. It's already targetted because it helps with another, similar issue (parted creating multiple device nodes for the same partition).

In progress.

Changed in multipath-tools (Ubuntu):
status: New → Fix Released
Changed in multipath-tools (Ubuntu Trusty):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
status: New → In Progress
importance: Undecided → High
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I just confirmed this is already applied in 0.5.0 (so in 15.10 and later).

Changed in multipath-tools (Ubuntu):
assignee: Mathieu Trudel-Lapierre (mathieu-tl) → nobody
Revision history for this message
bugproxy (bugproxy) wrote : dmesg

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : multipath -ll

Default Comment by Bridge

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Could you please verify that you're not seeing this issue on 16.04? It looks like the patch found should solve this, but it's hard to verify given that this is very dependent on the speed of the system on which multipath-tools run, as well as how fast udev can run, scan devices, etc.

description: updated
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-01-29 17:50 EDT-------
We are testing 16.04 soon and we'll let you know if the issue still exists.

Revision history for this message
Ming-Xun Zhong (zhongmx) wrote :

Hi,

I encounter this bug on Precise, and apply these two patch from upstream fix it.

http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=1b8082c8fc81374ba7b0204f6de8f71de5500192
http://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=9a632fff65df3bf82a3ef70f62319a1c45f3c028

In my environment it easily reproduce with

Case 1:
while true; do kpartx -d -p -part /dev/dm-2;udevadm settle; kpartx -a -p -part /dev/dm-2; udevadm settle;done

Case 2:
while true; do multipath -F; multipath; done

description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in multipath-tools (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (4.7 KiB)

This is working correctly w/ kpartx.
One instance it seems to be still generating symlinks/nodes intermixed is when using parted.
However, during reboot and post-boot kpartx invocation, it is pretty consistent w/ only nodes.
So, this seems correct with regards to the code in multipath-tools.
Marking verification-done.

After parted:

# ls -l /dev/mapper/mpath1*
brw-rw---- 1 root disk 252, 1 Feb 9 11:26 /dev/mapper/mpath1
brw-rw---- 1 root disk 252, 3 Feb 9 11:16 /dev/mapper/mpath1-part1
lrwxrwxrwx 1 root root 7 Feb 9 11:16 /dev/mapper/mpath1-part2 -> ../dm-6
lrwxrwxrwx 1 root root 7 Feb 9 11:17 /dev/mapper/mpath1-part3 -> ../dm-7
lrwxrwxrwx 1 root root 7 Feb 9 11:17 /dev/mapper/mpath1-part4 -> ../dm-8
lrwxrwxrwx 1 root root 7 Feb 9 11:26 /dev/mapper/mpath1-part5 -> ../dm-9
lrwxrwxrwx 1 root root 8 Feb 9 11:26 /dev/mapper/mpath1-part6 -> ../dm-10
lrwxrwxrwx 1 root root 8 Feb 9 11:26 /dev/mapper/mpath1-part7 -> ../dm-11
lrwxrwxrwx 1 root root 8 Feb 9 11:26 /dev/mapper/mpath1-part8 -> ../dm-12

After reboot:

# ls -l /dev/mapper/mpath1*
brw-rw---- 1 root disk 252, 1 Feb 9 11:28 /dev/mapper/mpath1
brw-rw---- 1 root disk 252, 3 Feb 9 11:28 /dev/mapper/mpath1-part1
brw-rw---- 1 root disk 252, 5 Feb 9 11:28 /dev/mapper/mpath1-part2
brw-rw---- 1 root disk 252, 7 Feb 9 11:28 /dev/mapper/mpath1-part3
brw-rw---- 1 root disk 252, 8 Feb 9 11:28 /dev/mapper/mpath1-part4
brw-rw---- 1 root disk 252, 9 Feb 9 11:28 /dev/mapper/mpath1-part5
brw-rw---- 1 root disk 252, 10 Feb 9 11:28 /dev/mapper/mpath1-part6
brw-rw---- 1 root disk 252, 11 Feb 9 11:28 /dev/mapper/mpath1-part7
brw-rw---- 1 root disk 252, 12 Feb 9 11:28 /dev/mapper/mpath1-part8

After a series of kpartx -d/-a -p -part:

# ls -l /dev/mapper/mpath1*
brw-rw---- 1 root disk 252, 1 Feb 9 11:34 /dev/mapper/mpath1
brw-rw---- 1 root disk 252, 3 Feb 9 11:34 /dev/mapper/mpath1-part1
brw-rw---- 1 root disk 252, 5 Feb 9 11:34 /dev/mapper/mpath1-part2
brw-rw---- 1 root disk 252, 7 Feb 9 11:34 /dev/mapper/mpath1-part3
brw-rw---- 1 root disk 252, 8 Feb 9 11:34 /dev/mapper/mpath1-part4
brw-rw---- 1 root disk 252, 9 Feb 9 11:34 /dev/mapper/mpath1-part5
brw-rw---- 1 root disk 252, 10 Feb 9 11:34 /dev/mapper/mpath1-part6
brw-rw---- 1 root disk 252, 11 Feb 9 11:34 /dev/mapper/mpath1-part7
brw-rw---- 1 root disk 252, 12 Feb 9 11:34 /dev/mapper/mpath1-part8
# kpartx -d -p -part /dev/mapper/mpath1
# ls -l /dev/mapper/mpath1*
brw-rw---- 1 root disk 252, 1 Feb 9 11:34 /dev/mapper/mpath1
# kpartx -a -p -part /dev/mapper/mpath1
# ls -l /dev/mapper/mpath1*
brw-rw---- 1 root disk 252, 1 Feb 9 11:34 /dev/mapper/mpath1
brw-rw---- 1 root disk 252, 3 Feb 9 11:34 /dev/mapper/mpath1-part1
brw-rw---- 1 root disk 252, 5 Feb 9 11:34 /dev/mapper/mpath1-part2
brw-rw---- 1 root disk 252, 7 Feb 9 11:34 /dev/mapper/mpath1-part3
brw-rw---- 1 root disk 252, 8 Feb 9 11:34 /dev/mapper/mpath1-part4
brw-rw---- 1 root disk 252, 9 Feb 9 11:34 /dev/mapper/mpath1-part5
brw-rw---- 1 root disk 252, 10 Feb 9 11:34 /dev/mapper/mpath1-part6
brw-rw---- 1 root disk 252, 11 Feb 9 11:34 /dev/mapper/mpath1-part7
brw-rw---- 1 root disk 252, 12 Feb 9 11:34 /dev/mapper/mpath...

Read more...

tags: added: verification-done
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello bugproxy, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.4.9-3ubuntu7.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

This was already verified successfully before; the additional update does not need reverification, only checking the regression in bug 1543430.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.9

---------------
multipath-tools (0.4.9-3ubuntu7.9) trusty; urgency=medium

  * debian/patches/kpartx-support-device-names-with-spaces.patch: fix loopback
    files unmapping. (LP: #1543430)

multipath-tools (0.4.9-3ubuntu7.8) trusty; urgency=medium

  * debian/patches/kpartx-support-device-names-with-spaces.patch: deal with
    spaces in device names in kpartx too (LP: #1432062)
  * debian/initramfs/local-premount: wait for udev to settle before mounting
    so the by-uuid/ symlinks have a chance to be updated by udev rules.
    (LP: #1503286)
  * Allow device detection all through the initramfs: run multipathd instead
    of only scanning once for devices, so those that come up slower can still
    be used as a root device (LP: #1526984):
    - debian/patches/0050-readonly-bindings_prefix.patch,
      debian/patches/0051-readonly-bindings_multipath.patch,
      debian/patches/0052-readonly-bindings_multipathd.patch,
      debian/patches/0053-readonly-bindings_multipathd_prod.patch: support -B
      to allow multipathd to handle cases where the bindings file is read-only.
    - debian/initramfs/hooks: install multipathd and required directories.
    - debian/initramfs/local-premount: also reload all maps to make sure
      they're ready before we mount.
    - debian/initramfs/local-top: run multipathd rather than a one-off call to
      multipath so that new paths can be correctly added as detected while
      we're still in the initramfs.
    - debian/initramfs/local-bottom: remember to stop multipathd.
    - debian/initramfs/local-bottom, debian/rules: install local-bottom for
      initramfs.
  * debian/patches/lp1496210_add_IBM_XIV_defaults.patch: add support (default
    config values) for the IBM 2810XIV storage system. (LP: #1496210)
  * debian/patches/0054-kpartx-update-option.patch: run kpartx -u rather than
    kpartx -a, so as to remove old partition entries if the partition table
    has changed. (LP: #1473903)
  * debian/patches/multipath_enable_sync_support_1b8082c8.patch,
    debian/patches/kpartx_rely_on_udev_dev_creation_9a632fff.patch: synchronize
    udev, device-mapper and multipath, and let udev deal with creating device
    nodes and symlinks. (LP: #1486370)
  * debian/initramfs/local-top: drop scsi_wait_scan stanza, that module is no
    longer available. (LP: #1538775)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 09 Feb 2016 16:03:10 -0500

Changed in multipath-tools (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Adam Conrad (adconrad) wrote : Update Released

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2016-09-02 10:45 EDT-------
Verified that it works on 16.04 system. This can be closed. Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.