udev produces unpredictable net names when PCI device is a bridge

> udev can produce unpredictable network interface names by default when multiple devices map to the same slot due to an intermediate bridge.

so, if I understand it right, the MR won't actually fix this for anyone without additional per-system work, right? specifically, any system with this problem will need to also add the 'net.naming-scheme=latest' boot parameter (or set it via systemd-udevd env var).

If that's the case, then it seems like a much simpler manual workaround for this would be to just avoid slot naming for the problematic nics, for example by dropping a link file into /etc/systemd/network/ with content like:

[Match]
Driver="whatever driver the DGX nics use, or use some other specific match"

[Link]
NamePolicy=keep kernel database onboard path
AlternativeNamesPolicy=database onboard path
MACAddressPolicy=persistent

essentially, override the 99-default.link to remove 'slot' naming.

> While MAAS does take care to always restore the names used during commissioning (eth3 will always be the same NIC on every deploy), these names can change each time the system is commissioned.

if the only change needed is during maas comissioning, that seems like the perfect time to use a link file to override the specific problematic nics by whatever matching logic is best.

On Wed, Oct 13, 2021 at 05:10:05PM -0000, Dan Streetman wrote:
> > udev can produce unpredictable network interface names by default when
> multiple devices map to the same slot due to an intermediate bridge.
>
> so, if I understand it right, the MR won't actually fix this for anyone
> without additional per-system work, right? specifically, any system with
> this problem will need to also add the 'net.naming-scheme=latest' boot
> parameter (or set it via systemd-udevd env var).

Correct.

> If that's the case, then it seems like a much simpler manual workaround
> for this would be to just avoid slot naming for the problematic nics,
> for example by dropping a link file into /etc/systemd/network/ with
> content like:
>
> [Match]
> Driver="whatever driver the DGX nics use, or use some other specific match"
>
> [Link]
> NamePolicy=keep kernel database onboard path
> AlternativeNamesPolicy=database onboard path
> MACAddressPolicy=persistent
>
>
> essentially, override the 99-default.link to remove 'slot' naming.
>
> > While MAAS does take care to always restore the names used during
> commissioning (eth3 will always be the same NIC on every deploy), these
> names can change each time the system is commissioned.
>
> if the only change needed is during maas comissioning, that seems like
> the perfect time to use a link file to override the specific problematic
> nics by whatever matching logic is best.

Such a link file would presumably do the trick, but how would one
insert such a file during (and early enough) in the commissioning
process?

> how would one insert such a file during (and early enough) in the commissioning process?

well, i'll leave that one up to the maas team to answer properly, but i think there are ways to use custom commissioning scripts:
https://maas.io/docs/commissioning-scripts-reference

or even custom commissioning images:
https://maas.io/docs/image-builder

OK, I'll reach out to the MAAS team to see if they have a good recommendation. To be clear of my goals here - I'm confident that I could come up with a workaround that would work for my environment. My goal here is to come up with an easy-to-apply workaround for other users of such systems. I like the idea of the kernel command line argument because it is easy to apply and consistent across install types.

> I like the idea of the kernel command line argument because it is easy to apply and consistent across install types.

I agree the kernel boot param is absolutely easier, especially in the context of maas.

TL;DR from me is: I think it's at least worth looking at using a link file, or some other simpler method for this specific situation/hardware; if that's significantly more work and/or more complex, I'm +1 on this MR.

For more detail; my main objection to the MR is 2 things:

1) it would (in very limited situations) change the interface naming for anyone who is manually setting net-naming to 'latest' (which can be done either with a boot param, or editing the env var used by systemd-udevd). Why would anyone manually set that? I don't know for sure, since it is (and as you say, has long been) the default for ubuntu builds of systemd, but of course that's exactly the requirement for anyone to actually use the change introduced by the MR, so it's possible.

2) it doesn't actually fix this for anyone currently experiencing the problem; they would have to know about this change, and then take the extra step of manually setting the net-naming. So this really is a change that primarily benefits a very limited group of possibly affected users.

Note that I don't think #2 is necessarily a blocker; I've done exactly that before, e.g. bug 304393. I do think your backport of the v247 naming scheme is the best way to handle this *if* there is no other way to address it (that isn't significantly more painful).

did you find any alternate way to handle this? If not we should probably review/merge the 247 naming approach, if @slyon approves

Thanks for checking in @ddstreet. I haven't found an alternate way yet - but I also haven't exhausted alternatives. I've got an open request w/ the MAAS team for which I'm awaiting a response.

So with regards to MAAS, if you wish to apply the link file, I'd second adding a commissioning script that writes the file and executes early enough in the commissioning process (scripts execute in the order of their names, i.e 40-my-script will execute before 50-my-other-script). You can then also select whether to apply this to all machines or specific machines. Alternatively, you can add custom preseed configuration (https://maas.io/docs/snap/3.1/ui/how-to-customise-machines#heading--about-pre-seeding) to write the link file as well, this way you can explicitly control when the file is written and it'll write the file for all commissions. Though the kernel parameter option is probably the simplest one from a MAAS perspective.

Thanks for the investigation, I'm 100% +1 on comment #5.

The MR introduces some (low) risk of breaking existing use cases (#5.1), while not fixing the issue for anyone directly. Instead, it provides an (easier/additional?) workaround to the issue, which still needs to be applied to any individual machine hitting this issue.

So If there's any way to deploy the .link override configs, as suggested by @ddstreet, we should push for that instead of patching systemd. I would assume most deployment mechanisms (be it MAAS/cloud-init/...) should allow for placing additional udev config files. Worst case, it needs a reboot in between, I guess.

If anybody can prove that the .link override approach is not viable, we should ask the SRU team for their risk assesment of #5.1 and continue with the MR if they are +1 on it.

I was not able to make the .link approach work, even in an installed environment. I fought with it for a while during spurts of machine access. Without that, I was not able to move on to the next step of trying to insert it into a commissioning script.

Hello dann, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.19) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-aws-5.13/5.13.0-1031.35~20.04.1 (amd64)
munin/2.0.56-1ubuntu1 (arm64)
linux-gcp-5.8/5.8.0-1039.41 (amd64)
gvfs/1.44.1-1ubuntu1.1 (amd64)
linux-gcp-5.11/5.11.0-1030.34~20.04.3 (amd64)
linux-hwe-5.15/5.15.0-52.58~20.04.1 (amd64)
linux-oem-5.13/5.13.0-1029.36 (amd64)
systemd/245.4-4ubuntu3.19 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Verification:

Before:
ubuntu@akis:~$ ls /sys/class/net
enp134s0f0 enp230s0 enp93s0 ens103 eth0 eth7
enp134s0f1 enp6s0 ens102 ens107 eth2 lo

After upgrade to systemd 245.4-4ubuntu3.19 (w/ net.naming-scheme=v247):
ubuntu@akis:~$ ls /sys/class/net
enp134s0f0 enp184s0 enp225s0 enp53s0 enp6s0 enp93s0
enp134s0f1 enp189s0 enp230s0 enp58s0 enp88s0 lo

This bug was fixed in the package systemd - 245.4-4ubuntu3.19

---------------
systemd (245.4-4ubuntu3.19) focal; urgency=medium

  [ dann frazier ]
  * Add support for the v247 network naming scheme, but keep v245 as default
    (LP: #1945225)
    Author: dann frazier
    Files:
    - debian/patches/lp1945225/0001-udev-net_id-parse-_SUN-ACPI-index-as-a-signed-intege.patch
    - debian/patches/lp1945225/0002-udev-net_id-don-t-generate-slot-based-names-if-multi.patch
    - debian/patches/lp1945225/0003-net_id-fix-newly-added-naming-scheme-name.patch
    - debian/patches/lp1945225/0004-Add-remaining-supported-schemes-as-options-for-defau.patch
    - debian/rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f569231b5134a8e4901621ee5b2c33826184dae6

  [ Dimitri John Ledkov ]
  * test: fix test-execute autotest failure with kernel 5.15 (LP: #1975587)
    File: debian/patches/test-make-test-execute-pass-on-Linux-5.15.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7b3140ab5916269c020978ce678f06869a769f5c

 -- Dimitri John Ledkov <email address hidden> Thu, 08 Sep 2022 10:58:15 +0100

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.