Ubuntu
lxc package

lxc-start failed with lxc_cgroup - Invalid argument - write /sys/fs/cgroup/devices/lxc/<NAME>/devices.deny : Invalid argument with kernel 3.10

Bug #1196518 reported by Jean-Baptiste Lallement on 2013-07-01

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	lxc (Ubuntu)	Fix Released	High	Serge Hallyn

Bug Description

With kernel 3.10.0 lxc-create fails with the following error:
lxc-start 1372673552.852 NOTICE lxc_conf - 'saucy-i386-20130701-1012' is setup.
lxc-start 1372673552.852 DEBUG lxc_cgroup - cgroup_path_get: called for subsys devices name lxc/saucy-i386-20130701-1012

      lxc-start 1372673552.852 DEBUG lxc_cgroup - using cgroup mounted at '/sys/fs/cgroup/devices'
      lxc-start 1372673552.852 DEBUG lxc_cgroup - cgroup_path_get: returning /sys/fs/cgroup/devices/lxc/saucy-i386-20130701-1012 for subsystem devices.deny
      lxc-start 1372673552.852 ERROR lxc_cgroup - Invalid argument - write /sys/fs/cgroup/devices/lxc/saucy-i386-20130701-1012/devices.deny : Invalid argument
      lxc-start 1372673552.852 ERROR lxc_conf - Error setting devices.deny to a for lxc/saucy-i386-20130701-1012

Second run of lxc-start works fine.
Downgrading to 3.9.0-7 fixes the issue.

The rootfs of the container is a loop-mounted squashfs from a loop-mounted ISO of latest saucy desktop image.

pre-start, pre-mount, post-stop, lxc config and fstab attached

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: lxc 0.9.0-0ubuntu15
ProcVersionSignature: Ubuntu 3.10.0-1.8-generic 3.10.0-rc7
Uname: Linux 3.10.0-1-generic x86_64
ApportVersion: 2.10.2-0ubuntu3
Architecture: amd64
Date: Mon Jul 1 14:26:35 2013
MarkForUpload: True
SourcePackage: lxc
UpgradeStatus: Upgraded to saucy on 2012-01-31 (516 days ago)
defaults.conf:
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
---
ApportVersion: 2.10.2-0ubuntu3
Architecture: amd64
DistroRelease: Ubuntu 13.10
InstallationDate: Installed on 2012-10-25 (251 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
MarkForUpload: True
Package: lxc 0.9.0-0ubuntu16
PackageArchitecture: amd64
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-3.10.0-2-generic root=UUID=a25fbaa5-8321-48be-bd4f-d5027b955669 ro quiet splash swapaccount=1 vt.handoff=7
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=fr_FR.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 3.10.0-2.9-generic 3.10.0
Tags: saucy apparmor
Uname: Linux 3.10.0-2-generic x86_64
UpgradeStatus: Upgraded to saucy on 2012-12-02 (214 days ago)
UserGroups: adm autopilot cdrom dip kvm lpadmin plugdev sambashare sudo wireshark
defaults.conf:
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
---
ApportVersion: 2.10.2-0ubuntu3
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/hwC0D1', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D3p', '/dev/snd/pcmC0D6c', '/dev/snd/pcmC0D6p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 13.10
HibernationDevice: RESUME=UUID=9661ff9a-af2b-4768-a741-0a313db5b763
InstallationDate: Installed on 2012-10-25 (251 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
MachineType: ASUSTeK Computer Inc. U3SG
MarkForUpload: True
Package: lxc 0.9.0-0ubuntu16
PackageArchitecture: amd64
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-3.10.0-2-generic root=UUID=a25fbaa5-8321-48be-bd4f-d5027b955669 ro quiet splash swapaccount=1 vt.handoff=7
ProcEnviron:
TERM=screen
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=fr_FR.UTF-8
SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.10.0-2-generic root=UUID=a25fbaa5-8321-48be-bd4f-d5027b955669 ro quiet splash swapaccount=1 vt.handoff=7
ProcVersionSignature: Ubuntu 3.10.0-2.9-generic 3.10.0
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
Tags: saucy saucy apparmor
Uname: Linux 3.10.0-2-generic x86_64
UpgradeStatus: Upgraded to saucy on 2012-12-02 (214 days ago)
UserGroups: adm autopilot cdrom dip kvm lpadmin plugdev sambashare sudo wireshark
defaults.conf:
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
dmi.bios.date: 01/28/2008
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 305
dmi.board.asset.tag: ATN12345678901234567
dmi.board.name: U3SG
dmi.board.vendor: ASUSTeK Computer Inc.
dmi.board.version: 1.0
dmi.chassis.asset.tag: ATN12345678901234567
dmi.chassis.type: 10
dmi.chassis.vendor: ASUSTeK Computer Inc.
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr305:bd01/28/2008:svnASUSTeKComputerInc.:pnU3SG:pvr1.0:rvnASUSTeKComputerInc.:rnU3SG:rvr1.0:cvnASUSTeKComputerInc.:ct10:cvr1.0:
dmi.product.name: U3SG
dmi.product.version: 1.0
dmi.sys.vendor: ASUSTeK Computer Inc.

See original description

Tags:

Related branches

lp:ubuntu/saucy/lxc

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

Dependencies.txt Edit (4.3 KiB, text/plain; charset="utf-8")
KernLog.txt Edit (4.0 KiB, text/plain; charset="utf-8")
ProcEnviron.txt Edit (103 bytes, text/plain; charset="utf-8")
RelatedPackageVersions.txt Edit (208 bytes, text/plain; charset="utf-8")
lxc-net.default.txt Edit (884 bytes, text/plain; charset="utf-8")
lxc.default.txt Edit (430 bytes, text/plain; charset="utf-8")
lxcsyslog.txt Edit (2.0 KiB, text/plain; charset="utf-8")

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

config Edit (2.0 KiB, text/plain)

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

fstab Edit (291 bytes, text/plain)

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

post-stop.sh Edit (1.8 KiB, text/x-sh)

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

pre-mount.sh Edit (6.4 KiB, text/x-sh)

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

pre-start.sh Edit (3.4 KiB, text/x-sh)

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-01:

saucy-i386-20130701-1012.log.broken Edit (19.4 KiB, text/plain)

Serge Hallyn (serge-hallyn) on 2013-07-01

Changed in lxc (Ubuntu):
importance:	Undecided → High

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2013-07-03:

Marking invalid as I couldn't reproduce, and IIUC you will re-try with fresh containers.

If you run into this again, please re-mark it new and do a 'apport-collect 1196518'.

Changed in lxc (Ubuntu):
status:	New → Invalid

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04:

I'm reopening because I can reproduce it on different hardware, but it doesn't fail when running the same command twice and it seems a bit racy. I'm searching for a way to reliably reproduce that problem.

Changed in lxc (Ubuntu):
status:	Invalid → Confirmed
status:	Confirmed → Incomplete

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04:

#10

This is reproducible on several systems, but only the 1rst time I start a container when /sys/fs/cgroup/devices/lxc/<NAME>/ doesn't already exist.

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: Dependencies.txt

#11

Dependencies.txt Edit (4.2 KiB, text/plain)

apport information

tags:	added: apparmor apport-collected
description:	updated

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: KernLog.txt

#12

KernLog.txt Edit (68.2 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: RelatedPackageVersions.txt

#13

RelatedPackageVersions.txt Edit (185 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxc-net.default.txt

#14

lxc-net.default.txt Edit (884 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxc.default.txt

#15

lxc.default.txt Edit (430 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxcsyslog.txt

#16

lxcsyslog.txt Edit (67.1 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04:

#17

With otto from lp:otto I can reproduce the problem with the command:
$ sudo ./bin/otto -d create -u lp1196518-$RANDOM ~/iso/ubuntu/saucy-desktop-amd64.iso

The write(fd, value, strlen(value)) in do_cgroup_set in cgroup.c fails with "Invalid Argument"

If the command is executed a second time with the same container name (replace $RANDOM by the number generated previoulsly), it works fine.

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04:

#18

For the moment, the workaround is to remove lxc.cgroup.devices.deny from the config file, which is okay in our case since the machines are just test hosts, but is not in the general case.

tags:

added: bot-stop-nagging

Revision history for this message

Brad Figg (brad-figg) wrote on 2013-07-04: Missing required logs.

#19

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1196518

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: AlsaInfo.txt

#20

AlsaInfo.txt Edit (20.0 KiB, text/plain)

apport information

description:

updated

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: BootDmesg.txt

#21

BootDmesg.txt Edit (65.1 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: CRDA.txt

#22

CRDA.txt Edit (257 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: CurrentDmesg.txt

#23

CurrentDmesg.txt Edit (26.9 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: Dependencies.txt

#24

Dependencies.txt Edit (4.2 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: IwConfig.txt

#25

IwConfig.txt Edit (546 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: KernLog.txt

#26

KernLog.txt Edit (71.6 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: Lspci.txt

#27

Lspci.txt Edit (17.8 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: Lsusb.txt

#28

Lsusb.txt Edit (706 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: ProcCpuinfo.txt

#29

ProcCpuinfo.txt Edit (1.5 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: ProcInterrupts.txt

#30

ProcInterrupts.txt Edit (2.1 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: ProcModules.txt

#31

ProcModules.txt Edit (4.7 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: RelatedPackageVersions.txt

#32

RelatedPackageVersions.txt Edit (185 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: RfKill.txt

#33

RfKill.txt Edit (112 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: UdevDb.txt

#34

UdevDb.txt Edit (129.8 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: UdevLog.txt

#35

UdevLog.txt Edit (289.3 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: WifiSyslog.txt

#36

WifiSyslog.txt Edit (337.1 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxc-net.default.txt

#37

lxc-net.default.txt Edit (884 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxc.default.txt

#38

lxc.default.txt Edit (430 bytes, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04: lxcsyslog.txt

#39

lxcsyslog.txt Edit (74.5 KiB, text/plain)

apport information

Revision history for this message

Jean-Baptiste Lallement (jibel) wrote on 2013-07-04:

#40

Here is a minimal test case to reproduce the problem:

NAME=test-deny-${RANDOM}
echo "lxc.hook.mount = /usr/share/lxc/hooks/mountcgroups" > /tmp/${NAME}.lxc
lxc-create -t ubuntu -n $NAME -f /tmp/${NAME}.lxc -- -r saucy
lxc-start -n $NAME

From what I understand, setting devices.deny or .allow = a is forbidden in 3.10 when a cgroup has children. And the hook mountcgroup creates a child called NAME.real.
This line comes from a default configuration file in raring.

This is a change in behavior between 3.9 and 3.10 and a bit confusing. Feel free to close if you think it is not a bug.

Serge Hallyn (serge-hallyn) on 2013-07-05

Changed in lxc (Ubuntu):
status:	Incomplete → Triaged

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2013-07-05:

#41

I'll post a patch to be less stringent about that check.

Changed in linux (Ubuntu):
importance:	Undecided → Medium
status:	Incomplete → Triaged

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2013-07-05:

#42

Ah, wait - the simpler solution would be for the mountcgroup hook to set the device cgroup accesses (per the container config) before creating ${container}.real.

Serge Hallyn (serge-hallyn) on 2013-07-05

no longer affects:

linux (Ubuntu)

Revision history for this message

Serge Hallyn (serge-hallyn) wrote on 2013-07-05:

#43

0011-cgroup-hook-handle-stricter-kernel Edit (10.0 KiB, text/plain)

I'm testing this patch to solve the issue. So far it works on my simplest case. I will run the lxc testsuite to check for some regressions.

Serge Hallyn (serge-hallyn) on 2013-07-05

Changed in lxc (Ubuntu):
assignee:	nobody → Serge Hallyn (serge-hallyn)
status:	Triaged → In Progress

Revision history for this message

Launchpad Janitor (janitor) wrote on 2013-07-06:

#44

This bug was fixed in the package lxc - 0.9.0-0ubuntu17

---------------
lxc (0.9.0-0ubuntu17) saucy; urgency=low

  * 0011-cgroup-hook-handle-stricter-kernel: fix the mountcgroups hook in the
    face of new restrictions imposed by the kernel on devices cgroups.
    (LP: #1196518)
-- Serge Hallyn <email address hidden> Fri, 05 Jul 2013 20:44:57 +0200

Changed in lxc (Ubuntu):
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulxc package

lxc-start failed with lxc_cgroup - Invalid argument - write /sys/fs/cgroup/devices/lxc/<NAME>/devices.deny : Invalid argument with kernel 3.10

Bug Description

Related branches

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
lxc package