Ubuntu
cloud-init package

fsck.ext3: Device or resource busy while trying to open /dev/xvda2

Bug #898373 reported by Scott Moser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
High
Unassigned

Bug Description

I'm opening this against cloud-init, but I do not actually think that is valid.
In an ec2 instance for test of alpha-1 images [1], we saw errors on boot like:

  Loading, please wait...
  [1342606.840604] udevd[81]: starting version 175
  Begin: Loading essential drivers ... done.
  Begin: Running /scripts/init-premount ... done.
  Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
  Begin: Running /scripts/local-premount ... done.
  [1342607.148903] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
  Begin: Running /scripts/local-bottom ... done.
  done.
  Begin: Running /scripts/init-bottom ... done.
  lxcmount stop/pre-start, process 178
  fsck from util-linux 2.19.1
  fsck from util-linux 2.19.1
  fsck: fsck.swap: not found
  fsck: Error 2 while executing fsck.swap for /dev/xvda3
  fsck.ext3: Device or resource busy while trying to open /dev/xvda2
  Filesystem mounted or opened exclusively by another program?
  cloud-init start-local running: Wed, 30 Nov 2011 19:25:20 +0000. up 3.98 seconds
  no instance data found in start-local

The swap errors are due to bug 898365.
The cloud-config given to the instance included:
  mounts:
   - [ ephemeral0, /opt , auto, "defaults,noexec" ]
   - [ swap, null ]

And given that bug, the /etc/fstab during this reboot probably had:
  /dev/xvda2 /opt auto defaults,noexec,comment=cloudconfig 0 2
  /dev/xvda3 None auto defaults,nobootwait,comment=cloudconfig 0 2

The thing that confuses me, and the reason I do not think this is cloud-init's fault is that this happens prior to cloud-init start-local. So, yes, there is a broken fstab, but not for /dev/xvda2.

The errors listed above are on first reboot after cloud-init wrote those fstab entries, and then issued 'mount -a' from a upstart job (cloud-config.conf) launched at:
   start on (filesystem and started rsyslog)

--
[1] https://jenkins.qa.ubuntu.com/job/precise-server-ec2/ARCH=i386,REGION=us-west-1,STORAGE=instance-store,TEST=cloud-config,label=ubuntu-server-ec2-testing/1/artifact/None/i386/m1.small/instance-store/i-835a3dc4/uec2-20111130-1920-5d6ee3bc489541-terminated.console.txt

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: cloud-init 0.6.2-0ubuntu1 [modified: usr/share/pyshared/cloudinit/CloudConfig/cc_mounts.py]
ProcVersionSignature: User Name 3.2.0-2.5-virtual 3.2.0-rc3
Uname: Linux 3.2.0-2-virtual i686
ApportVersion: 1.90-0ubuntu1
Architecture: i386
Date: Wed Nov 30 21:44:25 2011
Ec2AMI: ami-a7cc07ce
Ec2AMIManifest: ubuntu-us-east-1/images-testing/ubuntu-precise-daily-i386-server-20111130.manifest.xml
Ec2AvailabilityZone: us-east-1c
Ec2InstanceType: m1.small
Ec2Kernel: aki-805ea7e9
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

Related bugs:
 * bug 928990: fsck / dirty filesystem on instance is death

See original description

Related branches

lp:~cloud-init-dev/cloud-init/trunk
Chad Smith (community): Needs Fixing
Hannes: Pending requested
Diff: 51583 lines (+11/-49380) (has conflicts)
424 files modified
.bzrignore (+0/-4)
ChangeLog (+0/-761)
HACKING.rst (+0/-48)
LICENSE (+0/-674)
MANIFEST.in (+0/-8)
Makefile (+0/-82)
README (+11/-0)
TODO.rst (+0/-43)
bin/cloud-init (+0/-673)
cloudinit/__init__.py (+0/-21)
cloudinit/cloud.py (+0/-109)
cloudinit/config/__init__.py (+0/-58)
cloudinit/config/cc_apt_configure.py (+0/-292)
cloudinit/config/cc_apt_pipelining.py (+0/-57)
cloudinit/config/cc_bootcmd.py (+0/-54)
cloudinit/config/cc_byobu.py (+0/-80)
cloudinit/config/cc_ca_certs.py (+0/-104)
cloudinit/config/cc_chef.py (+0/-342)
cloudinit/config/cc_debug.py (+0/-109)
cloudinit/config/cc_disable_ec2_metadata.py (+0/-36)
cloudinit/config/cc_emit_upstart.py (+0/-69)
cloudinit/config/cc_fan.py (+0/-101)
cloudinit/config/cc_final_message.py (+0/-73)
cloudinit/config/cc_foo.py (+0/-52)
cloudinit/config/cc_growpart.py (+0/-300)
cloudinit/config/cc_grub_dpkg.py (+0/-73)
cloudinit/config/cc_keys_to_console.py (+0/-62)
cloudinit/config/cc_landscape.py (+0/-99)
cloudinit/config/cc_locale.py (+0/-37)
cloudinit/config/cc_lxd.py (+0/-176)
cloudinit/config/cc_mcollective.py (+0/-88)
cloudinit/config/cc_migrator.py (+0/-85)
cloudinit/config/cc_mounts.py (+0/-405)
cloudinit/config/cc_package_update_upgrade_install.py (+0/-99)
cloudinit/config/cc_phone_home.py (+0/-122)
cloudinit/config/cc_power_state_change.py (+0/-223)
cloudinit/config/cc_puppet.py (+0/-118)
cloudinit/config/cc_resizefs.py (+0/-185)
cloudinit/config/cc_resolv_conf.py (+0/-116)
cloudinit/config/cc_rh_subscription.py (+0/-408)
cloudinit/config/cc_rightscale_userdata.py (+0/-102)
cloudinit/config/cc_rsyslog.py (+0/-366)
cloudinit/config/cc_runcmd.py (+0/-38)
cloudinit/config/cc_salt_minion.py (+0/-59)
cloudinit/config/cc_scripts_per_boot.py (+0/-41)
cloudinit/config/cc_scripts_per_instance.py (+0/-41)
cloudinit/config/cc_scripts_per_once.py (+0/-41)
cloudinit/config/cc_scripts_user.py (+0/-42)
cloudinit/config/cc_scripts_vendor.py (+0/-43)
cloudinit/config/cc_seed_random.py (+0/-94)
cloudinit/config/cc_set_hostname.py (+0/-37)
cloudinit/config/cc_set_passwords.py (+0/-167)
cloudinit/config/cc_snappy.py (+0/-304)
cloudinit/config/cc_ssh.py (+0/-142)
cloudinit/config/cc_ssh_authkey_fingerprints.py (+0/-105)
cloudinit/config/cc_ssh_import_id.py (+0/-99)
cloudinit/config/cc_timezone.py (+0/-39)
cloudinit/config/cc_ubuntu_init_switch.py (+0/-162)
cloudinit/config/cc_update_etc_hosts.py (+0/-60)
cloudinit/config/cc_update_hostname.py (+0/-43)
cloudinit/config/cc_users_groups.py (+0/-34)
cloudinit/config/cc_write_files.py (+0/-103)
cloudinit/config/cc_yum_add_repo.py (+0/-107)
cloudinit/cs_utils.py (+0/-105)
cloudinit/distros/__init__.py (+0/-950)
cloudinit/distros/arch.py (+0/-201)
cloudinit/distros/debian.py (+0/-227)
cloudinit/distros/fedora.py (+0/-31)
cloudinit/distros/freebsd.py (+0/-417)
cloudinit/distros/gentoo.py (+0/-160)
cloudinit/distros/net_util.py (+0/-182)
cloudinit/distros/parsers/__init__.py (+0/-28)
cloudinit/distros/parsers/hostname.py (+0/-88)
cloudinit/distros/parsers/hosts.py (+0/-92)
cloudinit/distros/parsers/resolv_conf.py (+0/-169)
cloudinit/distros/parsers/sys_conf.py (+0/-113)
cloudinit/distros/rhel.py (+0/-222)
cloudinit/distros/rhel_util.py (+0/-89)
cloudinit/distros/sles.py (+0/-179)
cloudinit/distros/ubuntu.py (+0/-31)
cloudinit/ec2_utils.py (+0/-201)
cloudinit/filters/__init__.py (+0/-21)
cloudinit/filters/launch_index.py (+0/-75)
cloudinit/handlers/__init__.py (+0/-274)
cloudinit/handlers/boot_hook.py (+0/-70)
cloudinit/handlers/cloud_config.py (+0/-163)
cloudinit/handlers/shell_script.py (+0/-55)
cloudinit/handlers/upstart_job.py (+0/-119)
cloudinit/helpers.py (+0/-458)
cloudinit/importer.py (+0/-58)
cloudinit/log.py (+0/-155)
cloudinit/mergers/__init__.py (+0/-166)
cloudinit/mergers/m_dict.py (+0/-88)
cloudinit/mergers/m_list.py (+0/-89)
cloudinit/mergers/m_str.py (+0/-46)
cloudinit/net/__init__.py (+0/-771)
cloudinit/net/network_state.py (+0/-447)
cloudinit/net/udev.py (+0/-54)
cloudinit/netinfo.py (+0/-249)
cloudinit/patcher.py (+0/-58)
cloudinit/registry.py (+0/-37)
cloudinit/reporting/__init__.py (+0/-42)
cloudinit/reporting/events.py (+0/-248)
cloudinit/reporting/handlers.py (+0/-91)
cloudinit/safeyaml.py (+0/-32)
cloudinit/settings.py (+0/-68)
cloudinit/signal_handler.py (+0/-71)
cloudinit/sources/DataSourceAltCloud.py (+0/-293)
cloudinit/sources/DataSourceAzure.py (+0/-651)
cloudinit/sources/DataSourceBigstep.py (+0/-57)
cloudinit/sources/DataSourceCloudSigma.py (+0/-141)
cloudinit/sources/DataSourceCloudStack.py (+0/-253)
cloudinit/sources/DataSourceConfigDrive.py (+0/-424)
cloudinit/sources/DataSourceDigitalOcean.py (+0/-110)
cloudinit/sources/DataSourceEc2.py (+0/-211)
cloudinit/sources/DataSourceGCE.py (+0/-167)
cloudinit/sources/DataSourceMAAS.py (+0/-353)
cloudinit/sources/DataSourceNoCloud.py (+0/-336)
cloudinit/sources/DataSourceNone.py (+0/-57)
cloudinit/sources/DataSourceOVF.py (+0/-429)
cloudinit/sources/DataSourceOpenNebula.py (+0/-455)
cloudinit/sources/DataSourceOpenStack.py (+0/-173)
cloudinit/sources/DataSourceSmartOS.py (+0/-536)
cloudinit/sources/__init__.py (+0/-337)
cloudinit/sources/helpers/__init__.py (+0/-13)
cloudinit/sources/helpers/azure.py (+0/-279)
cloudinit/sources/helpers/openstack.py (+0/-499)
cloudinit/sources/helpers/vmware/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/__init__.py (+0/-13)
cloudinit/sources/helpers/vmware/imc/boot_proto.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config.py (+0/-95)
cloudinit/sources/helpers/vmware/imc/config_file.py (+0/-129)
cloudinit/sources/helpers/vmware/imc/config_namespace.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/config_nic.py (+0/-247)
cloudinit/sources/helpers/vmware/imc/config_source.py (+0/-23)
cloudinit/sources/helpers/vmware/imc/guestcust_error.py (+0/-24)
cloudinit/sources/helpers/vmware/imc/guestcust_event.py (+0/-27)
cloudinit/sources/helpers/vmware/imc/guestcust_state.py (+0/-25)
cloudinit/sources/helpers/vmware/imc/guestcust_util.py (+0/-128)
cloudinit/sources/helpers/vmware/imc/ipv4_mode.py (+0/-45)
cloudinit/sources/helpers/vmware/imc/nic.py (+0/-147)
cloudinit/sources/helpers/vmware/imc/nic_base.py (+0/-154)
cloudinit/ssh_util.py (+0/-314)
cloudinit/stages.py (+0/-844)
cloudinit/templater.py (+0/-149)
cloudinit/type_utils.py (+0/-52)
cloudinit/url_helper.py (+0/-509)
cloudinit/user_data.py (+0/-352)
cloudinit/util.py (+0/-2226)
cloudinit/version.py (+0/-27)
config/cloud.cfg (+0/-115)
config/cloud.cfg-freebsd (+0/-88)
config/cloud.cfg.d/05_logging.cfg (+0/-66)
config/cloud.cfg.d/README (+0/-3)
doc/README (+0/-4)
doc/examples/cloud-config-add-apt-repos.txt (+0/-34)
doc/examples/cloud-config-archive-launch-index.txt (+0/-30)
doc/examples/cloud-config-archive.txt (+0/-16)
doc/examples/cloud-config-boot-cmds.txt (+0/-15)
doc/examples/cloud-config-ca-certs.txt (+0/-31)
doc/examples/cloud-config-chef-oneiric.txt (+0/-90)
doc/examples/cloud-config-chef.txt (+0/-95)
doc/examples/cloud-config-datasources.txt (+0/-73)
doc/examples/cloud-config-disk-setup.txt (+0/-251)
doc/examples/cloud-config-final-message.txt (+0/-7)
doc/examples/cloud-config-gluster.txt (+0/-18)
doc/examples/cloud-config-growpart.txt (+0/-31)
doc/examples/cloud-config-install-packages.txt (+0/-15)
doc/examples/cloud-config-landscape.txt (+0/-22)
doc/examples/cloud-config-launch-index.txt (+0/-23)
doc/examples/cloud-config-lxd.txt (+0/-55)
doc/examples/cloud-config-mcollective.txt (+0/-49)
doc/examples/cloud-config-mount-points.txt (+0/-46)
doc/examples/cloud-config-phone-home.txt (+0/-14)
doc/examples/cloud-config-power-state.txt (+0/-40)
doc/examples/cloud-config-puppet.txt (+0/-51)
doc/examples/cloud-config-reporting.txt (+0/-17)
doc/examples/cloud-config-resolv-conf.txt (+0/-20)
doc/examples/cloud-config-rh_subscription.txt (+0/-49)
doc/examples/cloud-config-rsyslog.txt (+0/-46)
doc/examples/cloud-config-run-cmds.txt (+0/-22)
doc/examples/cloud-config-salt-minion.txt (+0/-53)
doc/examples/cloud-config-seed-random.txt (+0/-32)
doc/examples/cloud-config-ssh-keys.txt (+0/-46)
doc/examples/cloud-config-update-apt.txt (+0/-7)
doc/examples/cloud-config-update-packages.txt (+0/-8)
doc/examples/cloud-config-user-groups.txt (+0/-109)
doc/examples/cloud-config-vendor-data.txt (+0/-16)
doc/examples/cloud-config-write-files.txt (+0/-33)
doc/examples/cloud-config-yum-repo.txt (+0/-20)
doc/examples/cloud-config.txt (+0/-645)
doc/examples/include-once.txt (+0/-7)
doc/examples/include.txt (+0/-5)
doc/examples/kernel-cmdline.txt (+0/-18)
doc/examples/part-handler-v2.txt (+0/-38)
doc/examples/part-handler.txt (+0/-23)
doc/examples/plain-ignored.txt (+0/-2)
doc/examples/seed/README (+0/-22)
doc/examples/seed/meta-data (+0/-30)
doc/examples/seed/user-data (+0/-3)
doc/examples/upstart-cloud-config.txt (+0/-12)
doc/examples/upstart-rclocal.txt (+0/-12)
doc/examples/user-script.txt (+0/-8)
doc/merging.rst (+0/-188)
doc/rtd/conf.py (+0/-77)
doc/rtd/index.rst (+0/-31)
doc/rtd/static/logo.svg (+0/-89)
doc/rtd/topics/availability.rst (+0/-20)
doc/rtd/topics/capabilities.rst (+0/-24)
doc/rtd/topics/datasources.rst (+0/-200)
doc/rtd/topics/dir_layout.rst (+0/-81)
doc/rtd/topics/examples.rst (+0/-133)
doc/rtd/topics/format.rst (+0/-159)
doc/rtd/topics/hacking.rst (+0/-1)
doc/rtd/topics/merging.rst (+0/-5)
doc/rtd/topics/modules.rst (+0/-342)
doc/rtd/topics/moreinfo.rst (+0/-12)
doc/sources/altcloud/README.rst (+0/-87)
doc/sources/azure/README.rst (+0/-134)
doc/sources/cloudsigma/README.rst (+0/-38)
doc/sources/cloudstack/README.rst (+0/-29)
doc/sources/configdrive/README.rst (+0/-123)
doc/sources/digitalocean/README.rst (+0/-21)
doc/sources/kernel-cmdline.txt (+0/-48)
doc/sources/nocloud/README.rst (+0/-71)
doc/sources/opennebula/README.rst (+0/-142)
doc/sources/openstack/README.rst (+0/-24)
doc/sources/ovf/README (+0/-83)
doc/sources/ovf/example/ovf-env.xml (+0/-46)
doc/sources/ovf/example/ubuntu-server.ovf (+0/-130)
doc/sources/ovf/make-iso (+0/-156)
doc/sources/ovf/ovf-env.xml.tmpl (+0/-28)
doc/sources/ovf/ovfdemo.pem (+0/-27)
doc/sources/ovf/user-data (+0/-7)
doc/sources/smartos/README.rst (+0/-149)
doc/status.txt (+0/-53)
doc/userdata.txt (+0/-79)
doc/var-lib-cloud.txt (+0/-63)
doc/vendordata.txt (+0/-53)
packages/bddeb (+0/-260)
packages/brpm (+0/-277)
packages/debian/changelog.in (+0/-6)
packages/debian/cloud-init.postinst (+0/-16)
packages/debian/cloud-init.preinst (+0/-20)
packages/debian/compat (+0/-1)
packages/debian/control.in (+0/-29)
packages/debian/copyright (+0/-29)
packages/debian/dirs (+0/-6)
packages/debian/rules.in (+0/-23)
packages/debian/watch (+0/-2)
packages/redhat/cloud-init.spec.in (+0/-200)
packages/suse/cloud-init.spec.in (+0/-163)
requirements.txt (+0/-36)
setup.py (+0/-215)
systemd/cloud-config.service (+0/-16)
systemd/cloud-config.target (+0/-11)
systemd/cloud-final.service (+0/-17)
systemd/cloud-init-generator (+0/-133)
systemd/cloud-init-local.service (+0/-22)
systemd/cloud-init.service (+0/-18)
systemd/cloud-init.target (+0/-6)
sysvinit/debian/cloud-config (+0/-64)
sysvinit/debian/cloud-final (+0/-66)
sysvinit/debian/cloud-init (+0/-64)
sysvinit/debian/cloud-init-local (+0/-63)
sysvinit/freebsd/cloudconfig (+0/-35)
sysvinit/freebsd/cloudfinal (+0/-35)
sysvinit/freebsd/cloudinit (+0/-35)
sysvinit/freebsd/cloudinitlocal (+0/-35)
sysvinit/gentoo/cloud-config (+0/-13)
sysvinit/gentoo/cloud-final (+0/-11)
sysvinit/gentoo/cloud-init (+0/-12)
sysvinit/gentoo/cloud-init-local (+0/-13)
sysvinit/redhat/cloud-config (+0/-121)
sysvinit/redhat/cloud-final (+0/-121)
sysvinit/redhat/cloud-init (+0/-121)
sysvinit/redhat/cloud-init-local (+0/-124)
templates/chef_client.rb.tmpl (+0/-58)
templates/hosts.debian.tmpl (+0/-26)
templates/hosts.freebsd.tmpl (+0/-24)
templates/hosts.redhat.tmpl (+0/-24)
templates/hosts.suse.tmpl (+0/-26)
templates/resolv.conf.tmpl (+0/-30)
templates/sources.list.debian.tmpl (+0/-32)
templates/sources.list.ubuntu.tmpl (+0/-58)
test-requirements.txt (+0/-17)
tests/configs/sample1.yaml (+0/-52)
tests/data/filter_cloud_multipart.yaml (+0/-30)
tests/data/filter_cloud_multipart_1.email (+0/-11)
tests/data/filter_cloud_multipart_2.email (+0/-39)
tests/data/filter_cloud_multipart_header.email (+0/-11)
tests/data/merge_sources/expected1.yaml (+0/-1)
tests/data/merge_sources/expected10.yaml (+0/-7)
tests/data/merge_sources/expected11.yaml (+0/-5)
tests/data/merge_sources/expected12.yaml (+0/-5)
tests/data/merge_sources/expected2.yaml (+0/-3)
tests/data/merge_sources/expected3.yaml (+0/-1)
tests/data/merge_sources/expected4.yaml (+0/-2)
tests/data/merge_sources/expected5.yaml (+0/-7)
tests/data/merge_sources/expected6.yaml (+0/-9)
tests/data/merge_sources/expected7.yaml (+0/-38)
tests/data/merge_sources/expected8.yaml (+0/-7)
tests/data/merge_sources/expected9.yaml (+0/-5)
tests/data/merge_sources/source1-1.yaml (+0/-3)
tests/data/merge_sources/source1-2.yaml (+0/-5)
tests/data/merge_sources/source10-1.yaml (+0/-6)
tests/data/merge_sources/source10-2.yaml (+0/-6)
tests/data/merge_sources/source11-1.yaml (+0/-5)
tests/data/merge_sources/source11-2.yaml (+0/-3)
tests/data/merge_sources/source11-3.yaml (+0/-3)
tests/data/merge_sources/source12-1.yaml (+0/-8)
tests/data/merge_sources/source12-2.yaml (+0/-5)
tests/data/merge_sources/source2-1.yaml (+0/-6)
tests/data/merge_sources/source2-2.yaml (+0/-5)
tests/data/merge_sources/source3-1.yaml (+0/-4)
tests/data/merge_sources/source3-2.yaml (+0/-4)
tests/data/merge_sources/source4-1.yaml (+0/-3)
tests/data/merge_sources/source4-2.yaml (+0/-6)
tests/data/merge_sources/source5-1.yaml (+0/-6)
tests/data/merge_sources/source5-2.yaml (+0/-8)
tests/data/merge_sources/source6-1.yaml (+0/-5)
tests/data/merge_sources/source6-2.yaml (+0/-8)
tests/data/merge_sources/source7-1.yaml (+0/-27)
tests/data/merge_sources/source7-2.yaml (+0/-17)
tests/data/merge_sources/source8-1.yaml (+0/-7)
tests/data/merge_sources/source8-2.yaml (+0/-6)
tests/data/merge_sources/source9-1.yaml (+0/-5)
tests/data/merge_sources/source9-2.yaml (+0/-6)
tests/data/mountinfo_precise_ext4.txt (+0/-24)
tests/data/mountinfo_raring_btrfs.txt (+0/-13)
tests/data/roots/simple_ubuntu/etc/networks/interfaces (+0/-3)
tests/data/user_data.1.txt (+0/-15)
tests/data/vmware/cust-dhcp-2nic.cfg (+0/-34)
tests/data/vmware/cust-static-2nic.cfg (+0/-39)
tests/unittests/helpers.py (+0/-346)
tests/unittests/test__init__.py (+0/-220)
tests/unittests/test_builtin_handlers.py (+0/-73)
tests/unittests/test_cli.py (+0/-54)
tests/unittests/test_cs_util.py (+0/-84)
tests/unittests/test_data.py (+0/-559)
tests/unittests/test_datasource/test_altcloud.py (+0/-452)
tests/unittests/test_datasource/test_azure.py (+0/-646)
tests/unittests/test_datasource/test_azure_helper.py (+0/-421)
tests/unittests/test_datasource/test_cloudsigma.py (+0/-99)
tests/unittests/test_datasource/test_cloudstack.py (+0/-87)
tests/unittests/test_datasource/test_configdrive.py (+0/-407)
tests/unittests/test_datasource/test_digitalocean.py (+0/-127)
tests/unittests/test_datasource/test_gce.py (+0/-166)
tests/unittests/test_datasource/test_maas.py (+0/-163)
tests/unittests/test_datasource/test_nocloud.py (+0/-187)
tests/unittests/test_datasource/test_opennebula.py (+0/-300)
tests/unittests/test_datasource/test_openstack.py (+0/-347)
tests/unittests/test_datasource/test_smartos.py (+0/-585)
tests/unittests/test_distros/test_generic.py (+0/-233)
tests/unittests/test_distros/test_hostname.py (+0/-38)
tests/unittests/test_distros/test_hosts.py (+0/-41)
tests/unittests/test_distros/test_netconfig.py (+0/-321)
tests/unittests/test_distros/test_resolv.py (+0/-67)
tests/unittests/test_distros/test_sysconfig.py (+0/-82)
tests/unittests/test_distros/test_user_data_normalize.py (+0/-297)
tests/unittests/test_ec2_util.py (+0/-139)
tests/unittests/test_filters/test_launch_index.py (+0/-132)
tests/unittests/test_handler/test_handler_apt_configure.py (+0/-109)
tests/unittests/test_handler/test_handler_ca_certs.py (+0/-271)
tests/unittests/test_handler/test_handler_chef.py (+0/-192)
tests/unittests/test_handler/test_handler_debug.py (+0/-81)
tests/unittests/test_handler/test_handler_disk_setup.py (+0/-30)
tests/unittests/test_handler/test_handler_growpart.py (+0/-220)
tests/unittests/test_handler/test_handler_locale.py (+0/-67)
tests/unittests/test_handler/test_handler_lxd.py (+0/-134)
tests/unittests/test_handler/test_handler_mounts.py (+0/-133)
tests/unittests/test_handler/test_handler_power_state.py (+0/-127)
tests/unittests/test_handler/test_handler_rsyslog.py (+0/-174)
tests/unittests/test_handler/test_handler_seed_random.py (+0/-227)
tests/unittests/test_handler/test_handler_set_hostname.py (+0/-72)
tests/unittests/test_handler/test_handler_snappy.py (+0/-306)
tests/unittests/test_handler/test_handler_timezone.py (+0/-76)
tests/unittests/test_handler/test_handler_write_files.py (+0/-112)
tests/unittests/test_handler/test_handler_yum_add_repo.py (+0/-68)
tests/unittests/test_helpers.py (+0/-33)
tests/unittests/test_merging.py (+0/-257)
tests/unittests/test_net.py (+0/-130)
tests/unittests/test_pathprefix2dict.py (+0/-44)
tests/unittests/test_registry.py (+0/-28)
tests/unittests/test_reporting.py (+0/-369)
tests/unittests/test_rh_subscription.py (+0/-214)
tests/unittests/test_runs/test_merge_run.py (+0/-54)
tests/unittests/test_runs/test_simple_run.py (+0/-81)
tests/unittests/test_sshutil.py (+0/-171)
tests/unittests/test_templating.py (+0/-119)
tests/unittests/test_util.py (+0/-489)
tests/unittests/test_vmware_config_file.py (+0/-103)
tools/21-cloudinit.conf (+0/-6)
tools/Z99-cloud-locale-test.sh (+0/-98)
tools/build-on-freebsd (+0/-66)
tools/ccfg-merge-debug (+0/-90)
tools/cloud-init-per (+0/-60)
tools/hacking.py (+0/-170)
tools/make-dist-tarball (+0/-21)
tools/make-mime.py (+0/-60)
tools/make-tarball (+0/-39)
tools/mock-meta.py (+0/-454)
tools/motd-hook (+0/-35)
tools/read-dependencies (+0/-29)
tools/read-version (+0/-26)
tools/run-pep8 (+0/-22)
tools/run-pyflakes (+0/-18)
tools/run-pyflakes3 (+0/-2)
tools/tox-venv (+0/-42)
tools/uncloud-init (+0/-141)
tools/validate-yaml.py (+0/-25)
tools/write-ssh-key-fingerprints (+0/-38)
tox.ini (+0/-35)
udev/66-azure-ephemeral.rules (+0/-18)
udev/79-cloud-init-net-wait.rules (+0/-10)
udev/cloud-init-wait (+0/-70)
upstart/cloud-config.conf (+0/-9)
upstart/cloud-final.conf (+0/-10)
upstart/cloud-init-blocknet.conf (+0/-83)
upstart/cloud-init-container.conf (+0/-57)
upstart/cloud-init-local.conf (+0/-16)
upstart/cloud-init-nonet.conf (+0/-66)
upstart/cloud-init.conf (+0/-9)
upstart/cloud-log-shutdown.conf (+0/-19)
JJ Asghar: Pending requested
Diff: 0 lines
Scott Moser: Pending requested
Diff: 3061 lines (+2239/-175)
39 files modified
ChangeLog (+39/-9)
cloud-init.py (+31/-1)
cloudinit/CloudConfig/__init__.py (+1/-1)
cloudinit/CloudConfig/cc_apt_pipelining.py (+53/-0)
cloudinit/CloudConfig/cc_ca_certs.py (+3/-1)
cloudinit/CloudConfig/cc_chef.py (+5/-5)
cloudinit/CloudConfig/cc_landscape.py (+5/-0)
cloudinit/CloudConfig/cc_resizefs.py (+31/-12)
cloudinit/CloudConfig/cc_salt_minion.py (+2/-1)
cloudinit/CloudConfig/cc_update_etc_hosts.py (+1/-1)
cloudinit/DataSource.py (+4/-1)
cloudinit/DataSourceCloudStack.py (+92/-0)
cloudinit/DataSourceConfigDrive.py (+231/-0)
cloudinit/DataSourceEc2.py (+2/-84)
cloudinit/DataSourceMAAS.py (+345/-0)
cloudinit/DataSourceNoCloud.py (+75/-2)
cloudinit/DataSourceOVF.py (+1/-1)
cloudinit/SshUtil.py (+2/-0)
cloudinit/UserDataHandler.py (+2/-2)
cloudinit/__init__.py (+44/-9)
cloudinit/netinfo.py (+5/-5)
cloudinit/util.py (+253/-30)
config/cloud.cfg (+3/-1)
debian.trunk/control (+1/-0)
doc/configdrive/README (+118/-0)
doc/examples/cloud-config-chef-oneiric.txt (+90/-0)
doc/examples/cloud-config-chef.txt (+48/-6)
doc/examples/cloud-config-datasources.txt (+18/-0)
doc/examples/cloud-config.txt (+11/-0)
doc/kernel-cmdline.txt (+48/-0)
doc/nocloud/README (+55/-0)
setup.py (+1/-0)
tests/unittests/test__init__.py (+242/-0)
tests/unittests/test_datasource/test_maas.py (+153/-0)
tests/unittests/test_handler/test_handler_ca_certs.py (+4/-0)
tests/unittests/test_userdata.py (+107/-0)
tests/unittests/test_util.py (+18/-2)
tools/Z99-cloud-locale-test.sh (+92/-0)
tools/run-pylint (+3/-1)
lp:ubuntu/precise/cloud-init
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

Just a note, this did not occur on all instances with this user-data.

Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Scott Moser (smoser) wrote :

Seen in jenkins daily runs in build at [1]. Specifically, see console log of instance at [2]

--
[1] https://jenkins.qa.ubuntu.com/view/Precise%20ISO%20Testing%20Dashboard/view/Daily/job/precise-server-ec2-daily/ARCH=i386,REGION=eu-west-1,STORAGE=instance-store,label=ubuntu-server-ec2-testing/45/
[2] https://jenkins.qa.ubuntu.com/view/Precise%20ISO%20Testing%20Dashboard/view/Daily/job/precise-server-ec2-daily/ARCH=i386,REGION=eu-west-1,STORAGE=instance-store,label=ubuntu-server-ec2-testing/45/artifact/None/i386/m1.small/instance-store/i-929cf6db/uec2-20120119-0340-74e55fac26b44d-terminated.console.txt

Revision history for this message
Scott Moser (smoser) wrote :

after looking at the console pointed at in comment 3, I thought i might be able to trigger this by issuing 'reboot-instances' on a running node. I was not successful in doing so.

In a instance of:
 eu-west-1 ami-738fb007 ubuntu-precise-daily-i386-server-20120119

I issued reboot via 'euca-reboot-instances' 10 times and colllected log output (the low number is really because it takes 5m to get a console output). None of these resulted in the fsck issue.

Revision history for this message
Scott Moser (smoser) wrote :

after doing this, i caught a fsck device busy error:

args="--region=eu-west-1 i-087e0b41'
for((i=12;i<50;i++)); do
  echo $(date): i=$i; euca-reboot-instances $args && sleep 1m &&
  euca-get-console-output $args > /tmp/console.hard-reboot$i; done

the 37th boot (attaching) showed this error.

Revision history for this message
Scott Moser (smoser) wrote :

Seen again at [1] from tests for precise alpha-2 at [2]
--
 [1] https://jenkins.qa.ubuntu.com/view/Precise/job/precise-server-ec2/ARCH=i386,REGION=us-west-2,STORAGE=instance-store,TEST=cloud-config,label=ubuntu-server-ec2-testing/lastBuild/artifact/None/i386/m1.small/instance-store/i-e688ccd6/uec2-20120202-1453-d28473559d2145-terminated.console.txt
 [2] https://jenkins.qa.ubuntu.com/job/precise-server-ec2/3/ARCH=i386,REGION=us-west-2,STORAGE=instance-store,TEST=cloud-config,label=ubuntu-server-ec2-testing/

Scott Moser (smoser)
description: updated
Revision history for this message
Stefan Bader (smb) wrote :

Hm, when I look at /etc/init/cloud-init-local.conf and */cloud-init.conf they seem to say start on mounted MOUNTPOINT=/. So I'd somehow expect mountall to be finished with / at that point. Still the output of cloud-init and mountall seem to intermix often. As if they are executed around the same time...

Revision history for this message
Stefan Bader (smb) wrote :

Ok, / is non-sense as it is certainly mounted and also the failing fsck is fot /mnt. But generally it looks like mountall and cloud-init running concurrently and anything that opens the device may block a rw access for the fsck. So any blkid could do that. The window is not very big but who knows...

Revision history for this message
Scott Moser (smoser) wrote :

I think there are 2 things that could be causing this from cloud-init:
 a.) cloudinit/CloudConfig/cc_resize.py [1]
   This uses blkid, but almost certainly only passes blkid a device node that represents '/'. It basically does a stat on /, gets major and minor, creates a device node that has the same major, minor and calls blkid on that like:
    blkid -c /dev/null -sTYPE -ovalue /tmp/THATDEV
   It does this to find the filesystem type of /.

 b.) cloudinit/DataSourceOVF.py [2]
   This would run early in the boot, and possibly race with fsck. It is trying to determine if each device in /dev matching '^(sr[0-9]+|hd[a-z]|xvd.*)' is a cdrom. Notable in that list is 'xvd.*'. If that a device matching that is not mounted, then we do a read(512) in an attempt to determine if there is media in the CD-ROM drive, and if *that* succeeds, it will try to mount read-only and look for some files, then unmount.

It seems to me that 'b' is a potential source of the issue, as either the 'read()' or the mount, check, umount could cause the resource busy option.

2 things that could address that:
 a.) restrict devices being checked to 'xvd[a-z]' (eliminating the possible mount of /dev/xvda[0-9] entirely. This could still cause issue on other devices like /dev/xvdb.
 b.) mount with 'ro,noload' (per 'man mount')

I'm *guessing* that fsck is trying to open the device in read-only and the mount in 'b' is not truely read-only.

Any thoughts on how I can essentially poll these devices looking for "ovf data" and avoid this race?
Woudl a badly timed 'read()' interfere with fsck ? (ie, if it was an open RO, not RW?)

One thing that would filter out this device would be to filter out non-CD ROM devices, but I actually like the fact that the device does not have to be a CDROM.

--
[1] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/cloud-init/precise/view/head:/cloudinit/CloudConfig/cc_resizefs.py
[2] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/cloud-init/precise/view/head:/cloudinit/DataSourceOVF.py

Revision history for this message
Scott Moser (smoser) wrote :

Note, assuming our theory on the circumstances of the race is correct above (that it is 'read' or mount of that device), I was unable to reproduce with a loop device, which I thought would have caused a similar issue:

1. truncate --size 100m /tmp/my.img
2. mkfs -F -t ext3 /tmp/my.img
3. LODEV=$(sudo losetup --show --find /tmp/my.img)
4. sudo python -c 'import time, sys; print sys.argv[1]; fp=open(sys.argv[1]); time.sleep(30); fp.close();' /dev/loop1 &
   # inside the 30 seconds that that will have an open RO file handle, do step 5
5. sudo fsck.ext3 -f /dev/loop1 # '-f' forces fsck even if not marked dirty
   or
    sudo fsck.ext3 /dev/loop1

Other things I've found:
 * using 'open(sys.argv[1],"rw")' does not change anything, fsck will still continue.
 * changing step 4 to 'mount -o ro /dev/loop1 /mnt' *will* cause fsck.ext3 to complain that the filesystem is mounted and print a similar error message.

So, I'm suspecting that it is indeed the OVF code's mount -o,ro that is causing this problem.

I'm not sure how to resolve it, other than limiting the mounts checked to FS of iso9660 only.

Revision history for this message
Stefan Bader (smb) wrote :

Hm weird. I would have expected at least a rw open on the block device should fail, let alone a ro (since fsck probably wants to modify the fs). Though I'd need to check exactly what access mode mount -oro uses (maybe its some ro exclusive). Actually it is not really a read-only open on the block device (that much I know) because this still will cause changes to the superblock (mount time and so on) if that is possible and the block device itself is not read-only.
Beside that you could restrict the devices touched by looking at /sys/block/?/device/type which is 5 (cdrom) only for cd type devices.
The other option would be, if there is an event that mountall emits as the final one (done with automounting), that would be a safer time to start with the cloud-init, wouldn't it?

Revision history for this message
Scott Moser (smoser) wrote :

I believe this is fixed in cloud-init revision 515
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/515

Note, we could also restrict the device type searched by checking /sys/block/devname/device/type as stefan suggested.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.6.3~bzr519-0ubuntu1

---------------
cloud-init (0.6.3~bzr519-0ubuntu1) precise; urgency=low

  * New upstream snapshot.
    - [Mike Milner] add support for managing CA Certificates (LP: #915232)
    - in ci-info lines, use '.' to for empty field for easier machine reading
    - support empty lines in "#include" files (LP: #923043)
    - [Jef Baeur] support configuration of salt minions Bauer) (LP: #927795)
    - DataSourceOVF: only search for OVF data on ISO9660 filesystems (LP: #898373)
    - DataSourceConfigDrive: support getting data from openstack config drive
      (LP: #857378)
    - [Juerg Haefliger] formating and pylint cleanups
  * increase timeouts for initial config check for metadata service
    to address slow metadata service in openstack
  * add awareness of ConfigDrive data source
 -- Scott Moser <email address hidden> Thu, 16 Feb 2012 17:27:05 -0500

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.