Ubuntu
cloud-init package

Unable to configure swap space on ephemeral disk in Azure

Bug #1642383 reported by Stephen A. Zarkos
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init (Ubuntu)
Fix Released
High
Unassigned
Xenial
Fix Released
Medium
Unassigned
Yakkety
Confirmed
Medium
Unassigned
Zesty
Fix Released
High
Unassigned

Bug Description

I'm following the instructions to send cloud-config and repartition the ephemeral disk in Azure: https://wiki.ubuntu.com/AzureSwapPartitions. These instructions fail on Ubuntu 14.04 and 16.04 (did not test 12.04 yet).

Repro:

1) Add the following content to a file such as customdata.txt:

 #cloud-config
 disk_setup:
     ephemeral0:
         table_type: mbr
         layout: [66, [33, 82]]
         overwrite: True
 fs_setup:
     - device: ephemeral0.1
       filesystem: ext4
     - device: ephemeral0.2
       filesystem: swap
 mounts:
     - ["ephemeral0.1", "/mnt2"]
     - ["ephemeral0.2", "none", "swap", "sw", "0", "0"]

2) Create a VM in Azure with this cloud-config:

 $ azure group create ubtest 'east us'

 $ azure vm create ubtest ubtest-1604 Linux --image-urn canonical:UbuntuServer:16.04.0-LTS:latest --admin-username azureuser --admin-password '<PASSWORD>' --nic-name ubtest-1604 --location "East US" --vnet-name "ubtest" --vnet-address-prefix "10.0.0.0/24" --vnet-subnet-name "ubtest" --vnet-subnet-address-prefix "10.0.0.0/24" --public-ip-name "ubtest" --public-ip-domain-name "ubtest" --custom-data ./customdata.txt

Expected Result:

 - /dev/disk/cloud/azure_resource should have two partitions; one ext4 and a second swap
 - /dev/disk/cloud/azure_resource-part1 should be mounted on /mnt2
 - /dev/disk/cloud/azure_resource-part2 should be active swap space
 - /etc/fstab should be configured correctly

Actual Results:

 - /dev/disk/cloud/azure_resource-part1 is properly mounted on /mnt2.
 - On Ubuntu 14.04, /dev/disk/cloud/azure_resource is re-partitioned as expected.
 - On Ubuntu 16.04, /dev/disk/cloud/azure_resource is *not* re-partitioned. There is only a single partition (no swap space).
 - On Ubuntu 14.04 and 16.04, /etc/fstab includes an incorrect entry for the swap space:

   ephemeral0.2 none swap sw,comment=cloudconfig 0 0

This clearly won't work, it should resolve to "/dev/disk/cloud/azure_resource-part2". I do notice that on Ubuntu 14.04, the ephemeral disk is re-partitioned correctly, but there is no symlink in /dev/disk/azure/ to the second partition. These symlinks are created by /lib/udev/rules.d/66-azure-ephemeral.rules. Perhaps cloud-init needs to re-trigger this udev rule after repartitioning the disk.

FYI, I also tested with the latest cloud-init that is currently in xenial-proposed (0.7.8-47-gb6561a1-0ubuntu1~16.04.1) with the same results.

Related bugs:
 * bug 1460715: MBR disk setup fails because sfdisk no longer accepts M as a valid unit
 * bug 1647708: azure ephemeral disk not mounted at boot

See original description
Revision history for this message
Ryan Harper (raharper) wrote :
Download full text (3.7 KiB)

Attempting to re-run the specific config on an existing Trusty instance:

1) sudo umount /mnt
2) sudo vi /etc/fstab and remove /mnt entry, save, quit
3) write out #cloud-config from this bug to user-data.cfg
4) sudo cloud-init --files user-data.cfg single --name cc_disk_setup --frequency always
5) sudo cloud-init --files user-data.cfg single --name cc_mounts --frequency always

This correctly creates the partitions, links, mounts and fstab entries. Now to test
why this appears to fail during initial boot.

ubuntu@trusty-161117-1648:~$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

ubuntu@trusty-161117-1648:~$ cat /etc/cloud/build.info
build_name: server
serial: 20161111

ubuntu@trusty-161117-1648:~$ cat /proc/partitions
major minor #blocks name

   1 0 65536 ram0
   1 1 65536 ram1
   1 2 65536 ram2
   1 3 65536 ram3
   1 4 65536 ram4
   1 5 65536 ram5
   1 6 65536 ram6
   1 7 65536 ram7
   1 8 65536 ram8
   1 9 65536 ram9
   1 10 65536 ram10
   1 11 65536 ram11
   1 12 65536 ram12
   1 13 65536 ram13
   1 14 65536 ram14
   1 15 65536 ram15
  11 0 7168 sr0
   8 0 30720000 sda
   8 1 30718976 sda1
   8 16 298844160 sdb
   8 17 197236735 sdb1
   8 18 101607424 sdb2

ubuntu@trusty-161117-1648:~$ ls -al /dev/disk/azure/
total 0
drwxr-xr-x 2 root root 140 Nov 17 18:25 .
drwxr-xr-x 6 root root 120 Nov 17 16:50 ..
lrwxrwxrwx 1 root root 9 Nov 17 18:25 resource -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 17 18:28 resource-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Nov 17 18:28 resource-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 9 Nov 17 16:50 root -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 17 16:50 root-part1 -> ../../sda1

ubuntu@trusty-161117-1648:~$ cat /proc/swaps
Filename Type Size Used Priority
/dev/sdb2 partition 101607420 0 -1
ubuntu@trusty-161117-1648:~$ cat /etc/fstab
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
UUID=167596ac-7e64-403a-9407-bf06b48032a2 / ext4 defaults,discard 0 0

# CLOUD_IMG: This file was created/modified by the Cloud Image build process
# The following is used to dynamically configured additional
# NICs. Do not remove unless you know what you are doing.
none /etc/network/interfaces.dynamic.d tmpfs nodev,noexec,nosuid,size=64K 0 0
/dev/disk/azure/resource-part1 /mnt2 auto defaults,nobootwait,comment=cloudconfig 0 2
/dev/disk/azure/resource-part2 none swap sw,comment=cloudconfig 0 0

ubuntu@trusty-161117-1648:~$ cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=3561560k,nr_inodes=890390,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=713472k,mode=755 0 0
/dev/sda1 / ext4 rw,relatime,discard,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatim...

Read more...

Revision history for this message
Ryan Harper (raharper) wrote :

Same test on Xenial instance (with 0.7.8-1 cloud-init) fails with:

ubuntu@xenial-161117-1647:~$ sudo cloud-init --file user-data.cfg single --name cc_disk_setup --frequency always
Cloud-init v. 0.7.8 running 'single' at Thu, 17 Nov 2016 18:46:52 +0000. Up 6996.70 seconds.
2016-11-17 18:46:52,763 - util.py[WARNING]: Failed partitioning operation
Failed to partition device /dev/sdb
Unexpected error while running command.
Command: ['/sbin/sfdisk', '--Linux', '-uM', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: ''
Stderr: "sfdisk: --Linux option is unnecessary and deprecated\nsfdisk: unsupported unit 'M'\n"
2016-11-17 18:46:52,884 - util.py[WARNING]: Failed during filesystem operation
Failed during disk check for /dev/sdb2
Unexpected error while running command.
Command: ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb2', '--nodeps']
Exit code: 32
Reason: -
Stdout: ''
Stderr: 'lsblk: /dev/sdb2: not a block device\n'

This looks fatal on 16.04 for repartitioning, due to sfdisk changing parameters upstream

This is addressed in https://bugs.launchpad.net/cloud-init/+bug/1460715 which is not upstream yet in cloud-init, and trivially not in the cloud-init proposed package (0.7.8-47).

Scott Moser (smoser)
description: updated
description: updated
Scott Moser (smoser)
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
description: updated
Revision history for this message
Stephen A. Zarkos (stevez) wrote :

Hi Scott et al. Can we please raise the priority of this? It seems there's no longer a way to create swap space on Ubuntu VMs on Azure.

Jon Grimm (jgrimm)
Changed in cloud-init (Ubuntu):
importance: Medium → High
Revision history for this message
Jon Grimm (jgrimm) wrote :

Hi Stephen, it appears the fix is in trunk now, so hopefully easy enough to get into zesty & then back to xenial. Need to chat with Scott on outlook, but yes, priority high for us.

Revision history for this message
Stephen A. Zarkos (stevez) wrote :

Great news, thanks for the update! :)

Revision history for this message
Scott Moser (smoser) wrote :

Hi,
I uploaded a fix for bug 1460715 to xenial-proposed. It is still in the queue, and hopefully will be accepted into xenial-proposed soon.

Once 0.7.8-49-g9e904bb-0ubuntu1~16.04.2 is available the example cloud-config above will successfully enable swap, and the /mnt2 filesystem will be created. There is still an issue that /mnt2 will not be mounted on the first boot. A 'mount -a' or 'mount /mnt2' would fix that. I'll work further and get a fix in that will do that correctly and mark *that* chang to fix this bug.

Scott Moser (smoser)
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
status: New → Confirmed
Changed in cloud-init (Ubuntu Yakkety):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

This is currently fixed in cloud-init in zesty 0.7.8-67-gc9c9197-0ubuntu1 .
I've verified it functional with the user-data provided in the description on todays Azure image.

Changed in cloud-init (Ubuntu Zesty):
status: Confirmed → Fix Released
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

The current sru of cloud-init to xenial will fix the swap, but it will take bug 1647708 to be fixed in order to get the first device mounted. That should happen with the next cloud-init update to xenial.

Revision history for this message
Stephen A. Zarkos (stevez) wrote :

This seems resolved in Xenial, but the issue also impacts Trusty. Will this fix get backported?

Revision history for this message
Scott Moser (smoser) wrote :

This is tested as working on 0.7.8-49-g9e904bb-0ubuntu1~16.04.3 in xenial in azure
$ cat /etc/cloud/build.info
build_name: server
serial: 20170117
$ cat /proc/swaps
Filename Type Size Used Priority
/dev/sdb2 partition 17824764 0 -1
$ df -h /mnt2
 df -h /mnt2
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 33G 48M 31G 1% /mnt2
smoser@smoser0119x:~$

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Stephen,

I've just tested this on trusty images on azure
  b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-trusty-14_04_5-LTS-amd64-server-20170118-en-us-30GB
and also in openstack (serial 20170112), and found both working.

I just launched instance with the user-data in the description and verified swap is set up and /mnt2 is mounted.

If you find otherwise, please provide some more info.

Scott

Revision history for this message
Stephen A. Zarkos (stevez) wrote :

Hi Scott,

Thanks for testing. I tested again (~10 times) with the latest Trusty image and I wasn't able to reproduce the issue either. I'm not sure what happened, if it's some timing issue or was just typo on my part :/ I did not keep the repro VM around, unfortunately.

In that case I'm OK calling this resolved since I'm unable to repro again.

Thanks,
Steve

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.