OpenStack Compute (nova)

rebuild volume-backed instance with image does not replace image in root disk

Bug #1482040 reported by Zhenyu Zheng
62
This bug affects 10 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Ocata
Confirmed
Medium
Unassigned
Pike
Confirmed
Medium
Unassigned

Bug Description

When rebuild an volume-backed instance using normal images from glance,
nothing will be changed for the instance after rebuild except the image name
will be added to the instance, the instance is actually again built using the same
volume.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume daaddb77-4257-4ccd-86f2-220b31a0ce9b --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test_rebuild
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001a |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | KX5f2qfJBs9S |
| config_drive | |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | test_rebuild |
| os-extended-volumes:volumes_attached | [{"id": "daaddb77-4257-4ccd-86f2-220b31a0ce9b"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:09:06Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+--------------------------------------+--------------------------------------------------+

root@zheng-dev1:/var/log/nova# nova rebuild 2f91193c-c462-4bcf-ad87-692c3d263cfd 7bdff9a3-d051-4e75-bcd3-de69dbffe063
+-------------------+----------------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | GParrEjj3DEW |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | e9701452373fee3a5897683c6b914f4ba33b38b0fcff3b3719615bbd |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | cirros-0.3.4-x86_64-uec (7bdff9a3-d051-4e75-bcd3-de69dbffe063) |
| metadata | {} |
| name | test_rebuild |
| progress | 0 |
| public network | 2001:db8::19, 172.24.4.24 |
| status | REBUILD |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:09:50Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+-------------------+----------------------------------------------------------------+

root@zheng-dev1:/var/log/nova# nova show 2f91193c-c462-4bcf-ad87-692c3d263cfd
+--------------------------------------+----------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | zheng-dev1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | zheng-dev1 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000001a |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2015-08-06T04:10:00.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2015-08-06T04:09:06Z |
| flavor | m1.tiny (1) |
| hostId | e9701452373fee3a5897683c6b914f4ba33b38b0fcff3b3719615bbd |
| id | 2f91193c-c462-4bcf-ad87-692c3d263cfd |
| image | cirros-0.3.4-x86_64-uec (7bdff9a3-d051-4e75-bcd3-de69dbffe063) |
| key_name | - |
| metadata | {} |
| name | test_rebuild |
| os-extended-volumes:volumes_attached | [{"id": "daaddb77-4257-4ccd-86f2-220b31a0ce9b"}] |
| progress | 0 |
| public network | 2001:db8::19, 172.24.4.24 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| updated | 2015-08-06T04:10:00Z |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
+--------------------------------------+----------------------------------------------------------------+

add using virsh to see the xml, we can see that the instance is still booted from the volume

</disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/disk/by-path/ip-10.250.10.17:3260-iscsi-iqn.2010-10.org.openstack:volume-daaddb77-4257-4ccd-86f2-220b31a0ce9b-lun-1'/>
      <target dev='vda' bus='virtio'/>
      <serial>daaddb77-4257-4ccd-86f2-220b31a0ce9b</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

Zhenyu Zheng (zhengzhenyu)
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Markus Zoeller (markus_z) (mzoeller)
tags: added: libvirt rebuild volumes
Markus Zoeller (markus_z) (mzoeller)
tags: removed: rebuild
Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :

Hi Zhenyu,

I'm afraid I could not parse your problem description correctly, so I don't know what exactly is the problem you're facing.

Would be useful if you can write a list of sequential steps to reproduce the problem you're facing.

[A friendly note: https://wiki.openstack.org/wiki/BugFilingRecommendations]

Changed in nova:
status: New → Incomplete
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/305079

Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Laszlo Hegedus (laszlo-hegedus)
status: Incomplete → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Laszlo Hegedus (laszlo-hegedus) → Zhenyu Zheng (zhengzhenyu)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → Laszlo Hegedus (laszlo-hegedus)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Laszlo Hegedus (<email address hidden>) on branch: master
Review: https://review.openstack.org/305079
Reason: I abandon this, since a proper solution would require rethinking of rebuild. It is not defined yet how rebuild should work with volume-backed instances. Someone should write a bp about this.

Laszlo Hegedus (laszlo-hegedus)
Changed in nova:
assignee: Laszlo Hegedus (laszlo-hegedus) → nobody
status: In Progress → Confirmed
Revision history for this message
prescolt (prescolt) wrote : Re: rebuild volume-backed instance with image

Hi
I upgrade system to liberty but problem still remain
So for rebuild process with ceph backend i must do these step
- Delete instance
- Create neutron port with ip same as before
- Create new volumes base from os image template with size same as before
- Create new instance with same as flavor before, new volume and neutron port created before.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/442295

Changed in nova:
assignee: nobody → Akira KAMIO (kamio)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/442295
Reason: Abandoning as a duplicate of https://review.openstack.org/#/c/305079/

Matt Riedemann (mriedem)
tags: added: rebuild
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/509282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/509282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ae9542eb7da22ad6bcd59145c56b2781aa7b507a
Submitter: Jenkins
Branch: master

commit ae9542eb7da22ad6bcd59145c56b2781aa7b507a
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 3 16:34:26 2017 -0400

    api-ref: add note about rebuild not replacing volume-backed root disk

    We've had several bugs about this over the years, and until we
    actually decide to fix it (or not), we should point out the known
    limitation that volume-backed root disks are not replaced during
    a rebuild.

    Like, if you have a volume-backed instance and rebuild with a new
    image, the root disk is still the volume with the original image.

    Change-Id: I145cab88f782e4b1e630cc432322bc8436413e71
    Related-Bug: #1482040

Matt Riedemann (mriedem)
Changed in nova:
status: In Progress → Confirmed
assignee: Akira KAMIO (kamio) → nobody
summary: - rebuild volume-backed instance with image
+ rebuild volume-backed instance with image does not replace image in root
+ disk
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/520660

Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Chris Friesen (cbf123)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Chris Friesen (cbf123) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/520660
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=132636dd610d7e5cce24848776e942d229199e18
Submitter: Zuul
Branch: master

commit 132636dd610d7e5cce24848776e942d229199e18
Author: Matt Riedemann <email address hidden>
Date: Thu Nov 16 12:18:34 2017 -0500

    Fail fast if changing image on a volume-backed server rebuild

    We don't support changing the image in the root disk of a volume-backed
    server during a rebuild. The API will change the instance.image_ref
    attribute to the newly supplied image_href to the rebuild API but the
    actual image used by the server after the rebuild will be the original
    image, which is wrong.

    We need to just fail fast in this case in the API since the compute
    service doesn't support it. We also need to ensure that instance.image_ref
    doesn't get modified since a missing value here is used by novaclient and
    probably other HTTP API users as an indication of a volume-backed server.

    See the related mailing list discussion for more details:
    http://lists.openstack.org/pipermail/openstack-dev/2017-October/123255.html

    Co-Authored-By: Chris Friesen <email address hidden>
    Change-Id: If4c5fb782bb7e7714fb44f8ca9875121e066bc10
    Closes-Bug: #1482040

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b2

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

Revision history for this message
Belmiro Moreira (moreira-belmiro-email-lists) wrote :

This bug was marked as fixed, however in my opinion the original problem still persists.
When trying to rebuild a "boot for volume instance" the volume is not replaced by a new image (even if we use the same image).
What was introduced was to "fail fast" in the case if a different image is select. But, the issue remains (and is confusing for the user) if the same image is selected.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/830883
Committed: https://opendev.org/openstack/nova/commit/45c5b80fd076d0017f957a2150d7496f6d4a4fcf
Submitter: "Zuul (22348)"
Branch: master

commit 45c5b80fd076d0017f957a2150d7496f6d4a4fcf
Author: Dan Smith <email address hidden>
Date: Thu Feb 24 11:03:39 2022 -0800

    Add API support for rebuilding BFV instances

    This adds a microversion and API support for triggering a rebuild
    of volume-backed instances by leveraging cinder functionality to
    do so.

    Implements: blueprint volume-backed-server-rebuild
    Closes-Bug: #1482040

    Co-Authored-By: Rajat Dhasmana <email address hidden>

    Change-Id: I211ad6b8aa7856eb94bfd40e4fdb7376a7f5c358

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 26.0.0.0rc1

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.