"down" nova-compute service spuriously marked as "up" when disabled/enabled

Just curious, what is "incomplete" about this? Is there more information that I can provide?

Hi Chris, the "incomplete" is for me. I already checked in icehouse release. And need to check the other branches.

Eric, the Incomplete bug status means that more information is needed from the reporter before we can triage.

Sorry. This is my first time for bug fix.

Hi Eric, no worries. You can assign the bug to yourself if you're looking into it. Just be sure to unassign if you decide you don't wish to work on it anymore.

Please see this doc about bug triage to learn how it works: https://wiki.openstack.org/wiki/BugTriage

Add a new field to service table indicate the heart beat time might be a solution but it might introduce complex
such as migrations of existing db
except that ,we might have no good method on checking this guess staled for 1 min is acceptable?

I don't think it's acceptable, no. Any operation involving the scheduler could end up trying to place an instance on the "down" compute node for that minute.

And if we were enabling the service rather than disabling it (or doing any other operation on the service) then we could end up in a state where the scheduler thinks it's available. That could result in operations taking a long time as they block waiting for the RPC timeout since of course the compute node would never respond.

I think it's clearly a flawed design to rely on automatic database row timestamps for status, when both system and user-triggered operations cause those timestamps to be updated.

Not sure why it didn't auto-link, but there's a fix proposed at

https://review.openstack.org/163060

Related fix proposed to branch: master
Review: https://review.openstack.org/168418

Reviewed: https://review.openstack.org/163060
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b9bae02af2168ad64d3b3d28c97c3853cee73272
Submitter: Jenkins
Branch: master

commit b9bae02af2168ad64d3b3d28c97c3853cee73272
Author: Chris Friesen <email address hidden>
Date: Fri Mar 27 09:23:48 2015 -0600

    fix "down" nova-compute service spuriously marked as "up"

    Currently we use the auto-updated "updated_at" field to determine
    whether a service is "up". An end-user can cause the "updated_at"
    field to be updated by disabling or enabling the service, thus
    potentially causing a service that is unavailable to be detected
    as "up". This could result in the scheduler trying to assign
    instances to an unavailable compute node, or in the system
    mistakenly preventing evacuation of an instance.

    The fix is to add a new field to explicitly track the timestamp of
    the last time the service sent in a status report and use that if
    available when testing whether the service is up.

    DocImpact
    This commit will cause a behaviour change for the DB servicegroup
    driver. It will mean that enabling/disabling the service will
    cause the "updated_at" field to change (as before) but that will
    no longer be tied to the "up/down" status of the service. So
    "nova service-list" could show the service as "down" even if it
    shows a recent "updated_at". (But this could happen for the other
    servicegroup drivers already.)

    Closes-Bug: #1420848
    Change-Id: Ied7d47363d0489bca3cf2c711217e1a3b7d24a03

Change abandoned by Chris Friesen (<email address hidden>) on branch: master
Review: https://review.openstack.org/168418
Reason: No support for the change, abandoning.