Compare DB time with local to ensure that there is no mistiming

Registered by Boris Pavlovic on 2013-03-31

All tables in nova DB have 3 columns (deleted_at, updated_at, created_at) that have type datetime.

There are a lot of places where they are used. For example when we are checking is service available we are comparing local time with value in updated_at column.

So if our nodes become mistiming we will get a lot of different problems (for example unavailable servieses). And for administrators there could be not so clear why services are unavailable or something else is not working..

So to reduce problems with time we should implement 2 next things:

1. When nova starts, during init, compare local time to db time, and raise exception if the difference is too large

2. In periodic task, compare local time to db time, and if difference is to large do one of the next thing (it should be specified in conf):

a) nothing - do nothing
b) log - log warning
c) sync - sync time using ntpdate
d) die - kill nova process

Blueprint information

Status:
Not started
Approver:
Russell Bryant
Priority:
Undefined
Drafter:
Boris Pavlovic
Direction:
Needs approval
Assignee:
Boris Pavlovic
Definition:
Drafting
Series goal:
None
Implementation:
Not started
Milestone target:
None

Related branches

Sprints

Whiteboard

Boris, to tackle these, I'd like to help.

How about we do the following:

1. Identify all the 'problem' places
2. Split out fixing some 'problem' places (and document the fix)
3. Profit

Moving to Icehouse since this is still marked as not started. --russellb

Hi Boris,
I am writing because I would start working on this blueprint that you have assigned “Compare DB time with local to ensure that there is no mistiming”
Are you going to start working on this soon? Any problem if I assign it to me?
Thanks
--sjcazzol

Boris, could you please assign to me this blueprint?

Boris, to tackle these, I'd like to help.

How about we do the following:

1. Identify all the 'problem' places
2. Split out fixing some 'problem' places (and document the fix)
3. Profit

Moving to Icehouse since this is still marked as not started. --russellb

Hi Boris,
I am writing because I would start working on this blueprint that you have assigned “Compare DB time with local to ensure that there is no mistiming”
Are you going to start working on this soon? Any problem if I assign it to me?
Thanks
--sjcazzol

Boris, could you please assign to me this blueprint?

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)Boris, to tackle these, I'd like to help.

How about we do the following:

1. Identify all the 'problem' places
2. Split out fixing some 'problem' places (and document the fix)
3. Profit

Moving to Icehouse since this is still marked as not started. --russellb

Hi Boris,
I am writing because I would start working on this blueprint that you have assigned “Compare DB time with local to ensure that there is no mistiming”
Are you going to start working on this soon? Any problem if I assign it to me?
Thanks
--sjcazzol

Boris, could you please assign to me this blueprint?

Boris, to tackle these, I'd like to help.

How about we do the following:

1. Identify all the 'problem' places
2. Split out fixing some 'problem' places (and document the fix)
3. Profit

Moving to Icehouse since this is still marked as not started. --russellb

Hi Boris,
I am writing because I would start working on this blueprint that you have assigned “Compare DB time with local to ensure that there is no mistiming”
Are you going to start working on this soon? Any problem if I assign it to me?
Thanks
--sjcazzol

Boris, could you please assign to me this blueprint?

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.