Idempotency for OpenStack API

Registered by haruka tanizawa on 2013-09-12

-- Description --
* In this blueprint, I suggest ClientToken that user can specify, and post
  requests which idempotency is guaranteed by the previous ClientToken.
* First, ClientToken is implemented in the run_instance of AmazonEC2.
  It is also implemented in OpenStack EC2 run_instance.
  - https://bugs.launchpad.net/nova/+bug/1188327
  - https://review.openstack.org/#/c/32060/
* I think that there is no reason to ClientToken is limited to only the run_instance.
  I suggest implementation of ClientToken to other OpenStack API.
* This suggestion is based on following exchange...
  - https://lists.launchpad.net/openstack/msg13082.html

-- Use cases --
1. User boots own server.
2. Unfortunately, the client has gone down.
3. User is no way to know how was his server.(Even if, in fact, his server runs normally.)
4. User thought booting is failure, so user booted a new server with same param request as 3.

* This appears to be no problems at first glance, but money problems may occur in service.
(There is a possibility that the overage charges for two servers.)

So user need to way how is his server.
In this case, idempotency client token is so useful.
How many times user put POST method, it is guaranteed the state of the POST
which was same with return of user's first POST request.

-- How to use? --
In curl command, you add header of request 'X-Client-Token: foo'.
In this name, I already implemented python-novaclient implementation.
So, you can put request like 'nova --x-client-token boot --flavor 1 --image hoge ...'
If 'X-Client-Token' become another good name, I am going to refoctor.

-- Configuration --
You don't need any specific permission and policies.
Only just you put or not. It does not affect in the configuration.
If you don't put it, that API works same as before.

-- Implements --
* Plan to implement ClientToken across POST method of whole OpenStack for idempotency.
In OpenStackAPI, ClientToken shall be specified in the header.

-- Remaining challenges --
I implemented idempotent.py temporarily.
Appropriate file path is need to be considered.
This feature is decorator. And I applied this decorator to 'Create Server'
as first step.

-- I assume the code like this. --
https://github.com/ntt-sic/nova/tree/idempo (Nova)
nova/nova/openstack/common/idempotent.py
nova/nova/api/openstack/compute/servers.py

Blueprint information

Status:
Not started
Approver:
Andrew Laski
Priority:
Undefined
Drafter:
haruka tanizawa
Direction:
Needs approval
Assignee:
haruka tanizawa
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

@alaski Please change the state of "Completed by"'. I'd like to reopen a discussion in nova-spec #99576. Or need to start another BP? --tinoue

Summary of suggestion to my idea from [openstack-dev].
Thanks.
* https://blueprints.launchpad.net/nova/+spec/cross-service-request-id
* https://blueprints.launchpad.net/keystone/+spec/delegation-workplans
* https://blueprints.launchpad.net/nova/+spec/instance-tasks-api
* Need to persistent and how about a combination of cross-service-request-id and TaskFlow?
* This is also important for Heat blueprint.
* Generating correlation-id from Keystone is very appropriate and reliable. --haruka(2013/11/27)

I think this needs some more discussion and planning, perhaps on the mailing list. Should these tokens work for all requests or just boot? How are they going to be persisted? What is the scope of the token, per tenant? Is the token the only key used for idempotency, i.e. if a client uses the same token as a previous request somehow is there a way to tell that it's the same request or a different one? etc... --alaski

I agree with alaski, I would like to see what calls these tokens would work for. Also your sample code stores them in memcache, which isn't a hard requirement in OpenStack at the moment so I think this needs to go through the nova.db api. --jogo

So we're actually trying to solve this issue a different way. We're adding a tasks resource which can be used to track long running actions in the API. It's still being drafted but details will be at https://blueprints.launchpad.net/nova/+spec/instance-tasks-api. This should allow a client to check on whether they're request went through or not. If there are ways to make that easier for a client please leave details on the bp or reviews. --alaski

Please explain in this document the feature you are trying to attain. ClientTOken can mean many things. Are you referring to the Keystone tokens or something else. Referring to the Amazon doc does not clarify what you are trying to get at.

It sounds like you are looking for a way to track long workflows across the system. If that is the case, it would tie in with the Keystone blueprint for "Workplans" which are a way of specifying delegations for the various components involved. Is this correct? --Adam Young

I think that blueprint is a little bit different.
I just only want to implement 'mark' from outside of the openstack
because in case of async API, if client downs, I have no way to detect
that API became what state. --haruka

This feature occurred in Amazon's 'Ensuring Idempotency'
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html

Example implementation for using python decorator.
https://github.com/ntt-sic/nova/tree/idempo

Gerrit topic: https://review.openstack.org/#q,topic:bp/idempotentcy-client-token,n,z

Addressed by: https://review.openstack.org/99576
    Add client token to CreateServer

(?)

Work Items

Work items:
v2 Create server: DONE
v2 Create keypair: DONE

Dependency tree

* Blueprints in grey have been implemented.