Chaos drivers to test corner cases

Registered by Joshua Harlow on 2013-05-04

It would be very neat to implement a set of 'chaos' drivers that could be implemented for each pluggable 'driver-like' backend that can be provided in nova (for example the servicegroup driver, or the vm driver..). The concept would be that a single chaos driver would wrap a working driver and the chaos driver would randomly (likely via a specified seed and/or rate) raise different types of exceptions that the driver interface allows to be thrown. For the cases where it does not throw it would pass on the request to the underlying driver, thus it would act as a driver that sporadically fails at a much higher rate (depending on the randomness) than the underlying wrapped driver. This would be neat as a way to test the corner cases of the code using said driver. It also provides a repeatable (due to the set seed) & unique view into what state the system is left in after said exceptions occur. The results of such types of 'chaos' drivers could provide unique insights into how the overall system recovers from failures and could lead to new code to ensure the system is left in a consistent state (if it is not already).

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Joshua Harlow
Direction:
Needs approval
Assignee:
None
Definition:
Drafting
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

JH: It might also be useful to have this in cinder, heat, quantum ... to test there ability to handle 'chaos'
JH: Similarily one could create chaos clients that wrap the existing nova/glance... clients but cause spurious http errors (or similar)JH: It might also be useful to have this in cinder, heat, quantum ... to test there ability to handle 'chaos'
JH: Similarily one could create chaos clients that wrap the existing nova/glance... clients but cause spurious http errors (or similar)

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.