Improve API attribute test coverage for Nova API

Registered by Ken'ichi Ohmichi on 2014-03-05

Now Tempest does not check API responses of Nova API in many cases.
For example, Tempest does not check what API attributes("flavor", "image",
etc.) should be included in the response body of "create a server" API.
So we need to improve Tempest coverage from this viewpoint for blocking
any backward incompatibility change happens.

The tasks are managed on https://etherpad.openstack.org/p/nova-api-attribute-test

Blueprint information

Status:
Complete
Approver:
Sean Dague
Priority:
Not
Drafter:
Ken'ichi Ohmichi
Direction:
Needs approval
Assignee:
Ken'ichi Ohmichi
Definition:
Superseded
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
Matthew Treinish on 2014-06-03

Related branches

Sprints

Whiteboard

DESIGN

Tempest validates a response body with JSONSchema.
Tempest already contains a request body schema for the negative test generator.
A response body schema also is defined in the same schema file.

PROCESS
As the first step, we need to cleanup many Nova v2/v3 API tests with blueprint nova-api-test-inheritance.
After that, we will add this API attribute checks to clean v2/v3 API tests.

Gerrit topic: https://review.openstack.org/#q,topic:bp/nova-api-attribute-test,n,z

Addressed by: https://review.openstack.org/78149
    Verify service attributes through Nova "get services" API

Addressed by: https://review.openstack.org/79871
    Skip Nova API attribute tests if an XML response

Addressed by: https://review.openstack.org/77517
    Verify server attributes which admin can get through Nova APIs

Addressed by: https://review.openstack.org/78593
    Validate Volume attributes of Nova POST & GET API

Addressed by: https://review.openstack.org/78950
    Verify the list volume attributes of Nova APIs

Addressed by: https://review.openstack.org/79919
    JSON Schema validation for attach volume attribute

Addressed by: https://review.openstack.org/80135
    Verify service attributes through Nova os-flavor-access API

Addressed by: https://review.openstack.org/80147
    Verify "update a server" API response attributes

Addressed by: https://review.openstack.org/80150
    Verify service attributes through Nova os-quota-sets API

Addressed by: https://review.openstack.org/80174
    Making 'body' param optional for resp validation

Addressed by: https://review.openstack.org/80175
    Verify "get a server" API response attributes

Addressed by: https://review.openstack.org/77807
    Verify "get az list" API response attributes

Addressed by: https://review.openstack.org/80186
    Validate of Nova APIs Floating IPs attributes

Addressed by: https://review.openstack.org/80202
    Move verification of service attributes into service client

Addressed by: https://review.openstack.org/80527
    Verify Nova create & get Floating IP attributes

Addressed by: https://review.openstack.org/80531
    Check attach-detach FIP & get FIP pool attributes

Addressed by: https://review.openstack.org/80543
    Verify get flavor attributes of Nova V2/V3 APIs

Addressed by: https://review.openstack.org/78527
    Validate for list flavor attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/80548
    Verify detail_list flavor attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/78507
    Verify Image attributes through Nova V2 GET API

Addressed by: https://review.openstack.org/80890
    Validate get fixed-ips attributes of Nova V2 API

Addressed by: https://review.openstack.org/80904
    Verify the response attributes of 'list_hosts'

Addressed by: https://review.openstack.org/78908
    Verify list Image attributes through Nova V2 API

Addressed by: https://review.openstack.org/80911
    Check detail list Images attributes of Nova V2 API

Addressed by: https://review.openstack.org/80919
    Validate get keypair attributes of Nova V2/V3 API

Addressed by: https://review.openstack.org/80923
    Verify list agents attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/80924
    Validate list_keypair attribute of Nova V2/V3 APIs

Addressed by: https://review.openstack.org/80931
    Verify the response attributes of 'show_host_detail'

Addressed by: https://review.openstack.org/80936
    Verify "get version" API response attributes

Addressed by: https://review.openstack.org/81136
    Add os-migration tests for Nova v2 API

Addressed by: https://review.openstack.org/81151
    Validate get limits attributes of Nova V2 API

Addressed by: https://review.openstack.org/81194
    Check show_hypervisor attributes of Nova V2/V3 API

Addressed by: https://review.openstack.org/81219
    Verify list_hypervisors_detail Nova V2/V3 API

Addressed by: https://review.openstack.org/81222
    Validate hypervisors_servers Nova V2/V3 API

Addressed by: https://review.openstack.org/81439
    Verify quotas attributes through Nova "get/get default quotas" API

Addressed by: https://review.openstack.org/81442
    Check list_hypervisor attributes of Nova V2/V3 API

Addressed by: https://review.openstack.org/81444
    Verify "enable a service" API response attributes

Addressed by: https://review.openstack.org/81452
    Verify security_groups attributes through Nova "list security groups" API

Addressed by: https://review.openstack.org/81465
    Check hypervisor statistics attributes of Nova API

Addressed by: https://review.openstack.org/81476
    Verify hypervisor uptime attributes of Nova API

Addressed by: https://review.openstack.org/81477
    Verify certificate API response attributes

Addressed by: https://review.openstack.org/81483
    Verify the response attributes of 'list_aggregates'

Addressed by: https://review.openstack.org/81687
    Verify the response status of create delete Image

Addressed by: https://review.openstack.org/81699
    Validate image metadata attributes of Nova APIs

Addressed by: https://review.openstack.org/81701
    Verify "get quotas detail" API response attributes

Addressed by: https://review.openstack.org/81748
    Check attributes of image meta item Nova APIs

Addressed by: https://review.openstack.org/81766
    Verify tenant usages API response attributes

Addressed by: https://review.openstack.org/82400
    Define 'links' as a common parameter type

Addressed by: https://review.openstack.org/82464
    Check create/delete keypair attribute of Nova APIs

Addressed by: https://review.openstack.org/82681
    Verify attributes through Nova os-quota-sets API

Addressed by: https://review.openstack.org/82684
    Check reserve/unreserve fixed-ips APIs attributes

Addressed by: https://review.openstack.org/82712
    Verify the response attributes of 'get_aggregate'

Addressed by: https://review.openstack.org/82718
    Verify flavor extra specs attributes of Nova APIs

Addressed by: https://review.openstack.org/82728
    Verify list extensions attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/82998
    Verify attributes through Nova os-security-groups API

Addressed by: https://review.openstack.org/83012
    Verify the response attributes of 'startup_host' API

Addressed by: https://review.openstack.org/83266
    Verify the create/delete volume APIs attributes

Addressed by: https://review.openstack.org/83269
    Verify "list interfaces" Nova v2/v3 API response attributes

Addressed by: https://review.openstack.org/83303
    Validate get_instance_usage_audit_log Nova V2 API

Addressed by: https://review.openstack.org/83305
    Validate list_instance_usage_audit_log Nova V2 API

Addressed by: https://review.openstack.org/83331
    Verify the response attributes of 'shutdown/reboot_host' API

Addressed by: https://review.openstack.org/83661
    Add a response header validation

Addressed by: https://review.openstack.org/83666
    Verify v2 os-migration API response attributes

Addressed by: https://review.openstack.org/83674
    Verify "show/create interface" Nova v2/v3 API response attributes

Addressed by: https://review.openstack.org/83692
    Verify the response attributes of 'create-aggregate' API

Addressed by: https://review.openstack.org/83694
    Verify the response attributes of 'update-aggregate' API

Addressed by: https://review.openstack.org/84044
    Check add/remove flavor access APIs attributes

Addressed by: https://review.openstack.org/84333
    Check attributes of create/delete sec groups rule"

Addressed by: https://review.openstack.org/84344
    Verify create agent attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/84403
    Check attributes of change/get server password API

Addressed by: https://review.openstack.org/84625
    Verify delete agent response of V2/V3 APIs

Addressed by: https://review.openstack.org/84636
    Check get_vnc_console attributes of Nova APIs

Addressed by: https://review.openstack.org/84639
    Verify the response attributes of 'aggregate-add-host' API

Addressed by: https://review.openstack.org/84651
    validate server action 'console_output' attributes

Addressed by: https://review.openstack.org/84663
    Verify update agent attributes of V2/V3 APIs

Addressed by: https://review.openstack.org/84677
    Verify the response attributes of 'aggregate-remove-host' API

Addressed by: https://review.openstack.org/84944
    Verify delete quota response of V2/V3 APIs

Addressed by: https://review.openstack.org/84948
     Verify the response attributes of 'aggregate-set-metadata' API

Addressed by: https://review.openstack.org/84380
    Verify Server Actions attributes of Nova APIs

Addressed by: https://review.openstack.org/85234
    Check rescue_server action attributes of Nova APIs

Addressed by: https://review.openstack.org/85246
    Verify list_virtual_interfaces attributes of API

Addressed by: https://review.openstack.org/85250
    Verify delete interface response of V2/V3 APIs

Addressed by: https://review.openstack.org/85258
    Verify delete security group response of V2 API

Addressed by: https://review.openstack.org/85666
    Verify list_instance_action attributes of Nova API

Addressed by: https://review.openstack.org/85887
    Verify get_instance_action attributes of Nova API

Addressed by: https://review.openstack.org/85889
    Verify the response attributes of 'set-server-metadata' API

Addressed by: https://review.openstack.org/86243
    Verify Set/Get/Delete server meta item attributes

Addressed by: https://review.openstack.org/86851
    Verify set server-metadata response of V2/V3 API

Addressed by: https://review.openstack.org/88229
    Verify delete server attributes of Nova API

Addressed by: https://review.openstack.org/89316
    Check list servers attributes of Nova APIs

Addressed by: https://review.openstack.org/89478
    Verify delete aggregate attributes of Nova APIs

Addressed by: https://review.openstack.org/89772
    Verify 'list-server-metadata' APIs attributes

Addressed by: https://review.openstack.org/90633
    Verify update host Nova V2/V3 APIs attributes

Addressed by: https://review.openstack.org/91498
    Separate common 'start_up' definition from v2 schema

Addressed by: https://review.openstack.org/92529
    Verify server list_addresses V2/V3 APIs attributes

Addressed by: https://review.openstack.org/92532
    Verify list_addresses_by_network APIs attributes

Addressed by: https://review.openstack.org/95064
    Check create/get/delete V2 server_group attributes

Addressed by: https://review.openstack.org/95065
    Verify V2 list server_group API attributes

Addressed by: https://review.openstack.org/95407
    Fix update_server JSON Schema for Nova V2

Addressed by: https://review.openstack.org/95409
    Check V2 rebuild server action API attributes

Addressed by: https://review.openstack.org/97161
    Verify 'update-server-metadata' APIs attributes

Addressed by: https://review.openstack.org/99319
    make 'server' as 'required' in server update schema

Addressed by: https://review.openstack.org/98971
    Correcting 'id' type in compute schema

Addressed by: https://review.openstack.org/99836
    Validate server detail list attribute of Nova APIs

Addressed by: https://review.openstack.org/100816
    Check create-Floating-IP-Bulk Nova API attributes

Addressed by: https://review.openstack.org/101100
    Verify list-Floating-IP-Bulk Nova API attributes

Addressed by: https://review.openstack.org/101111
    Verify delete-Floating-IP-Bulk Nova API attributes

Addressed by: https://review.openstack.org/110206
    Separate create server schema for admin password

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.