3. Cloud Infrastructure Test Cases and Traceability to Requirements ¶
3.1. Introduction ¶
The scope of this chapter is to identify and list down test cases based on requirements defined in Reference Architecture-1 (RA-1) . This will serve as traceability between test cases and requirements.
Note that each requirement may have one or more test cases associated with it.
must : Test Cases that are marked as must are considered mandatory and must pass successfully.
should : Test Cases that are marked as should are expected to be fulfilled by NFVI but it is up to each service provider to accept an NFVI targeting reference architecture that is not reflecting on any of those requirements. The same applies to should not.
may : Test cases that are marked as may are considered optional. The same applies to may not.
3.2. Selection Criteria ¶
Test cases below are selected based on available test cases in open-source tools like FuncTest etc.
3.3. Traceability Matrix ¶
The following is a Requirements Traceability Matrix (RTM) mapping Test Case, and/or Test Case Coverage, to RM and RA-1 requirements (config and deployment).
The RTM contains RM config (i.e. .conf) requirements listed “per profile”, followed by RA-1 requirements. Requirements fall into 8 domains: general(gen), infrastructure(inf), VIM(vim), Interface & API(int), Tenants(tnt), LCM(lcm), Assurance(asr), Security(sec).
For detailed information on RM & RA-1 NFVI and VNF requirements, please refer to RI-1 Chapter 3 .
3.3.1. Architecture and OpenStack Requirements ¶
3.3.2. Infrastructure ¶
3.3.3. VIM ¶
3.3.4. Interfaces & APIs ¶
The OpenStack Gates verify all changes proposed mostly by running thousands of Tempest tests completed by Rally scenarios in a few cases. Skipping tests is allowed in all OpenStack Gates and only failures rate the review -1 because of the multiple capabilities and backends selected in the different Gate jobs. The classical Functest containers conform to this model which also fits the heterogeneous user deployments.
From a Anuket Compliance state point, the capabilities are well described in RA1 Core OpenStack Services APIs which allows tuning the test configurations and the test lists to avoid skipping any test. It results that all tests covering optional capabilities and all upstream skipped tests due to known bugs are not executed. All remaining tests must be executed and must pass successfully.
New Functest containers have been proposed for Anuket Compliance which simply override the default test configurations and the default test lists. Any optional capability or services (e.g. Barbican) can be still verified by the classical Functest containers.
The next subsections only detail the Tempest tests which must not be executed from a Compliance state point. The remaining tests have to pass successfully. They cover all together the API testing requirements as asked by RA1 Core OpenStack Services APIs
The following software versions are considered here to verify OpenStack Wallaby selected by Anuket:
software |
version |
---|---|
Functest |
wallaby |
Cinder Tempest plugin |
1.4.0 |
Keystone Tempest plugin |
0.7.0 |
Heat Tempest plugin |
1.2.0 |
Neutron Tempest plugin |
1.4.0 |
Rally OpenStack |
2.1.0 |
Tempest |
27.0.0 |
3.3.4.1. Identity - Keystone ¶
Keystone API is covered in the OpenStack Gates via Tempest and keystone-tempest-plugin as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*api.identity.v3.test_oauth1_tokens |
oauth1 |
.*scenario.test_federated_authentication |
federation |
.*identity.admin.v2 |
API v2 |
.*identity.v2 |
API v2 |
.*identity.v3.test_access_rules |
access_rules |
.*identity.v3.t est_application_credentials.ApplicationCredentialsV3 Test.test_create_application_credential_access_rules |
access_rules |
Keystone API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.keystone
-
KeystoneBasic.add_and_remove_user_role
-
KeystoneBasic.create_add_and_list_user_roles
-
KeystoneBasic.create_and_list_tenants
-
KeystoneBasic.create_and_delete_role
-
KeystoneBasic.create_and_delete_service
-
KeystoneBasic.get_entities
-
KeystoneBasic.create_update_and_delete_tenant
-
KeystoneBasic.create_user
-
KeystoneBasic.create_tenant
-
KeystoneBasic.create_and_list_users
-
KeystoneBasic.create_tenant_with_users
3.3.4.2. Image - Glance ¶
Glance API is covered in the OpenStack Gates via Tempest as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*image.v1 |
API v1 |
.*im age.v2.admin.test_images.ImportCopyImagesTest |
import_image |
.*image.v2.test_images_negative.ImagesNeg ativeTest.test_create_image_reserved_property |
os_glance_reserved |
.*image.v2.test_images_negative.ImagesNeg ativeTest.test_update_image_reserved_property |
os_glance_reserved |
.*image.v2 .test_images_negative.ImportImagesNegativeTes t.test_image_web_download_import_with_bad_url |
web-download import |
.*image.v2.test_images.ImportImagesTest |
import_image |
. * image.v2.test_images.MultiStoresImportImages |
import_image |
Glance API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.validate_glance
-
GlanceImages.create_and_delete_image
-
GlanceImages.create_and_list_image
-
GlanceImages.list_images
-
GlanceImages.create_image_and_boot_instances
3.3.4.3. Block Storage - Cinder ¶
Cinder API is covered in the OpenStack Gates via Tempest and cinder-tempest-plugin as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*test_incremental_backup |
|
.*test_consistencygroups |
consistency_group |
.*test_ba ckup_crossproject_admin_negative |
|
.*test_b ackup_crossproject_user_negative |
|
.*test_volume_encr ypted.TestEncryptedCinderVolumes |
attach_encrypted_volume |
.*test_encrypted_volumes_extend |
extend_attached_encrypted_volume |
.*test_group_s napshots.GroupSnapshotsV319Test. test_reset_group_snapshot_status |
|
.*test_multi_backend |
multi-backend |
.*test_volume_rety pe.VolumeRetypeWithMigrationTest |
multi-backend |
.*test_volume_delete_casca de.VolumesDeleteCascade.test_vol ume_from_snapshot_cascade_delete |
|
.*t est_volumes_backup.VolumesBackup sTest.test_volume_backup_create_ get_detailed_list_restore_delete |
ceph |
.*test_volu mes_extend.VolumesExtendAttached Test.test_extend_attached_volume |
extend_attached_volume |
.*tempest.scena rio.test_volume_migrate_attached |
multi-backend |
Cinder API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.validate_cinder
-
CinderVolumes.create_and_delete_snapshot
-
CinderVolumes.create_and_delete_volume
-
CinderVolumes.create_and_extend_volume
-
CinderVolumes.create_from_volume_and_delete_volume
-
CinderQos.create_and_list_qos
-
CinderQos.create_and_set_qos
-
CinderVolumeTypes.create_and_list_volume_types
-
CinderVolumeTypes.create_volume_type_and_encryption_type
-
Quotas.cinder_update_and_delete
-
Quotas.cinder_update
3.3.4.4. Object Storage - Swift ¶
Swift API is covered in the OpenStack Gates via Tempest as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*test _container_sync.ContainerSyncTes t.test_container_synchronization |
|
.*test_container_sync_middl eware.ContainerSyncMiddlewareTes t.test_container_synchronization |
container_sync |
.*test_obje ct_services.ObjectTest.test_crea te_object_with_transfer_encoding |
Swift API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
SwiftObjects.create_container_and_object_then_list_objects
-
SwiftObjects.list_objects_in_containers
-
SwiftObjects.create_container_and_object_then_download_object
-
SwiftObjects.create_container_and_object_then_delete_all
-
SwiftObjects.list_and_download_objects_in_containers
3.3.4.5. Networking - Neutron ¶
Neutron API is covered in the OpenStack Gates via Tempest and neutron-tempest-plugin as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*ad min.test_agent_availability_zone |
DHCP agent and L3 agent |
. * admin.test_dhcp_agent_scheduler |
dhcp_agent_scheduler |
.*admin.test_l3_agent_scheduler |
l3_agent_scheduler |
.*admin.test_logging |
logging |
.*admin.test_logging_negative |
logging |
.* admin.test_network_segment_range |
network-segment-range |
. * admin.test_ports.PortTestCasesA dmin.test_regenerate_mac_address |
port-mac-address-regenerate |
.*admin.test_po rts.PortTestCasesResourceRequest |
port-resource-request |
.*admin.test_routers_dvr |
dvr |
.*admin.test_routers_flavors |
l3-flavors |
.*admin.test_routers_ha |
l3-ha |
.*test_flo ating_ips.FloatingIPPoolTestJSON |
floatingip-pools |
.*test_floating_i ps.FloatingIPTestJSON.test_creat e_update_floatingip_port_details |
fip-port-details |
.*test_metering_extensions |
metering |
.*test_metering_negative |
metering |
.*test_ networks.NetworksSearchCriteriaT est.test_list_validation_filters |
filter-validation |
.*t est_networks.NetworksTestAdmin.t est_create_tenant_network_vxlan. |
vxlan |
.*test _networks.NetworksTestJSON.test_ create_update_network_dns_domain |
dns-integration |
.*test_port_forwardings |
floating-ip-port-forwarding |
.*test_port_forwarding_negative |
floating-ip-port-forwarding |
.*t est_ports.PortsTaggingOnCreation |
tag-ports-during-bulk-creation |
.*test_p orts.PortsTestJSON.test_create_p ort_with_propagate_uplink_status |
uplink-status-propagation |
.*test_port s.PortsTestJSON.test_create_port _without_propagate_uplink_status |
uplink-status-propagation |
.* test_ports.PortsTestJSON.test_cr eate_update_port_with_dns_domain |
dns-domain-ports |
.*test_ports.PortsTestJSON.test_ create_update_port_with_dns_name |
dns-integration |
.*t est_ports.PortsTestJSON.test_cre ate_update_port_with_no_dns_name |
dns-integration |
.*te st_revisions.TestRevisions.test_ update_dns_domain_bumps_revision |
dns-integration |
.*test_revisions. TestRevisions.test_update_router _extra_attributes_bumps_revision |
l3-ha |
.*test_router_interface_fip |
router-interface-fip |
.*test_routers.DvrRoutersTest |
dvr |
.*test_routers.HaRoutersTest |
l3-ha |
.*test_routers.RoutersI pV6Test.test_extra_routes_atomic |
extraroute-atomic |
.*test_routers.Rout ersTest.test_extra_routes_atomic |
extraroute-atomic |
.*test_routers _negative.DvrRoutersNegativeTest |
dvr |
.*test_routers_negativ e.DvrRoutersNegativeTestExtended |
dvr |
.*test_router s_negative.HaRoutersNegativeTest |
l3-ha |
.*test_security_gr oups.RbacSharedSecurityGroupTest |
rbac-security-groups |
.*test_subnetpool_prefix_ops |
subnetpool-prefix-ops |
.*tes t_subnetpools.RbacSubnetPoolTest |
rbac-subnetpool |
.*test_subnetpools_n egative.SubnetPoolsNegativeTestJ SON.test_tenant_create_subnetpoo l_associate_shared_address_scope |
rbac-address-scope |
.*test_subnet pools.SubnetPoolsSearchCriteriaT est.test_list_validation_filters |
filter-validation |
.*tes t_subnets.SubnetsSearchCriteriaT est.test_list_validation_filters |
filter-validation |
.*test_timestamp.TestTimeS tamp.test_segment_with_timestamp |
standard-attr-segment |
.*test_trunk.TrunkTest InheritJSONBase.test_add_subport |
|
.*test_trunk.TrunkTestMtusJSON |
vxlan |
. * test_trunk_negative.TrunkTestJS ON.test_create_subport_invalid_i nherit_network_segmentation_type |
vxlan |
.*test_ trunk_negative.TrunkTestMtusJSON |
vxlan |
.*test_qos .QosMinimumBandwidthRuleTestJSON |
|
.*network.test_tags |
tag-ext |
.*test_rout ers.RoutersIpV6Test.test_create_ router_set_gateway_with_fixed_ip |
|
.*test_ routers.RoutersTest.test_create_ router_set_gateway_with_fixed_ip |
|
.*tes t_network_basic_ops.TestNetworkB asicOps.test_router_rescheduling |
l3_agent_scheduler |
.*test_network _advanced_server_ops.TestNetwork AdvancedServerOps.test_server_co nnectivity_cold_migration_revert |
Neutron API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.validate_neutron
-
NeutronNetworks.create_and_delete_networks
-
NeutronNetworks.create_and_delete_ports
-
NeutronNetworks.create_and_delete_routers
-
NeutronNetworks.create_and_delete_subnets
-
NeutronNetworks.create_and_list_networks
-
NeutronNetworks.create_and_list_ports
-
NeutronNetworks.create_and_list_routers
-
NeutronNetworks.create_and_list_subnets
-
NeutronSecurityGroup.create_and_delete_security_groups
-
NeutronSecurityGroup.create_and_delete_security_group_rule
-
NeutronNetworks.set_and_clear_router_gateway
-
Quotas.neutron_update
3.3.4.6. Compute - Nova ¶
Nova API is covered in the OpenStack Gates via Tempest as integrated in Functest Smoke CNTT .
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*admin.test_agents |
xenapi_apis |
.*test_fixed_ips |
neutron |
.*test_fixed_ips_negative |
neutron |
.*test_auto_allocate_network |
shared networks |
.*test_flavors_m icroversions.FlavorsV255TestJSON |
max_microversion: 2.53 |
.*test_flavors_m icroversions.FlavorsV261TestJSON |
max_microversion: 2.53 |
.*test_floating_ips_bulk |
nova-network |
.*test _live_migration.LiveAutoBlockMig rationV225Test.test_iscsi_volume |
block live migration |
.*test_live_mi gration.LiveAutoBlockMigrationV2 25Test.test_live_block_migration |
block live migration |
.*test_live_migration .LiveAutoBlockMigrationV225Test. test_live_block_migration_paused |
block live migration |
.*test_live_migration. LiveAutoBlockMigrationV225Test.t est_volume_backed_live_migration |
volume-backed live migration |
.*test_live_migration.Liv eMigrationTest.test_iscsi_volume |
block live migration |
. * test_live_migration.LiveMigrati onTest.test_live_block_migration |
block live migration |
.*test_l ive_migration.LiveMigrationTest. test_live_block_migration_paused |
block live migration |
.*test_li ve_migration.LiveMigrationTest.t est_volume_backed_live_migration |
volume-backed live migration |
.*test_live_migration.Li veMigrationRemoteConsolesV26Test |
serial_console |
. * test_quotas.QuotasAdminTestV257 |
max_microversion: 2.53 |
.*test_ser vers.ServersAdminTestJSON.test_r eset_network_inject_network_info |
xenapi_apis |
.*certificates.test_certificates |
cert |
.*test_quotas_negative.Quot asSecurityGroupAdminNegativeTest |
|
.*test_novnc |
vnc_console |
.*test_server_personality |
personality |
.*test _servers.ServerShowV263Test.test _show_update_rebuild_list_server |
certified_image_ref |
.*test_servers_ microversions.ServerShowV254Test |
max_microversion: 2.53 |
.*test_servers_ microversions.ServerShowV257Test |
max_microversion: 2.53 |
.*test_servers_negative.Se rversNegativeTestJSON.test_perso nality_file_contents_not_encoded |
personality |
.*test_ server_actions.ServerActionsTest JSON.test_change_server_password |
change_password |
.*test_server_actions.ServerActi onsTestJSON.test_get_vnc_console |
vnc_console |
.*t est_server_actions.ServerActions TestJSON.test_reboot_server_soft |
|
.*test_server_rescue.Serv erBootFromVolumeStableRescueTest |
stable_rescue |
.*test_server_res cue.ServerStableDeviceRescueTest |
stable_rescue |
.*t est_security_group_default_rules |
|
.*test_securi ty_groups_negative.SecurityGroup sNegativeTestJSON.test_security_ group_create_with_duplicate_name |
neutron |
.*test_security_groups_n egative.SecurityGroupsNegativeTe stJSON.test_security_group_creat e_with_invalid_group_description |
|
.*test_security_g roups_negative.SecurityGroupsNeg ativeTestJSON.test_security_grou p_create_with_invalid_group_name |
|
.*test_securi ty_groups_negative.SecurityGroup sNegativeTestJSON.test_update_se curity_group_with_invalid_sg_des |
neutron |
.*test_secur ity_groups_negative.SecurityGrou psNegativeTestJSON.test_update_s ecurity_group_with_invalid_sg_id |
neutron |
.*test_securit y_groups_negative.SecurityGroups NegativeTestJSON.test_update_sec urity_group_with_invalid_sg_name |
neutron |
.*test_server _metadata.ServerMetadataTestJSON |
xenapi_apis |
.*test_ server_metadata_negative.ServerM etadataNegativeTestJSON.test_del ete_metadata_non_existent_server |
xenapi_apis |
.*test_server_metadata_n egative.ServerMetadataNegativeTe stJSON.test_metadata_items_limit |
xenapi_apis |
.*test_server_metadata_negat ive.ServerMetadataNegativeTestJS ON.test_set_metadata_invalid_key |
xenapi_apis |
.*te st_server_metadata_negative.Serv erMetadataNegativeTestJSON.test_ set_metadata_non_existent_server |
xenapi_apis |
. * test_server_metadata_negative.S erverMetadataNegativeTestJSON.te st_set_server_metadata_blank_key |
xenapi_apis |
.*test_s erver_metadata_negative.ServerMe tadataNegativeTestJSON.test_set_ server_metadata_missing_metadata |
xenapi_apis |
.*test_ server_metadata_negative.ServerM etadataNegativeTestJSON.test_upd ate_metadata_non_existent_server |
xenapi_apis |
.* test_server_metadata_negative.Se rverMetadataNegativeTestJSON.tes t_update_metadata_with_blank_key |
xenapi_apis |
.*test_list_server_filters. ListServerFiltersTestJSON.test_l ist_servers_filtered_by_ip_regex |
|
. * servers.test_virtual_interfaces |
nova-network |
.*compute. test_virtual_interfaces_negative |
nova-network |
.*compute.test_networks |
nova-network |
.*test_attac h_volume.AttachVolumeMultiAttach |
volume_multiattach |
.*test_volume_boot_pattern.T estVolumeBootPattern.test_boot_s erver_from_encrypted_volume_luks |
attach_encrypted_volume |
.*test_volume_swap |
swap_volume |
.*test_encrypted_cinder_volumes |
attach_encrypted_volume |
. * test_minbw_allocation_placement |
microversion |
.*tes t_volumes_negative.UpdateMultiat tachVolumeNegativeTest.test_mult iattach_rw_volume_update_failure |
volume_multiattach |
.*test_shelve_ instance.TestShelveInstance.test _cold_migrate_unshelved_instance |
shelve_migrate |
Nova API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.validate_nova
-
NovaServers.boot_and_live_migrate_server
-
NovaServers.boot_server_attach_created_volume_and_live_migrate
-
NovaServers.boot_server_from_volume_and_live_migrate
-
NovaKeypair.boot_and_delete_server_with_keypair
-
NovaServers.boot_server_from_volume_and_delete
-
NovaServers.pause_and_unpause_server
-
NovaServers.boot_and_migrate_server
-
NovaServers.boot_server_and_list_interfaces
-
NovaServers.boot_server_associate_and_dissociate_floating_ip
-
NovaServerGroups.create_and_delete_server_group
-
Quotas.nova_update
3.3.4.7. Orchestration - Heat ¶
Heat API is covered in the OpenStack Gates via heat-tempest-plugin as integrated in Functest Smoke CNTT
According to RA1 Core OpenStack Services APIs the following test names must not be executed:
test rejection regular expressions |
reasons |
---|---|
.*functional.test_lbaasv2 |
lbaasv2 |
.*fun ctional.test_encryption_vol_type |
|
.*RemoteStackTest.test_st ack_create_with_cloud_credential |
|
.*scenario.test_aodh_alarm |
aodh |
.*te sts.scenario.test_autoscaling_lb |
lbaas |
.*scenario.test_autoscaling_lbv2 |
lbaasv2 |
.*scen ario.test_server_software_config |
|
.*test_volumes.Vo lumeBackupRestoreIntegrationTest |
|
.*scenario.test_octavia_lbaas |
octavia |
.*scenario.test_server_cfn_init |
Heat API is also covered by Rally .
Here are the mainline tasks integrated in Functest Smoke CNTT :
-
Authenticate.validate_heat
-
HeatStacks.create_update_delete_stack
-
HeatStacks.create_check_delete_stack
-
HeatStacks.create_suspend_resume_delete_stack
-
HeatStacks.list_stacks_and_resources
3.3.5. Dashboard ¶
Horizon is covered in the OpenStack Gates via tempest-horizon as integrated in Functest Healthcheck .
3.3.6. OpenStack API benchmarking ¶
Rally is tool and framework that allows to perform OpenStack API benchmarking.
Here are the Rally-based test cases proposed by Functest Benchmarking CNTT
-
rally_full : Functest scenarios iterating 10 times the mainline Rally scenarios
-
rally_jobs : Neutron scenarios executed in the OpenStack gates
At the time of writing, no KPI is defined in RA1 Core OpenStack Services APIs which would have asked for an update of the default SLA (maximum failure rate of 0%) proposed in Functest Benchmarking CNTT
3.3.6.1. Identity - Keystone ¶
Scenarios |
Iterations |
---|---|
Authenticate.keystone |
10 |
KeystoneBasic.add_and_remove_user_role |
10 |
KeystoneBasic.create_add_and_list_user_roles |
10 |
KeystoneBasic.create_and_list_tenants |
10 |
KeystoneBasic.create_and_delete_role |
10 |
KeystoneBasic.create_and_delete_service |
10 |
KeystoneBasic.get_entities |
10 |
KeystoneBasic.create_update_and_delete_tenant |
10 |
KeystoneBasic.create_user |
10 |
KeystoneBasic.create_tenant |
10 |
KeystoneBasic.create_and_list_users |
10 |
KeystoneBasic.create_tenant_with_users |
10 |
3.3.6.2. Image - Glance ¶
Scenarios |
Iterations |
---|---|
Authenticate.validate_glance |
10 |
GlanceImages.create_and_delete_image |
10 |
GlanceImages.create_and_list_image |
10 |
GlanceImages.list_images |
10 |
GlanceImages.create_image_and_boot_instances |
10 |
GlanceImages.create_and_deactivate_image |
10 |
GlanceImages.create_and_download_image |
10 |
GlanceImages.create_and_get_image |
10 |
GlanceImages.create_and_update_image |
10 |
3.3.6.3. Block Storage - Cinder ¶
Scenarios |
Iterations |
---|---|
Authenticate.validate_glance |
10 |
CinderVolumes.create_and_attach_volume |
10 |
CinderVolumes.create_and_list_snapshots |
10 |
CinderVolumes.create_and_list_volume |
10 |
CinderVolumes.create_and_upload_volume_to_image |
10 |
CinderVolumes.create_nested_snapshots_and_attach_volume |
10 |
CinderVolumes.create_snapshot_and_attach_volume |
10 |
CinderVolumes.create_volume |
10 |
CinderVolumes.list_volumes |
10 |
CinderVolumes.create_and_delete_snapshot |
10 |
CinderVolumes.create_and_delete_volume |
10 |
CinderVolumes.create_and_extend_volume |
10 |
CinderVolumes.create_from_volume_and_delete_volume |
10 |
CinderQos.create_and_get_qos |
10 |
CinderQos.create_and_list_qos |
10 |
CinderQos.create_and_set_qos |
10 |
CinderVolumeTypes.create_and_get_volume_type |
10 |
CinderVolumeTypes.create_and_list_volume_types |
10 |
CinderVolumeTypes.create_and_update_volume_type |
10 |
CinderVolumeTypes.create_volume_type_and_encryption_type |
10 |
CinderVolumeTypes.create_volume_type_add_and_list_type_access |
10 |
Quotas.cinder_update_and_delete |
10 |
Quotas.cinder_update |
10 |
3.3.6.4. Object Storage - Swift ¶
Scenarios |
Iterations |
---|---|
SwiftObjects.create_container_and_object_then_list_objects |
10 |
SwiftObjects.list_objects_in_containers |
10 |
SwiftObjects.create_container_and_object_then_download_object |
10 |
SwiftObjects.create_container_and_object_then_delete_all |
10 |
SwiftObjects.list_and_download_objects_in_containers |
10 |
3.3.6.5. Networking - Neutron ¶
Scenarios |
Iterations |
---|---|
Authenticate.validate_neutron |
10 |
NeutronNetworks.create_and_update_networks |
10 |
NeutronNetworks.create_and_update_ports |
10 |
NeutronNetworks.create_and_update_routers |
10 |
NeutronNetworks.create_and_update_subnets |
10 |
NeutronNetworks.create_and_delete_networks |
10 |
NeutronNetworks.create_and_delete_ports |
10 |
NeutronNetworks.create_and_delete_routers |
10 |
NeutronNetworks.create_and_delete_subnets |
10 |
NeutronNetworks.create_and_list_networks |
10 |
NeutronNetworks.create_and_list_ports |
10 |
NeutronNetworks.create_and_list_routers |
10 |
NeutronNetworks.create_and_list_subnets |
10 |
NeutronSecurityGroup.create_and_delete_security_groups |
10 |
NeutronSecurityGroup.create_and_delete_security_group_rule |
10 |
NeutronSecurityGroup.create_and_list_security_group_rules |
10 |
NeutronSecurityGroup.create_and_show_security_group |
10 |
NeutronNetworks.set_and_clear_router_gateway |
10 |
NeutronNetworks.create_and_show_ports |
10 |
NeutronNetworks.create_and_show_routers |
10 |
NeutronNetworks.create_and_show_subnets |
10 |
Quotas.neutron_update |
10 |
Scenarios |
Iterations |
---|---|
NeutronNetworks.create_and_delete_networks |
40 |
NeutronNetworks.create_and_delete_ports |
40 |
NeutronNetworks.create_and_delete_routers |
40 |
NeutronNetworks.create_and_delete_subnets |
40 |
NeutronNetworks.create_and_list_networks |
100 |
NeutronNetworks.create_and_list_ports |
8 |
NeutronNetworks.create_and_list_routers |
40 |
NeutronNetworks.create_and_list_subnets |
40 |
NeutronNetworks.create_and_update_networks |
40 |
NeutronNetworks.create_and_update_ports |
40 |
NeutronNetworks.create_and_update_routers |
40 |
NeutronNetworks.create_and_update_subnets |
100 |
NeutronTrunks.create_and_list_trunks |
4 |
Quotas.neutron_update |
40 |
3.3.6.6. Compute - Nova ¶
Scenarios |
Iterations |
---|---|
Authenticate.validate_nova |
10 |
NovaKeypair.create_and_delete_keypair |
10 |
NovaKeypair.create_and_list_keypairs |
10 |
NovaServers.boot_and_bounce_server |
10 |
NovaServers.boot_and_delete_server |
10 |
NovaServers.boot_and_list_server |
10 |
NovaServers.boot_and_rebuild_server |
10 |
NovaServers.snapshot_server |
10 |
NovaServers.boot_server_from_volume |
10 |
NovaServers.boot_server |
10 |
NovaServers.list_servers |
10 |
NovaServers.resize_server |
10 |
NovaServers.boot_and_live_migrate_server |
10 |
NovaServers.boot_server_attach_created_volume_and_live_migrate |
10 |
NovaServers.boot_server_from_volume_and_live_migrate |
10 |
NovaKeypair.boot_and_delete_server_with_keypair |
10 |
NovaServers.boot_server_from_volume_and_delete |
10 |
NovaServers.pause_and_unpause_server |
10 |
NovaServers.boot_and_migrate_server |
10 |
NovaServers.boot_server_and_list_interfaces |
10 |
NovaServers.boot_and_get_console_url |
10 |
NovaServers.boot_server_and_attach_interface |
10 |
NovaServers.boot_server_attach_volume_and_list_attachments |
10 |
NovaServers.boot_server_associate_and_dissociate_floating_ip |
10 |
NovaServers.boot_and_associate_floating_ip |
10 |
NovaServerGroups.create_and_delete_server_group |
10 |
NovaServerGroups.create_and_get_server_group |
10 |
NovaServerGroups.create_and_list_server_groups |
10 |
Quotas.nova_update |
10 |
3.3.6.7. Orchestration - Heat ¶
Scenarios |
Iterations |
---|---|
Authenticate.validate_heat |
10 |
HeatStacks.create_and_delete_stack |
10 |
HeatStacks.create_and_list_stack |
10 |
HeatStacks.create_update_delete_stack |
10 |
HeatStacks.create_check_delete_stack |
10 |
HeatStacks.create_suspend_resume_delete_stack |
10 |
HeatStacks.list_stacks_and_resources |
10 |
3.3.7. Dataplane benchmarking ¶
Functest Benchmarking CNTT offers two benchmarking dataplane test cases leveraging on:
VMTP is a small python application that will automatically perform ping connectivity, round trip time measurement (latency) and TCP/UDP throughput measurement on any OpenStack deployment.
Shaker wraps around popular system network testing tools like iperf, iperf3 and netperf (with help of flent). Shaker is able to deploy OpenStack instances and networks in different topologies. Shaker scenario specifies the deployment and list of tests to execute.
At the time of writing, no KPIs are defined in Anuket specifications which would have asked for an update of the default SLA proposed in Functest Benchmarking CNTT
On top of this dataplane benchmarking described in VMTP & Shaker, we need to integrate testing as described in ETSI GS NFV-TST 009: Specification of Networking Benchmarks and Measurement Methods for NFVI . This type of testing is better suited to measure the networking capabilities of a compute node. The rapid scripts in conjunction with the PROX tool offers an open source implementation for this type of testing.
3.3.8. VMTP ¶
Here are the scenarios executed by Functest vmtp :
-
VM to VM same network fixed IP (intra-node)
-
VM to VM different network fixed IP (intra-node)
-
VM to VM different network floating IP (intra-node)
-
VM to VM same network fixed IP (inter-node)
-
VM to VM different network fixed IP (inter-node)
-
VM to VM different network floating IP (inter-node)
Here are all results per scenario:
protocol |
pkt_size |
results |
---|---|---|
ICMP |
64 |
rtt_avg_ms |
ICMP |
64 |
rtt_max_ms |
ICMP |
64 |
rtt_min_ms |
ICMP |
64 |
rtt_stddev |
ICMP |
391 |
rtt_avg_ms |
ICMP |
391 |
rtt_max_ms |
ICMP |
391 |
rtt_min_ms |
ICMP |
391 |
rtt_stddev |
ICMP |
1500 |
rtt_avg_ms |
ICMP |
1500 |
rtt_max_ms |
ICMP |
1500 |
rtt_min_ms |
ICMP |
1500 |
rtt_stddev |
UDP |
128 |
loss_rate |
UDP |
128 |
throughput_kbps |
UDP |
1024 |
loss_rate |
UDP |
1024 |
throughput_kbps |
UDP |
8192 |
loss_rate |
UDP |
8192 |
throughput_kbps |
TCP |
65536 |
rtt_ms |
TCP |
65536 |
throughput_kbps |
3.3.9. Shaker ¶
Here are the scenarios executed by Shaker:
-
OpenStack L2
-
OpenStack L3 East-West
-
OpenStack L3 North-South
-
OpenStack L3 North-South Performance
Here are all samples:
test |
samples |
---|---|
Bi-directional |
ping_icmp (ms) |
Bi-directional |
tcp_download (Mbits/s) |
Bi-directional |
tcp_upload (Mbits/s) |
Download |
ping_icmp (ms) |
Download |
tcp_download (Mbits/s) |
Upload |
ping_icmp (ms) |
Upload |
tcp_upload (Mbits/s) |
Ping |
ping_icmp (ms) |
Ping |
ping_udp (ms) |
TCP |
bandwidth (bit/s) |
TCP |
retransmits |
UDP |
packets (pps) |
3.3.10. PROX ¶
The generator used with the rapid scripts is PROX with a specific generator configuration file. When multiple flows are requested, the generator starts randomizing bits in the source and destination UDP ports. The number of flows to be generated during each run of the test is specified in the test files (e.g. TST009_Throughput.test). Packet size used during the test is also defined in the test file. IMIX is not supported yet, but you could take the average packet size of the IMIX for now. When defining n packet sizes with m different flow sizes, the test will run n x m times and will produce the results for these n x m combinations. All throughput benchmarking is done by a generator sending packets to a reflector. This results in bidirectional traffic which should be identical (src and dest IP and ports swapped) if all traffic goes through. The VMs or containers use only 1 vNIC for incoming and outgoing traffic. Multiple queues can be used. Multiple VMs or containers can be deployed prior to running any tests. This allows to use generator-reflector pairs on the same or different compute nodes, on the same or different NUMA nodes.
3.3.11. Opensource VNF onboarding and testing ¶
Running opensource VNFs is a key technical solution to ensure that the platforms meet Network Functions Virtualization requirements. Functest VNF offers 5 test cases which automatically onboard and test the following 3 opensource VNFs:
Here are the full list of orchestrators used for all these deployments:
The VNF are covered by upstream tests when possible (see clearwater-live-test ) and by Functest VNF tests in the other cases.
3.3.12. Tenants ¶
3.3.13. LCM ¶
3.3.14. Assurance ¶
3.3.15. Security ¶
3.3.16. Resilience ¶
3.3.17. Bare-metal validations ¶
3.4. Test Cases Traceability to Requirements ¶
3.4.1. RM/RA-1 Requirements ¶
According to RC1 Chapter04 the following test cases must pass as they are for Anuket NFVI Conformance:
container |
test case |
criteria |
---|---|---|
opnfv/functest-healthcheck:wallaby |
tempest_horizon |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_neutron_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_cinder_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_keystone_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
rally_sanity_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_full_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_scenario_cntt |
PASS |
opnfv/functest-smoke-cntt:wallaby |
tempest_slow_cntt |
PASS |
opnfv/functest-benchmarking-cntt:wallaby |
rally_full_cntt |
PASS |
opnfv/functest-benchmarking-cntt:wallaby |
rally_jobs_cntt |
PASS |
opnfv/functest-benchmarking-cntt:wallaby |
vmtp |
PASS |
opnfv/functest-benchmarking-cntt:wallaby |
shaker |
PASS |
opnfv/functest-vnf:wallaby |
cloudify |
PASS |
opnfv/functest-vnf:wallaby |
cloudify_ims |
PASS |
opnfv/functest-vnf:wallaby |
heat_ims |
PASS |
opnfv/functest-vnf:wallaby |
vyos_vrouter |
PASS |
opnfv/functest-vnf:wallaby |
juju_epc |
PASS |
3.4.2. TC Mapping to Requirements ¶
test case |
requirements |
---|---|
tempest_horizon |
Horizon testing |
tempest_neutron_cntt |
Neutron API testing |
tempest_cinder_cntt |
Cinder API testing |
tempest_keystone_cntt |
Keystone API testing |
rally_sanity_cntt |
Keystone, Glance, Cinder, Swift, Neutron, Nova and Heat API testing |
tempest_full_cntt |
Keystone, Glance, Cinder, Swift, Neutron and Nova API testing |
tempest_scenario_cntt |
Keystone, Glance, Cinder, Swift, Neutron and Nova API testing |
tempest_slow_cntt |
Keystone, Glance, Cinder, Swift, Neutron and Nova API testing |
rally_full_cntt |
Keystone, Glance, Cinder, Swift, Neutron, Nova and Heat API benchmarking |
rally_jobs_cntt |
Neutron API benchmarking |
vmtp |
Dataplane benchmarking |
shaker |
Dataplane benchmarking |
cloudify |
opensource VNF onboarding and testing |
cloudify_ims |
opensource VNF onboarding and testing |
heat_ims |
opensource VNF onboarding and testing |
vyos_vrouter |
opensource VNF onboarding and testing |
juju_epc |
opensource VNF onboarding and testing |