.. _3-cloud-infrastructure-test-cases-and-traceability-to-requirements:
Cloud Infrastructure Test Cases and Traceability to Requirements
================================================================
.. raw:: html
![Scope scope](../figures/bogo_ifo.png)
.. _31-introduction:
Introduction
------------
The scope of this chapter is to identify and list down test cases based
on requirements defined in `Reference Architecture-1
(RA-1) <../../../ref_arch/openstack/README.md>`__. This will serve as
traceability between test cases and requirements.
Note that each requirement may have one or more test cases associated
with it.
**must**: Test Cases that are marked as must are considered mandatory
and must pass successfully.
**should**: Test Cases that are marked as should are expected to be
fulfilled by NFVI but it is up to each service provider to accept an
NFVI targeting reference architecture that is not reflecting on any of
those requirements. The same applies to should not.
**may**: Test cases that are marked as may are considered optional. The
same applies to may not.
.. _32-selection-criteria:
Selection Criteria
------------------
Test cases below are selected based on available test cases in
open-source tools like FuncTest etc.
.. _33-traceability-matrix:
Traceability Matrix
-------------------
The following is a Requirements Traceability Matrix (RTM) mapping Test
Case, and/or Test Case Coverage, to RM and RA-1 requirements (config and
deployment).
The RTM contains RM config (i.e. .conf) requirements listed “per
profile”, followed by RA-1 requirements. Requirements fall into 8
domains: general(gen), infrastructure(inf), VIM(vim), Interface &
API(int), Tenants(tnt), LCM(lcm), Assurance(asr), Security(sec).
For detailed information on RM & RA-1 NFVI and VNF requirements, please
refer to `RI-1 Chapter
3 <../../../ref_impl/cntt-ri/chapters/chapter03.md>`__.
.. _331-architecture-and-openstack-requirements:
Architecture and OpenStack Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _332-infrastructure:
Infrastructure
~~~~~~~~~~~~~~
.. _333-vim:
VIM
~~~
.. _334-interfaces--apis:
Interfaces & APIs
~~~~~~~~~~~~~~~~~
The `OpenStack Gates `__
verify all changes proposed mostly by running thousands of Tempest tests
completed by Rally scenarios in a few cases. Skipping tests is allowed
in all OpenStack Gates and only failures rate the review -1 because of
the multiple capabilities and backends selected in the different Gate
jobs. The classical `Functest
containers `__
conform to this model which also fits the heterogeneous user
deployments.
From a Anuket Compliance state point, the capabilities are well
described in `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ which allows
tuning the test configurations and the test lists to avoid skipping any
test. It results that all tests covering optional capabilities and all
upstream skipped tests due to known bugs are not executed. All remaining
tests must be executed and must pass successfully.
New `Functest
containers `__ have
been proposed for Anuket Compliance which simply override the default
test configurations and the default test lists. Any optional capability
or services (e.g. Barbican) can be still verified by the classical
Functest containers.
The next subsections only detail the Tempest tests which must not be
executed from a Compliance state point. The remaining tests have to pass
successfully. They cover all together the API testing requirements as
asked by `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__
The following software versions are considered here to verify OpenStack
Wallaby selected by Anuket:
======================= =======
software version
======================= =======
Functest wallaby
Cinder Tempest plugin 1.4.0
Keystone Tempest plugin 0.7.0
Heat Tempest plugin 1.2.0
Neutron Tempest plugin 1.4.0
Rally OpenStack 2.1.0
Tempest 27.0.0
======================= =======
.. _3341-identity---keystone:
Identity - Keystone
^^^^^^^^^^^^^^^^^^^
Keystone API is covered in the OpenStack Gates via
`Tempest `__ and
`keystone-tempest-plugin `__
as integrated in `Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+------------------------------------------------------+--------------+
| test rejection regular expressions | reasons |
+======================================================+==============+
| .*api.identity.v3.test_oauth1_tokens | oauth1 |
+------------------------------------------------------+--------------+
| .*scenario.test_federated_authentication | federation |
+------------------------------------------------------+--------------+
| .*identity.admin.v2 | API v2 |
+------------------------------------------------------+--------------+
| .*identity.v2 | API v2 |
+------------------------------------------------------+--------------+
| .*identity.v3.test_access_rules | access_rules |
+------------------------------------------------------+--------------+
| .*identity.v3.t | access_rules |
| est_application_credentials.ApplicationCredentialsV3 | |
| Test.test_create_application_credential_access_rules | |
+------------------------------------------------------+--------------+
Keystone API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.keystone
- KeystoneBasic.add_and_remove_user_role
- KeystoneBasic.create_add_and_list_user_roles
- KeystoneBasic.create_and_list_tenants
- KeystoneBasic.create_and_delete_role
- KeystoneBasic.create_and_delete_service
- KeystoneBasic.get_entities
- KeystoneBasic.create_update_and_delete_tenant
- KeystoneBasic.create_user
- KeystoneBasic.create_tenant
- KeystoneBasic.create_and_list_users
- KeystoneBasic.create_tenant_with_users
.. _3342-image---glance:
Image - Glance
^^^^^^^^^^^^^^
Glance API is covered in the OpenStack Gates via
`Tempest `__ as integrated in
`Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+-----------------------------------------------+---------------------+
| test rejection regular expressions | reasons |
+===============================================+=====================+
| .*image.v1 | API v1 |
+-----------------------------------------------+---------------------+
| .*im | import_image |
| age.v2.admin.test_images.ImportCopyImagesTest | |
+-----------------------------------------------+---------------------+
| .*image.v2.test_images_negative.ImagesNeg | os_glance_reserved |
| ativeTest.test_create_image_reserved_property | |
+-----------------------------------------------+---------------------+
| .*image.v2.test_images_negative.ImagesNeg | os_glance_reserved |
| ativeTest.test_update_image_reserved_property | |
+-----------------------------------------------+---------------------+
| .*image.v2 | web-download import |
| .test_images_negative.ImportImagesNegativeTes | |
| t.test_image_web_download_import_with_bad_url | |
+-----------------------------------------------+---------------------+
| .*image.v2.test_images.ImportImagesTest | import_image |
+-----------------------------------------------+---------------------+
| . | import_image |
| *image.v2.test_images.MultiStoresImportImages | |
+-----------------------------------------------+---------------------+
Glance API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.validate_glance
- GlanceImages.create_and_delete_image
- GlanceImages.create_and_list_image
- GlanceImages.list_images
- GlanceImages.create_image_and_boot_instances
.. _3343-block-storage---cinder:
Block Storage - Cinder
^^^^^^^^^^^^^^^^^^^^^^
Cinder API is covered in the OpenStack Gates via
`Tempest `__ and
`cinder-tempest-plugin `__
as integrated in `Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+----------------------------------+----------------------------------+
| test rejection regular | reasons |
| expressions | |
+==================================+==================================+
| .*test_incremental_backup | `https://gerrit.o |
| | pnfv.org/gerrit/68881 `__ |
+----------------------------------+----------------------------------+
| .*test_consistencygroups | consistency_group |
+----------------------------------+----------------------------------+
| .*test_ba | `https://gerrit.o |
| ckup_crossproject_admin_negative | pnfv.org/gerrit/71011 `__ |
+----------------------------------+----------------------------------+
| .*test_b | `https://gerrit.o |
| ackup_crossproject_user_negative | pnfv.org/gerrit/71011 `__ |
+----------------------------------+----------------------------------+
| .*test_volume_encr | attach_encrypted_volume |
| ypted.TestEncryptedCinderVolumes | |
+----------------------------------+----------------------------------+
| .*test_encrypted_volumes_extend | extend_attached_encrypted_volume |
+----------------------------------+----------------------------------+
| .*test_group_s | `https://la |
| napshots.GroupSnapshotsV319Test. | unchpad.net/bugs/1770179 `__ |
+----------------------------------+----------------------------------+
| .*test_multi_backend | multi-backend |
+----------------------------------+----------------------------------+
| .*test_volume_rety | multi-backend |
| pe.VolumeRetypeWithMigrationTest | |
+----------------------------------+----------------------------------+
| .*test_volume_delete_casca | `https://la |
| de.VolumesDeleteCascade.test_vol | unchpad.net/bugs/1677525 `__ |
+----------------------------------+----------------------------------+
| .*t | ceph |
| est_volumes_backup.VolumesBackup | |
| sTest.test_volume_backup_create_ | |
| get_detailed_list_restore_delete | |
+----------------------------------+----------------------------------+
| .*test_volu | extend_attached_volume |
| mes_extend.VolumesExtendAttached | |
| Test.test_extend_attached_volume | |
+----------------------------------+----------------------------------+
| .*tempest.scena | multi-backend |
| rio.test_volume_migrate_attached | |
+----------------------------------+----------------------------------+
Cinder API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.validate_cinder
- CinderVolumes.create_and_delete_snapshot
- CinderVolumes.create_and_delete_volume
- CinderVolumes.create_and_extend_volume
- CinderVolumes.create_from_volume_and_delete_volume
- CinderQos.create_and_list_qos
- CinderQos.create_and_set_qos
- CinderVolumeTypes.create_and_list_volume_types
- CinderVolumeTypes.create_volume_type_and_encryption_type
- Quotas.cinder_update_and_delete
- Quotas.cinder_update
.. _3344-object-storage---swift:
Object Storage - Swift
^^^^^^^^^^^^^^^^^^^^^^
Swift API is covered in the OpenStack Gates via
`Tempest `__ as integrated in
`Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+----------------------------------+----------------------------------+
| test rejection regular | reasons |
| expressions | |
+==================================+==================================+
| .*test | `https://la |
| _container_sync.ContainerSyncTes | unchpad.net/bugs/1317133 `__ |
+----------------------------------+----------------------------------+
| .*test_container_sync_middl | container_sync |
| eware.ContainerSyncMiddlewareTes | |
| t.test_container_synchronization | |
+----------------------------------+----------------------------------+
| .*test_obje | `https://la |
| ct_services.ObjectTest.test_crea | unchpad.net/bugs/1905432 `__ |
+----------------------------------+----------------------------------+
Swift API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- SwiftObjects.create_container_and_object_then_list_objects
- SwiftObjects.list_objects_in_containers
- SwiftObjects.create_container_and_object_then_download_object
- SwiftObjects.create_container_and_object_then_delete_all
- SwiftObjects.list_and_download_objects_in_containers
.. _3345-networking---neutron:
Networking - Neutron
^^^^^^^^^^^^^^^^^^^^
Neutron API is covered in the OpenStack Gates via
`Tempest `__ and
`neutron-tempest-plugin `__
as integrated in `Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+----------------------------------+----------------------------------+
| test rejection regular | reasons |
| expressions | |
+==================================+==================================+
| .*ad | DHCP agent and L3 agent |
| min.test_agent_availability_zone | |
+----------------------------------+----------------------------------+
| . | dhcp_agent_scheduler |
| *admin.test_dhcp_agent_scheduler | |
+----------------------------------+----------------------------------+
| .*admin.test_l3_agent_scheduler | l3_agent_scheduler |
+----------------------------------+----------------------------------+
| .*admin.test_logging | logging |
+----------------------------------+----------------------------------+
| .*admin.test_logging_negative | logging |
+----------------------------------+----------------------------------+
| .* | network-segment-range |
| admin.test_network_segment_range | |
+----------------------------------+----------------------------------+
| . | port-mac-address-regenerate |
| *admin.test_ports.PortTestCasesA | |
| dmin.test_regenerate_mac_address | |
+----------------------------------+----------------------------------+
| .*admin.test_po | port-resource-request |
| rts.PortTestCasesResourceRequest | |
+----------------------------------+----------------------------------+
| .*admin.test_routers_dvr | dvr |
+----------------------------------+----------------------------------+
| .*admin.test_routers_flavors | l3-flavors |
+----------------------------------+----------------------------------+
| .*admin.test_routers_ha | l3-ha |
+----------------------------------+----------------------------------+
| .*test_flo | floatingip-pools |
| ating_ips.FloatingIPPoolTestJSON | |
+----------------------------------+----------------------------------+
| .*test_floating_i | fip-port-details |
| ps.FloatingIPTestJSON.test_creat | |
| e_update_floatingip_port_details | |
+----------------------------------+----------------------------------+
| .*test_metering_extensions | metering |
+----------------------------------+----------------------------------+
| .*test_metering_negative | metering |
+----------------------------------+----------------------------------+
| .*test_ | filter-validation |
| networks.NetworksSearchCriteriaT | |
| est.test_list_validation_filters | |
+----------------------------------+----------------------------------+
| .*t | vxlan |
| est_networks.NetworksTestAdmin.t | |
| est_create_tenant_network_vxlan. | |
+----------------------------------+----------------------------------+
| .*test | dns-integration |
| _networks.NetworksTestJSON.test_ | |
| create_update_network_dns_domain | |
+----------------------------------+----------------------------------+
| .*test_port_forwardings | floating-ip-port-forwarding |
+----------------------------------+----------------------------------+
| .*test_port_forwarding_negative | floating-ip-port-forwarding |
+----------------------------------+----------------------------------+
| .*t | tag-ports-during-bulk-creation |
| est_ports.PortsTaggingOnCreation | |
+----------------------------------+----------------------------------+
| .*test_p | uplink-status-propagation |
| orts.PortsTestJSON.test_create_p | |
| ort_with_propagate_uplink_status | |
+----------------------------------+----------------------------------+
| .*test_port | uplink-status-propagation |
| s.PortsTestJSON.test_create_port | |
| _without_propagate_uplink_status | |
+----------------------------------+----------------------------------+
| .* | dns-domain-ports |
| test_ports.PortsTestJSON.test_cr | |
| eate_update_port_with_dns_domain | |
+----------------------------------+----------------------------------+
| .*test_ports.PortsTestJSON.test_ | dns-integration |
| create_update_port_with_dns_name | |
+----------------------------------+----------------------------------+
| .*t | dns-integration |
| est_ports.PortsTestJSON.test_cre | |
| ate_update_port_with_no_dns_name | |
+----------------------------------+----------------------------------+
| .*te | dns-integration |
| st_revisions.TestRevisions.test_ | |
| update_dns_domain_bumps_revision | |
+----------------------------------+----------------------------------+
| .*test_revisions. | l3-ha |
| TestRevisions.test_update_router | |
| _extra_attributes_bumps_revision | |
+----------------------------------+----------------------------------+
| .*test_router_interface_fip | router-interface-fip |
+----------------------------------+----------------------------------+
| .*test_routers.DvrRoutersTest | dvr |
+----------------------------------+----------------------------------+
| .*test_routers.HaRoutersTest | l3-ha |
+----------------------------------+----------------------------------+
| .*test_routers.RoutersI | extraroute-atomic |
| pV6Test.test_extra_routes_atomic | |
+----------------------------------+----------------------------------+
| .*test_routers.Rout | extraroute-atomic |
| ersTest.test_extra_routes_atomic | |
+----------------------------------+----------------------------------+
| .*test_routers | dvr |
| _negative.DvrRoutersNegativeTest | |
+----------------------------------+----------------------------------+
| .*test_routers_negativ | dvr |
| e.DvrRoutersNegativeTestExtended | |
+----------------------------------+----------------------------------+
| .*test_router | l3-ha |
| s_negative.HaRoutersNegativeTest | |
+----------------------------------+----------------------------------+
| .*test_security_gr | rbac-security-groups |
| oups.RbacSharedSecurityGroupTest | |
+----------------------------------+----------------------------------+
| .*test_subnetpool_prefix_ops | subnetpool-prefix-ops |
+----------------------------------+----------------------------------+
| .*tes | rbac-subnetpool |
| t_subnetpools.RbacSubnetPoolTest | |
+----------------------------------+----------------------------------+
| .*test_subnetpools_n | rbac-address-scope |
| egative.SubnetPoolsNegativeTestJ | |
| SON.test_tenant_create_subnetpoo | |
| l_associate_shared_address_scope | |
+----------------------------------+----------------------------------+
| .*test_subnet | filter-validation |
| pools.SubnetPoolsSearchCriteriaT | |
| est.test_list_validation_filters | |
+----------------------------------+----------------------------------+
| .*tes | filter-validation |
| t_subnets.SubnetsSearchCriteriaT | |
| est.test_list_validation_filters | |
+----------------------------------+----------------------------------+
| .*test_timestamp.TestTimeS | standard-attr-segment |
| tamp.test_segment_with_timestamp | |
+----------------------------------+----------------------------------+
| .*test_trunk.TrunkTest | `https://la |
| InheritJSONBase.test_add_subport | unchpad.net/bugs/1863707 `__ |
+----------------------------------+----------------------------------+
| .*test_trunk.TrunkTestMtusJSON | vxlan |
+----------------------------------+----------------------------------+
| . | vxlan |
| *test_trunk_negative.TrunkTestJS | |
| ON.test_create_subport_invalid_i | |
| nherit_network_segmentation_type | |
+----------------------------------+----------------------------------+
| .*test_ | vxlan |
| trunk_negative.TrunkTestMtusJSON | |
+----------------------------------+----------------------------------+
| .*test_qos | `https://gerrit.o |
| .QosMinimumBandwidthRuleTestJSON | pnfv.org/gerrit/69105 `__ |
+----------------------------------+----------------------------------+
| .*network.test_tags | tag-ext |
+----------------------------------+----------------------------------+
| .*test_rout | `https://la |
| ers.RoutersIpV6Test.test_create_ | unchpad.net/bugs/1676207 `__ |
+----------------------------------+----------------------------------+
| .*test_ | `https://la |
| routers.RoutersTest.test_create_ | unchpad.net/bugs/1676207 `__ |
+----------------------------------+----------------------------------+
| .*tes | l3_agent_scheduler |
| t_network_basic_ops.TestNetworkB | |
| asicOps.test_router_rescheduling | |
+----------------------------------+----------------------------------+
| .*test_network | `https://la |
| _advanced_server_ops.TestNetwork | unchpad.net/bugs/1836595 `__ |
| nnectivity_cold_migration_revert | |
+----------------------------------+----------------------------------+
Neutron API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.validate_neutron
- NeutronNetworks.create_and_delete_networks
- NeutronNetworks.create_and_delete_ports
- NeutronNetworks.create_and_delete_routers
- NeutronNetworks.create_and_delete_subnets
- NeutronNetworks.create_and_list_networks
- NeutronNetworks.create_and_list_ports
- NeutronNetworks.create_and_list_routers
- NeutronNetworks.create_and_list_subnets
- NeutronSecurityGroup.create_and_delete_security_groups
- NeutronSecurityGroup.create_and_delete_security_group_rule
- NeutronNetworks.set_and_clear_router_gateway
- Quotas.neutron_update
.. _3346-compute---nova:
Compute - Nova
^^^^^^^^^^^^^^
Nova API is covered in the OpenStack Gates via
`Tempest `__ as integrated in
`Functest Smoke
CNTT `__.
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+----------------------------------+----------------------------------+
| test rejection regular | reasons |
| expressions | |
+==================================+==================================+
| .*admin.test_agents | xenapi_apis |
+----------------------------------+----------------------------------+
| .*test_fixed_ips | neutron |
+----------------------------------+----------------------------------+
| .*test_fixed_ips_negative | neutron |
+----------------------------------+----------------------------------+
| .*test_auto_allocate_network | shared networks |
+----------------------------------+----------------------------------+
| .*test_flavors_m | max_microversion: 2.53 |
| icroversions.FlavorsV255TestJSON | |
+----------------------------------+----------------------------------+
| .*test_flavors_m | max_microversion: 2.53 |
| icroversions.FlavorsV261TestJSON | |
+----------------------------------+----------------------------------+
| .*test_floating_ips_bulk | nova-network |
+----------------------------------+----------------------------------+
| .*test | block live migration |
| _live_migration.LiveAutoBlockMig | |
| rationV225Test.test_iscsi_volume | |
+----------------------------------+----------------------------------+
| .*test_live_mi | block live migration |
| gration.LiveAutoBlockMigrationV2 | |
| 25Test.test_live_block_migration | |
+----------------------------------+----------------------------------+
| .*test_live_migration | block live migration |
| .LiveAutoBlockMigrationV225Test. | |
| test_live_block_migration_paused | |
+----------------------------------+----------------------------------+
| .*test_live_migration. | volume-backed live migration |
| LiveAutoBlockMigrationV225Test.t | |
| est_volume_backed_live_migration | |
+----------------------------------+----------------------------------+
| .*test_live_migration.Liv | block live migration |
| eMigrationTest.test_iscsi_volume | |
+----------------------------------+----------------------------------+
| . | block live migration |
| *test_live_migration.LiveMigrati | |
| onTest.test_live_block_migration | |
+----------------------------------+----------------------------------+
| .*test_l | block live migration |
| ive_migration.LiveMigrationTest. | |
| test_live_block_migration_paused | |
+----------------------------------+----------------------------------+
| .*test_li | volume-backed live migration |
| ve_migration.LiveMigrationTest.t | |
| est_volume_backed_live_migration | |
+----------------------------------+----------------------------------+
| .*test_live_migration.Li | serial_console |
| veMigrationRemoteConsolesV26Test | |
+----------------------------------+----------------------------------+
| . | max_microversion: 2.53 |
| *test_quotas.QuotasAdminTestV257 | |
+----------------------------------+----------------------------------+
| .*test_ser | xenapi_apis |
| vers.ServersAdminTestJSON.test_r | |
| eset_network_inject_network_info | |
+----------------------------------+----------------------------------+
| .*certificates.test_certificates | cert |
+----------------------------------+----------------------------------+
| .*test_quotas_negative.Quot | `https://la |
| asSecurityGroupAdminNegativeTest | unchpad.net/bugs/1186354 `__ |
+----------------------------------+----------------------------------+
| .*test_novnc | vnc_console |
+----------------------------------+----------------------------------+
| .*test_server_personality | personality |
+----------------------------------+----------------------------------+
| .*test | certified_image_ref |
| _servers.ServerShowV263Test.test | |
| _show_update_rebuild_list_server | |
+----------------------------------+----------------------------------+
| .*test_servers_ | max_microversion: 2.53 |
| microversions.ServerShowV254Test | |
+----------------------------------+----------------------------------+
| .*test_servers_ | max_microversion: 2.53 |
| microversions.ServerShowV257Test | |
+----------------------------------+----------------------------------+
| .*test_servers_negative.Se | personality |
| rversNegativeTestJSON.test_perso | |
| nality_file_contents_not_encoded | |
+----------------------------------+----------------------------------+
| .*test_ | change_password |
| server_actions.ServerActionsTest | |
| JSON.test_change_server_password | |
+----------------------------------+----------------------------------+
| .*test_server_actions.ServerActi | vnc_console |
| onsTestJSON.test_get_vnc_console | |
+----------------------------------+----------------------------------+
| .*t | `https://la |
| est_server_actions.ServerActions | unchpad.net/bugs/1014647 `__ |
+----------------------------------+----------------------------------+
| .*test_server_rescue.Serv | stable_rescue |
| erBootFromVolumeStableRescueTest | |
+----------------------------------+----------------------------------+
| .*test_server_res | stable_rescue |
| cue.ServerStableDeviceRescueTest | |
+----------------------------------+----------------------------------+
| .*t | `https://la |
| est_security_group_default_rules | unchpad.net/bugs/1311500 `__ |
+----------------------------------+----------------------------------+
| .*test_securi | neutron |
| ty_groups_negative.SecurityGroup | |
| sNegativeTestJSON.test_security_ | |
| group_create_with_duplicate_name | |
+----------------------------------+----------------------------------+
| .*test_security_groups_n | `https://la |
| egative.SecurityGroupsNegativeTe | unchpad.net/bugs/1161411 `__ |
| e_with_invalid_group_description | |
+----------------------------------+----------------------------------+
| .*test_security_g | `https://la |
| roups_negative.SecurityGroupsNeg | unchpad.net/bugs/1161411 `__ |
| p_create_with_invalid_group_name | |
+----------------------------------+----------------------------------+
| .*test_securi | neutron |
| ty_groups_negative.SecurityGroup | |
| sNegativeTestJSON.test_update_se | |
| curity_group_with_invalid_sg_des | |
+----------------------------------+----------------------------------+
| .*test_secur | neutron |
| ity_groups_negative.SecurityGrou | |
| psNegativeTestJSON.test_update_s | |
| ecurity_group_with_invalid_sg_id | |
+----------------------------------+----------------------------------+
| .*test_securit | neutron |
| y_groups_negative.SecurityGroups | |
| NegativeTestJSON.test_update_sec | |
| urity_group_with_invalid_sg_name | |
+----------------------------------+----------------------------------+
| .*test_server | xenapi_apis |
| _metadata.ServerMetadataTestJSON | |
+----------------------------------+----------------------------------+
| .*test_ | xenapi_apis |
| server_metadata_negative.ServerM | |
| etadataNegativeTestJSON.test_del | |
| ete_metadata_non_existent_server | |
+----------------------------------+----------------------------------+
| .*test_server_metadata_n | xenapi_apis |
| egative.ServerMetadataNegativeTe | |
| stJSON.test_metadata_items_limit | |
+----------------------------------+----------------------------------+
| .*test_server_metadata_negat | xenapi_apis |
| ive.ServerMetadataNegativeTestJS | |
| ON.test_set_metadata_invalid_key | |
+----------------------------------+----------------------------------+
| .*te | xenapi_apis |
| st_server_metadata_negative.Serv | |
| erMetadataNegativeTestJSON.test_ | |
| set_metadata_non_existent_server | |
+----------------------------------+----------------------------------+
| . | xenapi_apis |
| *test_server_metadata_negative.S | |
| erverMetadataNegativeTestJSON.te | |
| st_set_server_metadata_blank_key | |
+----------------------------------+----------------------------------+
| .*test_s | xenapi_apis |
| erver_metadata_negative.ServerMe | |
| tadataNegativeTestJSON.test_set_ | |
| server_metadata_missing_metadata | |
+----------------------------------+----------------------------------+
| .*test_ | xenapi_apis |
| server_metadata_negative.ServerM | |
| etadataNegativeTestJSON.test_upd | |
| ate_metadata_non_existent_server | |
+----------------------------------+----------------------------------+
| .* | xenapi_apis |
| test_server_metadata_negative.Se | |
| rverMetadataNegativeTestJSON.tes | |
| t_update_metadata_with_blank_key | |
+----------------------------------+----------------------------------+
| .*test_list_server_filters. | `https://la |
| ListServerFiltersTestJSON.test_l | unchpad.net/bugs/1540645 `__ |
+----------------------------------+----------------------------------+
| . | nova-network |
| *servers.test_virtual_interfaces | |
+----------------------------------+----------------------------------+
| .*compute. | nova-network |
| test_virtual_interfaces_negative | |
+----------------------------------+----------------------------------+
| .*compute.test_networks | nova-network |
+----------------------------------+----------------------------------+
| .*test_attac | volume_multiattach |
| h_volume.AttachVolumeMultiAttach | |
+----------------------------------+----------------------------------+
| .*test_volume_boot_pattern.T | attach_encrypted_volume |
| estVolumeBootPattern.test_boot_s | |
| erver_from_encrypted_volume_luks | |
+----------------------------------+----------------------------------+
| .*test_volume_swap | swap_volume |
+----------------------------------+----------------------------------+
| .*test_encrypted_cinder_volumes | attach_encrypted_volume |
+----------------------------------+----------------------------------+
| . | microversion |
| *test_minbw_allocation_placement | |
+----------------------------------+----------------------------------+
| .*tes | volume_multiattach |
| t_volumes_negative.UpdateMultiat | |
| tachVolumeNegativeTest.test_mult | |
| iattach_rw_volume_update_failure | |
+----------------------------------+----------------------------------+
| .*test_shelve_ | shelve_migrate |
| instance.TestShelveInstance.test | |
| _cold_migrate_unshelved_instance | |
+----------------------------------+----------------------------------+
Nova API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.validate_nova
- NovaServers.boot_and_live_migrate_server
- NovaServers.boot_server_attach_created_volume_and_live_migrate
- NovaServers.boot_server_from_volume_and_live_migrate
- NovaKeypair.boot_and_delete_server_with_keypair
- NovaServers.boot_server_from_volume_and_delete
- NovaServers.pause_and_unpause_server
- NovaServers.boot_and_migrate_server
- NovaServers.boot_server_and_list_interfaces
- NovaServers.boot_server_associate_and_dissociate_floating_ip
- NovaServerGroups.create_and_delete_server_group
- Quotas.nova_update
.. _3347-orchestration---heat:
Orchestration - Heat
^^^^^^^^^^^^^^^^^^^^
Heat API is covered in the OpenStack Gates via
`heat-tempest-plugin `__
as integrated in `Functest Smoke
CNTT `__
According to `RA1 Core OpenStack Services
APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__ the
following test names must not be executed:
+----------------------------------+----------------------------------+
| test rejection regular | reasons |
| expressions | |
+==================================+==================================+
| .*functional.test_lbaasv2 | lbaasv2 |
+----------------------------------+----------------------------------+
| .*fun | `https:// |
| ctional.test_encryption_vol_type | storyboard.openstack.org/#!/stor |
| | y/2007804 `__ |
+----------------------------------+----------------------------------+
| .*RemoteStackTest.test_st | `https://ge |
| ack_create_with_cloud_credential | rrit.opnfv.org/gerrit/c/functest |
| | /+/69926 `__ |
+----------------------------------+----------------------------------+
| .*scenario.test_aodh_alarm | aodh |
+----------------------------------+----------------------------------+
| .*te | lbaas |
| sts.scenario.test_autoscaling_lb | |
+----------------------------------+----------------------------------+
| .*scenario.test_autoscaling_lbv2 | lbaasv2 |
+----------------------------------+----------------------------------+
| .*scen | `https://ge |
| ario.test_server_software_config | rrit.opnfv.org/gerrit/c/functest |
| | /+/69926 `__ |
+----------------------------------+----------------------------------+
| .*test_volumes.Vo | `https://ge |
| lumeBackupRestoreIntegrationTest | rrit.opnfv.org/gerrit/c/functest |
| | /+/69931 `__ |
+----------------------------------+----------------------------------+
| .*scenario.test_octavia_lbaas | octavia |
+----------------------------------+----------------------------------+
| .*scenario.test_server_cfn_init | `https://ge |
| | rrit.opnfv.org/gerrit/c/functest |
| | /+/70004 `__ |
+----------------------------------+----------------------------------+
Heat API is also covered by
`Rally `__.
Here are the mainline tasks integrated in `Functest Smoke
CNTT `__:
- Authenticate.validate_heat
- HeatStacks.create_update_delete_stack
- HeatStacks.create_check_delete_stack
- HeatStacks.create_suspend_resume_delete_stack
- HeatStacks.list_stacks_and_resources
.. _335-dashboard:
Dashboard
~~~~~~~~~
Horizon is covered in the OpenStack Gates via
`tempest-horizon `__ as
integrated in `Functest
Healthcheck `__.
.. _336-openstack-api-benchmarking:
OpenStack API benchmarking
~~~~~~~~~~~~~~~~~~~~~~~~~~
`Rally `__ is tool and framework
that allows to perform OpenStack API benchmarking.
Here are the Rally-based test cases proposed by `Functest Benchmarking
CNTT `__
- `rally_full `__:
Functest scenarios iterating 10 times the mainline Rally scenarios
- `rally_jobs `__:
Neutron scenarios executed in the OpenStack gates
At the time of writing, no KPI is defined in `RA1 Core OpenStack
Services APIs <../../../ref_arch/openstack/chapters/chapter05.md>`__
which would have asked for an update of the default SLA (maximum failure
rate of 0%) proposed in `Functest Benchmarking
CNTT `__
.. _3361-identity---keystone:
Identity - Keystone
^^^^^^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
============================================== ==========
Scenarios Iterations
============================================== ==========
Authenticate.keystone 10
KeystoneBasic.add_and_remove_user_role 10
KeystoneBasic.create_add_and_list_user_roles 10
KeystoneBasic.create_and_list_tenants 10
KeystoneBasic.create_and_delete_role 10
KeystoneBasic.create_and_delete_service 10
KeystoneBasic.get_entities 10
KeystoneBasic.create_update_and_delete_tenant 10
KeystoneBasic.create_user 10
KeystoneBasic.create_tenant 10
KeystoneBasic.create_and_list_users 10
KeystoneBasic.create_tenant_with_users 10
============================================== ==========
.. _3362-image---glance:
Image - Glance
^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
============================================ ==========
Scenarios Iterations
============================================ ==========
Authenticate.validate_glance 10
GlanceImages.create_and_delete_image 10
GlanceImages.create_and_list_image 10
GlanceImages.list_images 10
GlanceImages.create_image_and_boot_instances 10
GlanceImages.create_and_deactivate_image 10
GlanceImages.create_and_download_image 10
GlanceImages.create_and_get_image 10
GlanceImages.create_and_update_image 10
============================================ ==========
.. _3363-block-storage---cinder:
Block Storage - Cinder
^^^^^^^^^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
============================================================= ==========
Scenarios Iterations
============================================================= ==========
Authenticate.validate_glance 10
CinderVolumes.create_and_attach_volume 10
CinderVolumes.create_and_list_snapshots 10
CinderVolumes.create_and_list_volume 10
CinderVolumes.create_and_upload_volume_to_image 10
CinderVolumes.create_nested_snapshots_and_attach_volume 10
CinderVolumes.create_snapshot_and_attach_volume 10
CinderVolumes.create_volume 10
CinderVolumes.list_volumes 10
CinderVolumes.create_and_delete_snapshot 10
CinderVolumes.create_and_delete_volume 10
CinderVolumes.create_and_extend_volume 10
CinderVolumes.create_from_volume_and_delete_volume 10
CinderQos.create_and_get_qos 10
CinderQos.create_and_list_qos 10
CinderQos.create_and_set_qos 10
CinderVolumeTypes.create_and_get_volume_type 10
CinderVolumeTypes.create_and_list_volume_types 10
CinderVolumeTypes.create_and_update_volume_type 10
CinderVolumeTypes.create_volume_type_and_encryption_type 10
CinderVolumeTypes.create_volume_type_add_and_list_type_access 10
Quotas.cinder_update_and_delete 10
Quotas.cinder_update 10
============================================================= ==========
.. _3364-object-storage---swift:
Object Storage - Swift
^^^^^^^^^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
============================================================= ==========
Scenarios Iterations
============================================================= ==========
SwiftObjects.create_container_and_object_then_list_objects 10
SwiftObjects.list_objects_in_containers 10
SwiftObjects.create_container_and_object_then_download_object 10
SwiftObjects.create_container_and_object_then_delete_all 10
SwiftObjects.list_and_download_objects_in_containers 10
============================================================= ==========
.. _3365-networking---neutron:
Networking - Neutron
^^^^^^^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
========================================================== ==========
Scenarios Iterations
========================================================== ==========
Authenticate.validate_neutron 10
NeutronNetworks.create_and_update_networks 10
NeutronNetworks.create_and_update_ports 10
NeutronNetworks.create_and_update_routers 10
NeutronNetworks.create_and_update_subnets 10
NeutronNetworks.create_and_delete_networks 10
NeutronNetworks.create_and_delete_ports 10
NeutronNetworks.create_and_delete_routers 10
NeutronNetworks.create_and_delete_subnets 10
NeutronNetworks.create_and_list_networks 10
NeutronNetworks.create_and_list_ports 10
NeutronNetworks.create_and_list_routers 10
NeutronNetworks.create_and_list_subnets 10
NeutronSecurityGroup.create_and_delete_security_groups 10
NeutronSecurityGroup.create_and_delete_security_group_rule 10
NeutronSecurityGroup.create_and_list_security_group_rules 10
NeutronSecurityGroup.create_and_show_security_group 10
NeutronNetworks.set_and_clear_router_gateway 10
NeutronNetworks.create_and_show_ports 10
NeutronNetworks.create_and_show_routers 10
NeutronNetworks.create_and_show_subnets 10
Quotas.neutron_update 10
========================================================== ==========
`Functest
rally_jobs_cntt `__:
========================================== ==========
Scenarios Iterations
========================================== ==========
NeutronNetworks.create_and_delete_networks 40
NeutronNetworks.create_and_delete_ports 40
NeutronNetworks.create_and_delete_routers 40
NeutronNetworks.create_and_delete_subnets 40
NeutronNetworks.create_and_list_networks 100
NeutronNetworks.create_and_list_ports 8
NeutronNetworks.create_and_list_routers 40
NeutronNetworks.create_and_list_subnets 40
NeutronNetworks.create_and_update_networks 40
NeutronNetworks.create_and_update_ports 40
NeutronNetworks.create_and_update_routers 40
NeutronNetworks.create_and_update_subnets 100
NeutronTrunks.create_and_list_trunks 4
Quotas.neutron_update 40
========================================== ==========
.. _3366-compute---nova:
Compute - Nova
^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
+----------------------------------------------------------------+------------+
| Scenarios | Iterations |
+================================================================+============+
| Authenticate.validate_nova | 10 |
+----------------------------------------------------------------+------------+
| NovaKeypair.create_and_delete_keypair | 10 |
+----------------------------------------------------------------+------------+
| NovaKeypair.create_and_list_keypairs | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_bounce_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_delete_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_list_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_rebuild_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.snapshot_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_from_volume | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.list_servers | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.resize_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_live_migrate_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_attach_created_volume_and_live_migrate | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_from_volume_and_live_migrate | 10 |
+----------------------------------------------------------------+------------+
| NovaKeypair.boot_and_delete_server_with_keypair | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_from_volume_and_delete | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.pause_and_unpause_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_migrate_server | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_and_list_interfaces | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_get_console_url | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_and_attach_interface | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_attach_volume_and_list_attachments | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_server_associate_and_dissociate_floating_ip | 10 |
+----------------------------------------------------------------+------------+
| NovaServers.boot_and_associate_floating_ip | 10 |
+----------------------------------------------------------------+------------+
| NovaServerGroups.create_and_delete_server_group | 10 |
+----------------------------------------------------------------+------------+
| NovaServerGroups.create_and_get_server_group | 10 |
+----------------------------------------------------------------+------------+
| NovaServerGroups.create_and_list_server_groups | 10 |
+----------------------------------------------------------------+------------+
| Quotas.nova_update | 10 |
+----------------------------------------------------------------+------------+
.. _3367-orchestration---heat:
Orchestration - Heat
^^^^^^^^^^^^^^^^^^^^
`Functest
rally_full_cntt `__:
============================================= ==========
Scenarios Iterations
============================================= ==========
Authenticate.validate_heat 10
HeatStacks.create_and_delete_stack 10
HeatStacks.create_and_list_stack 10
HeatStacks.create_update_delete_stack 10
HeatStacks.create_check_delete_stack 10
HeatStacks.create_suspend_resume_delete_stack 10
HeatStacks.list_stacks_and_resources 10
============================================= ==========
.. _337-dataplane-benchmarking:
Dataplane benchmarking
~~~~~~~~~~~~~~~~~~~~~~
`Functest Benchmarking
CNTT `__
offers two benchmarking dataplane test cases leveraging on:
- `VMTP `__
- `Shaker `__
`VMTP `__ is a small python
application that will automatically perform ping connectivity, round
trip time measurement (latency) and TCP/UDP throughput measurement on
any OpenStack deployment.
`Shaker `__ wraps around
popular system network testing tools like iperf, iperf3 and netperf
(with help of flent).
`Shaker `__ is able to deploy
OpenStack instances and networks in different topologies.
`Shaker `__ scenario
specifies the deployment and list of tests to execute.
At the time of writing, no KPIs are defined in Anuket specifications
which would have asked for an update of the default SLA proposed in
`Functest Benchmarking
CNTT `__
On top of this dataplane benchmarking described in VMTP & Shaker, we
need to integrate testing as described in `ETSI GS NFV-TST 009:
Specification of Networking Benchmarks and Measurement Methods for
NFVI `__.
This type of testing is better suited to measure the networking
capabilities of a compute node. The `rapid
scripts `__ in
conjunction with the `PROX
tool `__
offers an open source implementation for this type of testing.
.. _3371-vmtp:
VMTP
~~~~
Here are the
`scenarios `__
executed by `Functest
vmtp `__:
- VM to VM same network fixed IP (intra-node)
- VM to VM different network fixed IP (intra-node)
- VM to VM different network floating IP (intra-node)
- VM to VM same network fixed IP (inter-node)
- VM to VM different network fixed IP (inter-node)
- VM to VM different network floating IP (inter-node)
Here are all results per scenario:
======== ======== ===============
protocol pkt_size results
======== ======== ===============
ICMP 64 rtt_avg_ms
ICMP 64 rtt_max_ms
ICMP 64 rtt_min_ms
ICMP 64 rtt_stddev
ICMP 391 rtt_avg_ms
ICMP 391 rtt_max_ms
ICMP 391 rtt_min_ms
ICMP 391 rtt_stddev
ICMP 1500 rtt_avg_ms
ICMP 1500 rtt_max_ms
ICMP 1500 rtt_min_ms
ICMP 1500 rtt_stddev
UDP 128 loss_rate
UDP 128 throughput_kbps
UDP 1024 loss_rate
UDP 1024 throughput_kbps
UDP 8192 loss_rate
UDP 8192 throughput_kbps
TCP 65536 rtt_ms
TCP 65536 throughput_kbps
======== ======== ===============
.. _3372-shaker:
Shaker
~~~~~~
Here are the
`scenarios `__
executed by Shaker:
- OpenStack L2
- OpenStack L3 East-West
- OpenStack L3 North-South
- OpenStack L3 North-South Performance
Here are all samples:
============== ======================
test samples
============== ======================
Bi-directional ping_icmp (ms)
Bi-directional tcp_download (Mbits/s)
Bi-directional tcp_upload (Mbits/s)
Download ping_icmp (ms)
Download tcp_download (Mbits/s)
Upload ping_icmp (ms)
Upload tcp_upload (Mbits/s)
Ping ping_icmp (ms)
Ping ping_udp (ms)
TCP bandwidth (bit/s)
TCP retransmits
UDP packets (pps)
============== ======================
.. _3373-prox:
PROX
~~~~
The generator used with the rapid scripts is PROX with a specific
generator configuration file. When multiple flows are requested, the
generator starts randomizing bits in the source and destination UDP
ports. The number of flows to be generated during each run of the test
is specified in the test files (e.g. TST009_Throughput.test). Packet
size used during the test is also defined in the test file. IMIX is not
supported yet, but you could take the average packet size of the IMIX
for now. When defining n packet sizes with m different flow sizes, the
test will run n x m times and will produce the results for these n x m
combinations. All throughput benchmarking is done by a generator sending
packets to a reflector. This results in bidirectional traffic which
should be identical (src and dest IP and ports swapped) if all traffic
goes through. The VMs or containers use only 1 vNIC for incoming and
outgoing traffic. Multiple queues can be used. Multiple VMs or
containers can be deployed prior to running any tests. This allows to
use generator-reflector pairs on the same or different compute nodes, on
the same or different NUMA nodes.
.. _338-opensource-vnf-onboarding-and-testing:
Opensource VNF onboarding and testing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Running opensource VNFs is a key technical solution to ensure that the
platforms meet Network Functions Virtualization requirements. `Functest
VNF `__
offers 5 test cases which automatically onboard and test the following 3
opensource VNFs:
- `Clearwater IMS `__
- `VyOS vRouter `__
- `OpenAirInterface vEPC `__
Here are the full list of orchestrators used for all these deployments:
- `Cloudify `__
- `Heat `__
- `Juju `__
The VNF are covered by upstream tests when possible (see
`clearwater-live-test `__)
and by Functest VNF tests in the other cases.
.. _339-tenants:
Tenants
~~~~~~~
.. _3310-lcm:
LCM
~~~
.. _3311-assurance:
Assurance
~~~~~~~~~
.. _3312-security:
Security
~~~~~~~~
.. _3313-resilience:
Resilience
~~~~~~~~~~
.. _3314-bare-metal-validations:
Bare-metal validations
~~~~~~~~~~~~~~~~~~~~~~
.. _34-test-cases-traceability-to-requirements:
Test Cases Traceability to Requirements
---------------------------------------
.. _341-rmra-1-requirements:
RM/RA-1 Requirements
~~~~~~~~~~~~~~~~~~~~
According to `RC1 Chapter04 `__ the following test cases
must pass as they are for Anuket NFVI Conformance:
======================================== ===================== ========
container test case criteria
======================================== ===================== ========
opnfv/functest-healthcheck:wallaby tempest_horizon PASS
opnfv/functest-smoke-cntt:wallaby tempest_neutron_cntt PASS
opnfv/functest-smoke-cntt:wallaby tempest_cinder_cntt PASS
opnfv/functest-smoke-cntt:wallaby tempest_keystone_cntt PASS
opnfv/functest-smoke-cntt:wallaby rally_sanity_cntt PASS
opnfv/functest-smoke-cntt:wallaby tempest_full_cntt PASS
opnfv/functest-smoke-cntt:wallaby tempest_scenario_cntt PASS
opnfv/functest-smoke-cntt:wallaby tempest_slow_cntt PASS
opnfv/functest-benchmarking-cntt:wallaby rally_full_cntt PASS
opnfv/functest-benchmarking-cntt:wallaby rally_jobs_cntt PASS
opnfv/functest-benchmarking-cntt:wallaby vmtp PASS
opnfv/functest-benchmarking-cntt:wallaby shaker PASS
opnfv/functest-vnf:wallaby cloudify PASS
opnfv/functest-vnf:wallaby cloudify_ims PASS
opnfv/functest-vnf:wallaby heat_ims PASS
opnfv/functest-vnf:wallaby vyos_vrouter PASS
opnfv/functest-vnf:wallaby juju_epc PASS
======================================== ===================== ========
.. _342-tc-mapping-to-requirements:
TC Mapping to Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------+---------------------------------------------+
| test case | requirements |
+=======================+=============================================+
| tempest_horizon | Horizon testing |
+-----------------------+---------------------------------------------+
| tempest_neutron_cntt | Neutron API testing |
+-----------------------+---------------------------------------------+
| tempest_cinder_cntt | Cinder API testing |
+-----------------------+---------------------------------------------+
| tempest_keystone_cntt | Keystone API testing |
+-----------------------+---------------------------------------------+
| rally_sanity_cntt | Keystone, Glance, Cinder, Swift, Neutron, |
| | Nova and Heat API testing |
+-----------------------+---------------------------------------------+
| tempest_full_cntt | Keystone, Glance, Cinder, Swift, Neutron |
| | and Nova API testing |
+-----------------------+---------------------------------------------+
| tempest_scenario_cntt | Keystone, Glance, Cinder, Swift, Neutron |
| | and Nova API testing |
+-----------------------+---------------------------------------------+
| tempest_slow_cntt | Keystone, Glance, Cinder, Swift, Neutron |
| | and Nova API testing |
+-----------------------+---------------------------------------------+
| rally_full_cntt | Keystone, Glance, Cinder, Swift, Neutron, |
| | Nova and Heat API benchmarking |
+-----------------------+---------------------------------------------+
| rally_jobs_cntt | Neutron API benchmarking |
+-----------------------+---------------------------------------------+
| vmtp | Dataplane benchmarking |
+-----------------------+---------------------------------------------+
| shaker | Dataplane benchmarking |
+-----------------------+---------------------------------------------+
| cloudify | opensource VNF onboarding and testing |
+-----------------------+---------------------------------------------+
| cloudify_ims | opensource VNF onboarding and testing |
+-----------------------+---------------------------------------------+
| heat_ims | opensource VNF onboarding and testing |
+-----------------------+---------------------------------------------+
| vyos_vrouter | opensource VNF onboarding and testing |
+-----------------------+---------------------------------------------+
| juju_epc | opensource VNF onboarding and testing |
+-----------------------+---------------------------------------------+