| .. This work is licensed under a |
| Creative Commons Attribution 4.0 International License. |
| .. integration-tests: |
| |
| Tests |
| ===== |
| |
| .. important:: |
| Integration is in charged of several types of tests: |
| |
| - Use Cases: developped by use case teams, usually complex, demonstrating high value capabilities ofr ONAP. They may be partially automated and even |
| integrated in CD. |
| - CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository |
| - Automatic Test Cases: these use cases are usually more simple and aims to validate that ONAP is working properly. |
| These tests have been developed to validate ONAP as a software solution. |
| In theory all the main function shall be covered by such tests in order to have more robust CI/CD and then avoid regressions. |
| These tests are usually developed and maintained by the integration team. |
| |
| We may also indicate that when the development of the test framework python-onapsk |
| follows standard developement quality rules and imposes the creation of |
| unit/functional/integration tests. |
| As an example python-onapsdk requires a unit test coverage of 98% before merging |
| a new feature, which is far above the project criteria in SonarCLoud today. |
| |
| Use Cases |
| --------- |
| |
| The use cases are described in :ref:`Verified Use cases <docs_usecases>`. |
| |
| CSIT Tests |
| ---------- |
| |
| The CSIT tests are functional tests executed by the projects on mocked |
| environement to validate their components. |
| Historically it was hosted in a CSIT repository. |
| |
| Integration team invited the projects to bring back such tests bacdk to home |
| repository for 2 main reasons: |
| |
| - integration cannot be a bottleneck: +2/merge from integration needed for each |
| project |
| - most of the tests are abandonned and not maintained when hosted in a third party |
| repository leading to CI/CD time waste and misleading test reporting |
| |
| In Guilin a PoC to help the project to re-insource their functional tests have |
| been initiated. |
| See `CSIT wiki page <https://wiki.onap.org/display/DW/Maximizing+Benefits+of+CSIT+in+ONAP+Development>`__ |
| for details. |
| |
| Automatic Tests |
| --------------- |
| |
| These tests are run daily/weekly on each new gate (new patchset in OOM, clamp |
| or SO). They can be in any language (bash, go, python,...), leveraging any test |
| framework (robotframework, MTS, python-onapsdk). |
| They are all embedded in `xtesting <https://pypi.org/project/xtesting/>`__ dockers. |
| |
| .. hint:: |
| Automatic tests are currently divided in 4 different categories: |
| |
| - infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates,.. |
| - healthcheck: basic tests on components |
| - smoke tests: end to end tests |
| - security tests |
| |
| A dashboard summarizing the status and providing the links to the test result |
| page or the logs is automatically created at the end of the execution of the |
| tests. |
| |
| .. figure:: files/tests/test-dashboard.png |
| |
| All the pages and artifacts are pushed to LF backend: |
| |
| - Daily chaines: https://logs.onap.org/onap-integration/daily |
| - Weekly chains: https://logs.onap.org/onap-integration/weekly |
| - Gating chains: https://logs.onap.org/onap-integration/gating |
| |
| |
| Infrastructure Healthcheck Tests |
| ................................ |
| |
| .. csv-table:: Infrastructure Healthcheck Tests |
| :file: ./files/csv/tests-infrastructure-healthcheck.csv |
| :widths: 20,40,20,20 |
| :delim: ; |
| :header-rows: 1 |
| |
| See `Infrastructure Healthcheck README <https://git.onap.org/integration/xtesting/tree/infra-healthcheck/README.md>`__ |
| to adapt then run infrastructure healthcheck tests on your own system. |
| |
| Please note that the onap-k8s is run 2 times in CD chains. It is run just after |
| the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown) |
| in order to collect the logs of the different components during the test execution. |
| |
| .. figure:: files/tests/test-onap-k8s.png |
| |
| Healhcheck Tests |
| ................ |
| |
| .. csv-table:: Healthcheck Tests |
| :file: ./files/csv/tests-healthcheck.csv |
| :widths: 20,40,20,20 |
| :delim: ; |
| :header-rows: 1 |
| |
| See `Healthcheck README <https://git.onap.org/integration/xtesting/tree/healthcheck/README.md>`__ |
| to adapt then run healthcheck tests on your own system. |
| |
| Smoke Tests |
| ........... |
| |
| .. csv-table:: Smoke Tests |
| :file: ./files/csv/tests-smoke.csv |
| :widths: 20,40,20,20 |
| :delim: ; |
| :header-rows: 1 |
| |
| See `Python smoke test README <https://git.onap.org/integration/xtesting/tree/smoke-usecases-robot/README.md>`__ |
| to adapt and run robot based smoke tests. |
| An html page is generated by the pythonsdk-test tests. |
| |
| .. figure:: files/tests/test-basic-cnf.png |
| |
| |
| See `Robot smoke test README <https://git.onap.org/integration/xtesting/tree/smoke-usecases-pythonsdk/README.md>`__ |
| to adapt and run pythonsdk based smoke tests. |
| Standard Robot html pages are generated. See :ref:`Robot page <docs_robot>`. |
| |
| Security Tests |
| ............... |
| |
| .. csv-table:: Security Tests |
| :file: ./files/csv/tests-security.csv |
| :widths: 20,40,20,20 |
| :delim: ; |
| :header-rows: 1 |
| |
| See `Security test README <https://git.onap.org/integration/xtesting/tree/security/README.md>`__ |
| to adapt then run the security tests on your own system. |
| |
| Note for security tests, integration team follows `SECCOM recommendations and |
| apply waivers granted by SECCOM if needed through xfail lists <https://git.onap.org/integration/seccom/tree/>`__. |
| |
| Stability Testing |
| ----------------- |
| |
| Ensuring the stability of ONAP is one of the missions of the Integration team. |
| CI chains and stability tests are performed to help stabilising the release. |
| See :ref:`Integration stability tests <integration-s3p>` for details. |