docs/integration-tests.rst

.. This work is licensed under a
   Creative Commons Attribution 4.0 International License.
.. _integration-tests:

Tests
=====

.. important::
   Integration is in charge of several types of tests:

   - Use Cases: developed by use case teams, usually complex, demonstrating high value capabilities of ONAP. They may be partially automated and even
     integrated in CD.
   - CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository
   - Automatic Test Cases: these use cases are usually more simple and aim to validate that ONAP is working properly.
     These tests have been developed to validate ONAP as a software solution.
     In theory all the main functions shall be covered by such tests in order to have more robust CI/CD and then avoid regressions.
     These tests are usually developed and maintained by the integration team.

We may also indicate that when the development of the test framework python-onapsdk
follows standard development quality rules and imposes the creation of
unit/functional/integration tests.
As an example python-onapsdk requires a unit test coverage of 98% before merging
a new feature, which is far above the project criteria in SonarCloud today.

Use Cases
---------

The use cases of the last release are described in
:ref:`Verified Use cases <docs_usecases_release>`.

The history of the different use cases across the different releases can be
found in :ref:`Use Cases page <docs_usecases>`.

CSIT Tests
----------

The CSIT tests are functional tests executed by the projects on mocked
environment to validate their components.
Historically it was hosted in a CSIT repository.

Integration team invited the projects to bring back such tests back to home
repository for 2 main reasons:

- integration cannot be a bottleneck: +2/merge from integration needed for each
  project
- most of the tests are abandoned and not maintained when hosted in a third party
  repository leading to CI/CD resource waste and misleading test reporting

In Guilin a PoC to help the project to re-insource their functional tests have
been initiated.
See `CSIT wiki page <https://wiki.onap.org/display/DW/Maximizing+Benefits+of+CSIT+in+ONAP+Development>`_
for details.

Automatic Tests
---------------

These tests are run daily/weekly on each new gate (new patchset in OOM, clamp
or SO). They can be in any language (bash, go, python,...), leveraging any test
framework (robotframework, MTS, python-onapsdk).
They are all embedded in `xtesting <https://pypi.org/project/xtesting/>`_ dockers.

.. hint::
   Automatic tests are currently divided in 4 different categories:

   - infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates...
   - healthcheck: basic tests on components
   - smoke tests: end to end tests
   - security tests

A dashboard summarizing the status and providing the links to the test result
page or the logs is automatically created at the end of the execution of the
tests.

.. figure:: files/tests/test-dashboard.png

All the pages and artifacts are pushed to LF backend:

- Daily chaines: https://logs.onap.org/onap-integration/daily
- Weekly chains: https://logs.onap.org/onap-integration/weekly
- Gating chains: https://logs.onap.org/onap-integration/gating


Infrastructure Healthcheck Tests
................................

.. csv-table:: Infrastructure Healthcheck Tests
    :file: ./files/csv/tests-infrastructure-healthcheck.csv
    :widths: 20,40,20,20
    :delim: ;
    :header-rows: 1

See `Infrastructure Healthcheck README <https://git.onap.org/integration/xtesting/tree/infra-healthcheck/README.md>`_
to adapt then run infrastructure healthcheck tests on your own system.

Please note that the onap-k8s is run 2 times in CD chains. It is run just after
the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown)
in order to collect the logs of the different components during the test execution.

.. figure:: files/tests/test-onap-k8s.png

Healthcheck Tests
.................

.. csv-table:: Healthcheck Tests
    :file: ./files/csv/tests-healthcheck.csv
    :widths: 20,40,20,20
    :delim: ;
    :header-rows: 1

See `Healthcheck README <https://git.onap.org/integration/xtesting/tree/healthcheck/README.md>`_
to adapt then run healthcheck tests on your own system.

Smoke Tests
...........

.. csv-table:: Smoke Tests
    :file: ./files/csv/tests-smoke.csv
    :widths: 20,40,20,20
    :delim: ;
    :header-rows: 1

See `Python smoke test README <https://git.onap.org/integration/xtesting/tree/smoke-usecases-robot/README.md>`_
to adapt and run robot based smoke tests.
An html page is generated by the pythonsdk-test tests.

.. figure:: files/tests/test-basic-cnf.png


See `Robot smoke test README <https://git.onap.org/integration/xtesting/tree/smoke-usecases-pythonsdk/README.md>`_
to adapt and run python-onapsdk based smoke tests.
Standard Robot html pages are generated. See :ref:`Robot page <docs_robot>`.

Security Tests
...............

.. csv-table:: Security Tests
    :file: ./files/csv/tests-security.csv
    :widths: 20,40,20,20
    :delim: ;
    :header-rows: 1

See `Security test README <https://git.onap.org/integration/xtesting/tree/security/README.md>`_
to adapt then run the security tests on your own system.

Note for security tests, integration team follows `SECCOM recommendations and
apply waivers granted by SECCOM if needed through xfail lists <https://git.onap.org/integration/seccom/tree/>`_.

Stability Testing
-----------------

Ensuring the stability of ONAP is one of the missions of the Integration team.
CI chains and stability tests are performed to help stabilising the release.
See :ref:`Integration stability tests  <integration-s3p>` for details.