Testing VPP
===========

As of this writing, the vpp source tree includes over 1,000 unit test
vectors. Best practices prior to pushing patches for code review: make
sure that all of the “make test” test vectors pass.

We attempt to maintain the top-level “make test-help” command so that it
accurately describes all of the “make test” options.

Examples
--------

Basic test run, all test vectors, single-vpp instance, optimized image:

::

     $ make test

10-way parallel basic test run:

::

     $ make TEST_JOBS=10 test

Run a specific test suite (mpls, in this case):

::

     $ make TEST=test_mpls test

Run a specific test suite, debug image, pause prior to running the test
suite; attach to the vpp image in gdb:

::

     $ make TEST=xxx DEBUG=gdb test-debug

Detailed Documentation
----------------------

Current “make test-help” output:

::

    $ make test-help
    Running tests:

     test                   - build and run (basic) functional tests
     test-debug             - build and run (basic) functional tests (debug build)
     test-all               - build and run functional and extended tests
     test-all-debug         - build and run functional and extended tests (debug build)
     retest                 - run functional tests
     retest-debug           - run functional tests (debug build)
     retest-all             - run functional and extended tests
     retest-all-debug       - run functional and extended tests (debug build)
     test-cov               - generate code coverage report for test framework
     test-gcov                      - build and run functional tests (gcov build)
     test-wipe              - wipe (temporary) files generated by unit tests
     test-wipe-cov          - wipe code coverage report for test framework
     test-wipe-papi         - rebuild vpp_papi sources
     test-wipe-all          - wipe (temporary) files generated by unit tests, and coverage
     test-shell             - enter shell with test environment
     test-shell-debug       - enter shell with test environment (debug build)
     test-checkstyle        - check PEP8 compliance for test framework
     test-refresh-deps      - refresh the Python dependencies for the tests

    Arguments controlling test runs:

     V=[0|1|2]              - set test verbosity level
                              0=ERROR, 1=INFO, 2=DEBUG
     TEST_JOBS=[<n>|auto]   - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
     MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
     CACHE_OUTPUT=[0|n|no]  - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes)
     FAILFAST=[1|y|yes]     - fail fast if 1, otherwise complete all tests
     TIMEOUT=<timeout>      - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
     RETRIES=<n>            - retry failed tests <n> times
     DEBUG=<type>           - set VPP debugging kind
        DEBUG=core          - detect coredump and load it in gdb on crash
        DEBUG=gdb           - allow easy debugging by printing VPP PID
                              and waiting for user input before running
                              and tearing down a testcase
        DEBUG=gdbserver     - run gdb inside a gdb server, otherwise
                              same as above
        DEBUG=attach        - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
     STEP=[1|y|yes]         - enable stepping through a testcase (for testcase debugging)
     SANITY=[0|n|no]        - disable sanity import of vpp-api/sanity vpp run before running tests
     EXTENDED_TESTS=[1|y|yes] - run extended tests
     TEST=<filter>          - filter the set of tests:
        by file-name        - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
        by file-suffix      - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
        by wildcard         - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
                              e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
                                   TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
                                   TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
                                   TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
                                   TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
     VARIANT=<variant>      - specify which march node variant to unit test
                              e.g. VARIANT=skx test the skx march variants
                              e.g. VARIANT=icl test the icl march variants
     COREDUMP_SIZE=<size>   - pass <size> as unix { coredump-size <size> } argument to vpp
                              e.g. COREDUMP_SIZE=4g
                                   COREDUMP_SIZE=unlimited
     COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them
     EXTERN_TESTS=<path>    - path to out-of-tree test_<name>.py files containing test cases
     EXTERN_PLUGINS=<path>  - path to out-of-tree plugins to be loaded by vpp under test
     EXTERN_COV_DIR=<path>  - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
     PROFILE=[1|y|yes]      - enable profiling of test framework via cProfile module
     PROFILE_SORT_BY=opt    - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
     PROFILE_OUTPUT=file    - output profiling info to file - use absolute path (default: stdout)
     TEST_DEBUG=[1|y|yes]   - enable debugging of the test framework itself (expert)
     API_FUZZ=[1|y|yes]     - enable VPP api fuzz testing
     RND_SEED=<seed>        - Seed RND with given seed

    Starting VPP in GDB for use with DEBUG=attach:

     test-start-vpp-in-gdb       - start VPP in gdb (release)
     test-start-vpp-debug-in-gdb - start VPP in gdb (debug)

    Creating test code coverage report:

     test-cov               - generate code coverage report for test framework
     test-wipe-cov          - wipe code coverage report for test framework

    Verifying code-style:

     test-checkstyle        - check PEP8 compliance