| Testing VPP |
| =========== |
| |
| As of this writing, the vpp source tree includes over 1,000 unit test |
| vectors. Best practices prior to pushing patches for code review: make |
| sure that all of the “make test” test vectors pass. |
| |
| We attempt to maintain the top-level “make test-help” command so that it |
| accurately describes all of the “make test” options. |
| |
| Examples |
| -------- |
| |
| Basic test run, all test vectors, single-vpp instance, optimized image: |
| |
| :: |
| |
| $ make test |
| |
| 10-way parallel basic test run: |
| |
| :: |
| |
| $ make TEST_JOBS=10 test |
| |
| Run a specific test suite (mpls, in this case): |
| |
| :: |
| |
| $ make TEST=test_mpls test |
| |
| Run a specific test suite, debug image, pause prior to running the test |
| suite; attach to the vpp image in gdb: |
| |
| :: |
| |
| $ make TEST=xxx DEBUG=gdb test-debug |
| |
| Detailed Documentation |
| ---------------------- |
| |
| Current “make test-help” output: |
| |
| :: |
| |
| $ make test-help |
| Running tests: |
| |
| test - build and run (basic) functional tests |
| test-debug - build and run (basic) functional tests (debug build) |
| test-all - build and run functional and extended tests |
| test-all-debug - build and run functional and extended tests (debug build) |
| retest - run functional tests |
| retest-debug - run functional tests (debug build) |
| retest-all - run functional and extended tests |
| retest-all-debug - run functional and extended tests (debug build) |
| test-cov - generate code coverage report for test framework |
| test-gcov - build and run functional tests (gcov build) |
| test-wipe - wipe (temporary) files generated by unit tests |
| test-wipe-cov - wipe code coverage report for test framework |
| test-wipe-papi - rebuild vpp_papi sources |
| test-wipe-all - wipe (temporary) files generated by unit tests, and coverage |
| test-shell - enter shell with test environment |
| test-shell-debug - enter shell with test environment (debug build) |
| test-checkstyle - check PEP8 compliance for test framework |
| test-refresh-deps - refresh the Python dependencies for the tests |
| |
| Arguments controlling test runs: |
| |
| V=[0|1|2] - set test verbosity level |
| 0=ERROR, 1=INFO, 2=DEBUG |
| TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1) |
| MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) |
| CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes) |
| FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests |
| TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600) |
| RETRIES=<n> - retry failed tests <n> times |
| DEBUG=<type> - set VPP debugging kind |
| DEBUG=core - detect coredump and load it in gdb on crash |
| DEBUG=gdb - allow easy debugging by printing VPP PID |
| and waiting for user input before running |
| and tearing down a testcase |
| DEBUG=gdbserver - run gdb inside a gdb server, otherwise |
| same as above |
| DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) |
| STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging) |
| SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests |
| EXTENDED_TESTS=[1|y|yes] - run extended tests |
| TEST=<filter> - filter the set of tests: |
| by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py |
| by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py |
| by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*' |
| e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name |
| TEST='bfd.*.*' is equivalent to above example of filter by file-suffix |
| TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class |
| TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase |
| TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes |
| VARIANT=<variant> - specify which march node variant to unit test |
| e.g. VARIANT=skx test the skx march variants |
| e.g. VARIANT=icl test the icl march variants |
| COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp |
| e.g. COREDUMP_SIZE=4g |
| COREDUMP_SIZE=unlimited |
| COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them |
| EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases |
| EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test |
| EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report |
| PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module |
| PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) |
| PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) |
| TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert) |
| API_FUZZ=[1|y|yes] - enable VPP api fuzz testing |
| RND_SEED=<seed> - Seed RND with given seed |
| |
| Starting VPP in GDB for use with DEBUG=attach: |
| |
| test-start-vpp-in-gdb - start VPP in gdb (release) |
| test-start-vpp-debug-in-gdb - start VPP in gdb (debug) |
| |
| Creating test code coverage report: |
| |
| test-cov - generate code coverage report for test framework |
| test-wipe-cov - wipe code coverage report for test framework |
| |
| Verifying code-style: |
| |
| test-checkstyle - check PEP8 compliance |