b23ffd7ef2
Implement command line argument parsing instead of passing arguments via environment variables. Add script for running tests without having to invoke make. Deprecate running tests via make. Type: improvement Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
128 lines
6.5 KiB
ReStructuredText
128 lines
6.5 KiB
ReStructuredText
Testing VPP
|
|
===========
|
|
|
|
As of this writing, the vpp source tree includes over 1,000 unit test
|
|
vectors. Best practices prior to pushing patches for code review: make
|
|
sure that all of the “make test” test vectors pass.
|
|
|
|
We attempt to maintain the top-level “make test-help” command so that it
|
|
accurately describes all of the “make test” options.
|
|
|
|
Examples
|
|
--------
|
|
|
|
Basic test run, all test vectors, single-vpp instance, optimized image:
|
|
|
|
::
|
|
|
|
$ make test
|
|
|
|
10-way parallel basic test run:
|
|
|
|
::
|
|
|
|
$ make TEST_JOBS=10 test
|
|
|
|
Run a specific test suite (mpls, in this case):
|
|
|
|
::
|
|
|
|
$ make TEST=test_mpls test
|
|
|
|
Run a specific test suite, debug image, pause prior to running the test
|
|
suite; attach to the vpp image in gdb:
|
|
|
|
::
|
|
|
|
$ make TEST=xxx DEBUG=gdb test-debug
|
|
|
|
Detailed Documentation
|
|
----------------------
|
|
|
|
Current “make test-help” output:
|
|
|
|
::
|
|
|
|
$ make test-help
|
|
Running tests:
|
|
|
|
test - build and run (basic) functional tests
|
|
test-debug - build and run (basic) functional tests (debug build)
|
|
test-all - build and run functional and extended tests
|
|
test-all-debug - build and run functional and extended tests (debug build)
|
|
retest - run functional tests
|
|
retest-debug - run functional tests (debug build)
|
|
retest-all - run functional and extended tests
|
|
retest-all-debug - run functional and extended tests (debug build)
|
|
test-cov - generate code coverage report for test framework
|
|
test-gcov - build and run functional tests (gcov build)
|
|
test-wipe - wipe (temporary) files generated by unit tests
|
|
test-wipe-cov - wipe code coverage report for test framework
|
|
test-wipe-papi - rebuild vpp_papi sources
|
|
test-wipe-all - wipe (temporary) files generated by unit tests, and coverage
|
|
test-shell - enter shell with test environment
|
|
test-shell-debug - enter shell with test environment (debug build)
|
|
test-checkstyle - check PEP8 compliance for test framework
|
|
test-refresh-deps - refresh the Python dependencies for the tests
|
|
|
|
Arguments controlling test runs:
|
|
|
|
V=[0|1|2] - set test verbosity level
|
|
0=ERROR, 1=INFO, 2=DEBUG
|
|
TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
|
|
MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
|
|
CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes)
|
|
FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests
|
|
TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
|
|
RETRIES=<n> - retry failed tests <n> times
|
|
DEBUG=<type> - set VPP debugging kind
|
|
DEBUG=core - detect coredump and load it in gdb on crash
|
|
DEBUG=gdb - allow easy debugging by printing VPP PID
|
|
and waiting for user input before running
|
|
and tearing down a testcase
|
|
DEBUG=gdbserver - run gdb inside a gdb server, otherwise
|
|
same as above
|
|
DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
|
|
STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging)
|
|
SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests
|
|
EXTENDED_TESTS=[1|y|yes] - run extended tests
|
|
TEST=<filter> - filter the set of tests:
|
|
by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
|
|
by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
|
|
by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
|
|
e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
|
|
TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
|
|
TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
|
|
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
|
|
TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
|
|
VARIANT=<variant> - specify which march node variant to unit test
|
|
e.g. VARIANT=skx test the skx march variants
|
|
e.g. VARIANT=icl test the icl march variants
|
|
COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
|
|
e.g. COREDUMP_SIZE=4g
|
|
COREDUMP_SIZE=unlimited
|
|
COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them
|
|
EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
|
|
EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
|
|
EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
|
|
PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module
|
|
PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
|
|
PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
|
|
TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert)
|
|
API_FUZZ=[1|y|yes] - enable VPP api fuzz testing
|
|
RND_SEED=<seed> - Seed RND with given seed
|
|
|
|
Starting VPP in GDB for use with DEBUG=attach:
|
|
|
|
test-start-vpp-in-gdb - start VPP in gdb (release)
|
|
test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
|
|
|
|
Creating test code coverage report:
|
|
|
|
test-cov - generate code coverage report for test framework
|
|
test-wipe-cov - wipe code coverage report for test framework
|
|
|
|
Verifying code-style:
|
|
|
|
test-checkstyle - check PEP8 compliance
|