ligang/vpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tests: make tests less make dependent

Browse Source 
Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.

Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
This commit is contained in:
Klement Sekera
2021-05-31 16:08:53 +02:00
committed by Andrew Yourtchenko
parent 8ccc6b3507
commit b23ffd7ef2
48 changed files with 805 additions and 516 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
docs/developer/build-run-debug/testing_vpp.rst
View File

@ -43,98 +43,85 @@ Current “make test-help” output:
::
$ make test-help
test - build and run (basic) functional tests
test-debug - build and run (basic) functional tests (debug build)
test-all - build and run functional and extended tests
test-all-debug - build and run functional and extended tests (debug build)
retest - run functional tests
retest-debug - run functional tests (debug build)
retest-all - run functional and extended tests
retest-all-debug - run functional and extended tests (debug build)
test-cov - generate code coverage report for test framework
test-gcov - build and run functional tests (gcov build)
test-wipe - wipe (temporary) files generated by unit tests
test-wipe-cov - wipe code coverage report for test framework
test-wipe-doc - wipe documentation for test framework
test-wipe-papi - rebuild vpp_papi sources
test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage
test-shell - enter shell with test environment
test-shell-debug - enter shell with test environment (debug build)
test-checkstyle - check PEP8 compliance for test framework
test-refresh-deps - refresh the Python dependencies for the tests
$ make test-help
Running tests:
Arguments controlling test runs:
V=[0|1|2] - set test verbosity level
0=ERROR, 1=INFO, 2=DEBUG
TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
FAILFAST=[0|1] - fail fast if 1, complete all tests if 0
TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
RETRIES=<n> - retry failed tests <n> times
DEBUG=<type> - set VPP debugging kind
DEBUG=core - detect coredump and load it in gdb on crash
DEBUG=gdb - allow easy debugging by printing VPP PID
and waiting for user input before running
and tearing down a testcase
DEBUG=gdbserver - run gdb inside a gdb server, otherwise
same as above
DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
test - build and run (basic) functional tests
test-debug - build and run (basic) functional tests (debug build)
test-all - build and run functional and extended tests
test-all-debug - build and run functional and extended tests (debug build)
retest - run functional tests
retest-debug - run functional tests (debug build)
retest-all - run functional and extended tests
retest-all-debug - run functional and extended tests (debug build)
test-cov - generate code coverage report for test framework
test-gcov - build and run functional tests (gcov build)
test-wipe - wipe (temporary) files generated by unit tests
test-wipe-cov - wipe code coverage report for test framework
test-wipe-papi - rebuild vpp_papi sources
test-wipe-all - wipe (temporary) files generated by unit tests, and coverage
test-shell - enter shell with test environment
test-shell-debug - enter shell with test environment (debug build)
test-checkstyle - check PEP8 compliance for test framework
test-refresh-deps - refresh the Python dependencies for the tests
STEP=[yes|no] - ease debugging by stepping through a testcase
SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
TEST=<filter> - filter the set of tests:
by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
Arguments controlling test runs:
VARIANT=<variant> - specify which march node variant to unit test
e.g. VARIANT=skx test the skx march variants
e.g. VARIANT=icl test the icl march variants
V=[0|1|2] - set test verbosity level
0=ERROR, 1=INFO, 2=DEBUG
TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes)
FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests
TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
RETRIES=<n> - retry failed tests <n> times
DEBUG=<type> - set VPP debugging kind
DEBUG=core - detect coredump and load it in gdb on crash
DEBUG=gdb - allow easy debugging by printing VPP PID
and waiting for user input before running
and tearing down a testcase
DEBUG=gdbserver - run gdb inside a gdb server, otherwise
same as above
DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging)
SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests
EXTENDED_TESTS=[1|y|yes] - run extended tests
TEST=<filter> - filter the set of tests:
by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
VARIANT=<variant> - specify which march node variant to unit test
e.g. VARIANT=skx test the skx march variants
e.g. VARIANT=icl test the icl march variants
COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
e.g. COREDUMP_SIZE=4g
COREDUMP_SIZE=unlimited
COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them
EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module
PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert)
API_FUZZ=[1|y|yes] - enable VPP api fuzz testing
RND_SEED=<seed> - Seed RND with given seed
COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
e.g. COREDUMP_SIZE=4g
COREDUMP_SIZE=unlimited
COREDUMP_COMPRESS=1 - compress core files if not debugging them
EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
Starting VPP in GDB for use with DEBUG=attach:
PROFILE=1 - enable profiling of test framework via cProfile module
PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
test-start-vpp-in-gdb - start VPP in gdb (release)
test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
TEST_DEBUG=1 - turn on debugging of the test framework itself (expert)
Creating test code coverage report:
SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI
test-cov - generate code coverage report for test framework
test-wipe-cov - wipe code coverage report for test framework
RND_SEED=seed - Seed RND with given seed
Verifying code-style:
Starting VPP in GDB for use with DEBUG=attach:
test-start-vpp-in-gdb - start VPP in gdb (release)
test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
Arguments controlling VPP in GDB runs:
VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead
VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line
Creating test documentation
test-doc - generate documentation for test framework
test-wipe-doc - wipe documentation for test framework
Creating test code coverage report
test-cov - generate code coverage report for test framework
test-wipe-cov - wipe code coverage report for test framework
Verifying code-style
test-checkstyle - check PEP8 compliance
test-checkstyle - check PEP8 compliance