When sanity test is not done, API files are not loaded until the
first test case is run. Hence, it is not possible to use enums, etc.
outside of a test class.
By preloading API files before running any tests, it prevents its
issue.
Type: fix
Change-Id: I8730150374e6c5f8d6933ec037811372ac2a8da0
Signed-off-by: Maxime Peim <mpeim@cisco.com>
Allow settings default values explicitly in positive_int_or_default and
positive_float_or_default.
It allows setting setting default 0 test retries explicitly despite it
being not positive.
Type: improvement
Signed-off-by: Dmitry Valter <d-valter@yandex-team.com>
Change-Id: Id23a9fdae0ef174eea8992c1f9fc2530aade6194
- Make framework.py classes a subset of asfframework.py classes
- Remove all packet related code from asfframework.py
- Add test class and test case set up debug output to log
- Repatriate packet tests from asf to test directory
- Remove non-packet related code from framework.py and
inherit them from asfframework.py classes
- Clean up unused import variables
- Re-enable BFD tests on Ubuntu 22.04 and fix
intermittent test failures in echo_looped_back
testcases (where # control packets verified but
not guaranteed to be received during test)
- Re-enable Wireguard tests on Ubuntu 22.04 and fix
intermittent test failures in handshake ratelimiting
testcases and event testcase
- Run Wiregard testcase suites solo
- Improve debug output in log.txt
- Increase VCL/LDP post sleep timeout to allow iperf server
to finish cleanly.
- Fix pcap history files to be sorted by suite and testcase
and ensure order/timestamp is correct based on creation
in the testcase.
- Decode pcap files for each suite and testcase for all
errors or if configured via comandline option / env var
- Improve vpp corefile detection to allow complete corefile
generation
- Disable vm vpp interfaces testcases on debian11
- Clean up failed unittest dir when retrying failed testcases
and unify testname directory and failed linknames into
framwork functions
Type: test
Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
When custom-packaging the VPP artifacts, it can be useful to exclude
some of the core plugins from packaging/testing, for some reasons.
A removal of a plugin(s) from the worktree needs to be tracked as
a separate change, and thus is tricky from the maintenance
point of view.
This change adds the ability to "pretend they do not exist" -
plugins which are added to the comma-separated environment
variable "VPP_EXCLUDED_PLUGINS" will not be added to the build
process and not packaged.
The tests do not have the 1:1 relationship as plugins,
so they might need to be modified separately. This change
includes some of these modifications as an example.
Type: feature
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: Id31562d00a01ced1acbb4996a633517cbd6f09d8
cdf73b9731 has added the qemu tests as part of the default test run,
which results in "make test" failure in more restricted environments which do not allow the
namespace creation.
Add a config flag to skip those tests, and skip them if the namespace creation fails.
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: Ie631f7fb2a80864f77c79619eba4a43712e950e5
Support multiple comma-delimited filter expressions,
e.g. to run both bfd and ip4 tests, it's now possible to do:
make test TEST=bfd,ip4
Same goes for wildcards, e.g.:
make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error
Type: improvement
Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
To support testing of external plugins, add support to the test framework and PAPI
for specifying a list of locations to look for api.json files.
Type: improvement
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I128a306e3c091dc8ef994801b1470b82d2f4595d
Signed-off-by: Ole Troan <ot@cisco.com>
Usage:
test/run.py -r -t {test_filter}
Instead of starting a new instance of VPP, when the -r argument
is provided, test is run against a running VPP instance. Optionally,
one can also set the VPP socket directory using the -d
argument. The default location for socket files is
/var/run/user/${uid}/vpp and /var/run/vpp if VPP is started
as root.
Type: improvement
Change-Id: I05e57a067fcb90fb49973f8159fc17925b741f1a
Signed-off-by: Naveen Joy <najoy@cisco.com>
Type: refactor
- refactored VPP test code to remove "ignore_path" variable
from "discover_tests" function and "run_test" code
- configured VPP test makefile, config file, and 'run.sh' shell script
to move "venv" directory from "test" dir to "build-root" dir
Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com>
Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
When running tests via run.sh, default setting of None would cause
failed directory symlink to appear in vpp workspace with an ugly name.
This patch places the symlink in temporary directory.
Type: fix
Fixes: b23ffd7ef2
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: Ic1715eba7ac1f82f71855e2aeb9b659d27bbb3af
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.
Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.
Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.
Also, fixstyle-python is now available for automatic style formatting.
Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location. This is required to simply the automated generation of
docker executor images in the CI.
Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
The random seed is not an integer, so the current code does not allow reproducing a test run by running e.g.
RND_SEED=1647595144.0940742 make test
Solution: make the random seed a positive float.
Also, add the missing positiveness check to the positive_integer function.
Type: fix
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: I858bab0a9b828b99c20a2252aeecb9e2dda4ee21
Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.
Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
