- Required for Ubuntu 24.04 LTS jobs
- temporarily disable TestIpsecEsp1 and
TestIpsecAhAll tests until a patch can
be added to fix them
Type: test
Change-Id: I1ae7b170117182c3252629bbbb770775e2c496c9
Signed-off-by: Benoît Ganne <bganne@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Introduce "none", "all" and "failed" options for --decode-pcaps
parameter. Keep "failed" as default to be consistent with current
behaviour. Add missing documentation to test/Makefile and passthrough to
Makefile.
Rationale: running tshark binary takes about 100-150ms and if there are
thousands of pcap files, it takes minutes to decode them. This might not
be desirable if rerunning the tests repeatedly during development.
Type: improvement
Change-Id: Ie033521d51d18b9d499b9bc40fe6eff21c94622d
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
- Remove code from test report that is effectively
untested and categorize based on reason for lack
of testing.
Type: test
Change-Id: I6ca5444055b3a81a4880945b6845afc867556277
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
VPP requires GNU Make to build, on GNU systems (such as Debian), GNU
Make is installed as 'make', typically with a symlink from 'gmake'.
On other systems (such as FreeBSD), 'make' is a BSD Make derriviative
and GNU Make is installed a 'gmake'.
Use $(MAKE) variable for make calls from within Makefiles. This
variable is set to the path of the calling make program, i.e.,
/usr/local/bin/gmake on a bsd system.
This is the recommended way to call make from Makefiles in the GNU Make
documentation.
Type: improvement
Change-Id: Id9162a34a0f8358f22090718087918dae31c0fce
Signed-off-by: Tom Jones <thj@freebsd.org>
When sanity test is not done, API files are not loaded until the
first test case is run. Hence, it is not possible to use enums, etc.
outside of a test class.
By preloading API files before running any tests, it prevents its
issue.
Type: fix
Change-Id: I8730150374e6c5f8d6933ec037811372ac2a8da0
Signed-off-by: Maxime Peim <mpeim@cisco.com>
- Make framework.py classes a subset of asfframework.py classes
- Remove all packet related code from asfframework.py
- Add test class and test case set up debug output to log
- Repatriate packet tests from asf to test directory
- Remove non-packet related code from framework.py and
inherit them from asfframework.py classes
- Clean up unused import variables
- Re-enable BFD tests on Ubuntu 22.04 and fix
intermittent test failures in echo_looped_back
testcases (where # control packets verified but
not guaranteed to be received during test)
- Re-enable Wireguard tests on Ubuntu 22.04 and fix
intermittent test failures in handshake ratelimiting
testcases and event testcase
- Run Wiregard testcase suites solo
- Improve debug output in log.txt
- Increase VCL/LDP post sleep timeout to allow iperf server
to finish cleanly.
- Fix pcap history files to be sorted by suite and testcase
and ensure order/timestamp is correct based on creation
in the testcase.
- Decode pcap files for each suite and testcase for all
errors or if configured via comandline option / env var
- Improve vpp corefile detection to allow complete corefile
generation
- Disable vm vpp interfaces testcases on debian11
- Clean up failed unittest dir when retrying failed testcases
and unify testname directory and failed linknames into
framwork functions
Type: test
Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
When custom-packaging the VPP artifacts, it can be useful to exclude
some of the core plugins from packaging/testing, for some reasons.
A removal of a plugin(s) from the worktree needs to be tracked as
a separate change, and thus is tricky from the maintenance
point of view.
This change adds the ability to "pretend they do not exist" -
plugins which are added to the comma-separated environment
variable "VPP_EXCLUDED_PLUGINS" will not be added to the build
process and not packaged.
The tests do not have the 1:1 relationship as plugins,
so they might need to be modified separately. This change
includes some of these modifications as an example.
Type: feature
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: Id31562d00a01ced1acbb4996a633517cbd6f09d8
- Package update performed by
1. updating pip, pip-tools, setuptools
2. 'make test-refresh-deps' on ubuntu 22.04
3. fixing 'make test' and 'make docs' issues
on ubuntu 22.04
4. 'make test-refresh-deps' on ubuntu 20.04
- Add dependency for 'make test-refresh-deps'
to insure python venv is set up.
- Update of python formatter, black,
caused reformating of 41 python code
files.
Type: make
Change-Id: I7cafdf4b5189065ac57cb6b254937f6e0897a924
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Remove test code & non-vpp code from coverage report
- Remove driver/hardware vpp code which cannot be tested
in 'make test' from coverage report
Type: fix
Change-Id: I04b50c14bc3437b845f2afafae47297189e61e3f
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Updated/rebased version of https://gerrit.fd.io/r/c/vpp/+/34199
Type: test
Change-Id: I43913ecfd11a4578bdb10c4be76253fe38d57976
Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Support multiple comma-delimited filter expressions,
e.g. to run both bfd and ip4 tests, it's now possible to do:
make test TEST=bfd,ip4
Same goes for wildcards, e.g.:
make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error
Type: improvement
Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
To support testing of external plugins, add support to the test framework and PAPI
for specifying a list of locations to look for api.json files.
Type: improvement
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I128a306e3c091dc8ef994801b1470b82d2f4595d
Signed-off-by: Ole Troan <ot@cisco.com>
Type: refactor
- refactored VPP test code to remove "ignore_path" variable
from "discover_tests" function and "run_test" code
- configured VPP test makefile, config file, and 'run.sh' shell script
to move "venv" directory from "test" dir to "build-root" dir
Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com>
Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
Use the script test/run.py to run a test named test_vm_tap
inside a QEMU VM. The run script builds out a virtual env,
launches a light weight QEMU VM, mounts host directories,
starts VPP inside the VM and runs the test. The test named
test_vm_tap, creates two tap v2 interfaces in separate Linux
namespaces and using iPerf, streams traffic between the VM
and VPP. All data files are stored in the directory named
/tmp/vpp-vm-tests. To clean up, use the make test-wipe
command.
Usage:
test/run.py --vm --debug --test test_vm_tap
Type: improvement
Change-Id: I4425dbef52acee1e5b8af5acaa169b89a2c0f171
Signed-off-by: Naveen Joy <najoy@cisco.com>
Add --check to black to pass back error value and fail checkstyle if
a reformat would occur.
Type: fix
Fixes: d9b0c6fbf7aa5bd9af84264105b39c82028a4a29
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I9a1fbe224929fc461ff833a589f73ca06e7cc9d6
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.
Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.
Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.
Also, fixstyle-python is now available for automatic style formatting.
Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location. This is required to simply the automated generation of
docker executor images in the CI.
Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Make make test-start-vpp-in-gdb work again.
Fix incorrect temp directory when using DEBUG=attach.
Type: fix
Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec
Change-Id: Ie98b637acbbe0221606ccdc7b54f63885e5951a4
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.
Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Use 'python3' for all python calls in virtualenv. Using $(PYTHON_INTERP)
(set to $(PYTHON)) is incorrect when $(PYTHON) contains absolute path
(e.g. '/usr/bin/pypy3') as it uses interpreter outside of virtualenv.
Type: fix
Change-Id: I19878360cfc1ab6a591e09a079b0b69c1a1a58cf
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Add API to disable full reassembly of "forus" packets. Mark packets
passing through ip[4|6]-local nodes with a new buffer flag and check for
that flag in reassembly.
Enable IP6 "forus" full reassembly by default to be consistent with
existing IP4 setting.
Type: improvement
Change-Id: I7067792fcd4304182654237968e4c4d9293c6143
Signed-off-by: Klement Sekera <ksekera@cisco.com>
- Also fix docs requirements and venv cleanup
required for docker executor building
Type: test
Change-Id: I839423224d95c45af42f459217887f4201cbb11a
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
This patch refactors the VPP sphinx docs
in order to make it easier to consume
for external readers as well as VPP developers.
It also makes sphinx the single source
of documentation, which simplifies maintenance
and operation.
Most important updates are:
- reformat the existing documentation as rst
- split RELEASE.md and move it into separate rst files
- remove section 'events'
- remove section 'archive'
- remove section 'related projects'
- remove section 'feature by release'
- remove section 'Various links'
- make (Configuration reference, CLI docs,
developer docs) top level items in the list
- move 'Use Cases' as part of 'About VPP'
- move 'Troubleshooting' as part of 'Getting Started'
- move test framework docs into 'Developer Documentation'
- add a 'Contributing' section for gerrit,
docs and other contributer related infos
- deprecate doxygen and test-docs targets
- redirect the "make doxygen" target to "make docs"
Type: refactor
Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f
Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com>
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
- Upgrade python package requirements for test & docs
- Clean up docs generation warnings
- Consolidate python requirements for docs in test
requirements specs.
- Upgrade pip
Type: make
Change-Id: I74a3924b43ed93d15b32ec9f6fc41ed1ba95b69b
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Generate copyright year and version
instead of using hard-coded data
Type: refactor
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I6058f5025323b3aa483f5df4a2c4371e27b5914e
Introduce MAX_CPUS parameters to control maximum number of CPUs used by
VPP(s) during testing, with default value 'auto' corresponding to all
CPUs available.
Calculate test CPU requirements by taking into account the number of
workers, so a test requires 1 (main thread) + # of worker CPUs.
When running tests, keep track of both running test jobs (controlled by
TEST_JOBS parameter) and free CPUs. This then causes two limits in the
system - to not exceed number of jobs in parallel but also to not exceed
number of CPUs available.
Skip tests which require more CPUs than are available in system (or more
than MAX_CPUS) and print a warning message.
Type: improvement
Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Introduce a new option DEBUG=attach to run a test against existing
already running vpp. A new target 'make test-start-gdb' will spawn VPP
in gdb for this purpose. Customization options explained in test-help.
Type: improvement
Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Make test-checkstyle-diff is a new target which checks PEP8 compliance
only for changed files. This makes it faster to execute and also more
readable as most of the time, only changed files will fail.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I71baca76ab3a21a7a3790617cbfb0d48aacbd9ec
- Required to relocate test infra modules
to be co-located in feature test directory.
- Remove/regenerate build-root/build-test/src/
soft links when running make test to allow
re-running 'make test' after moving source files.
- Suppress annoying 'ls' error message when gathering
test source files caused by 'c' test application
directory being named 'test'.
- Use PYTHONPYCACHEPREFIX to generate all
compiled source under build-root/build-test.
This only works for python 3.8 or greater, but
existing 'make test-wipe' target cleans up
in-situ __pycache__ directories for older
python versions.
Type: fix
Change-Id: If3c3ff0c58fb26a17395babc34783e06b1dd84e7
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Even though we don't use shm transport anymore, vpp still populates shm.
Add cleanup before running tests to avoid the nuisance of running out of
space on /dev/shm.
Type: fix
Fixes: eb506becd038f081bc7ac340412d3685425bb8b4
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I2ef87e991223ee60892d56af1482025d91e77d46
Not used anymore with socket transport.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
Type: fix
with hard links this sequence doesn't work
echo "GARBAGE" >> test/vpp_ipsec.py
git checkout test/vpp_ipsec.py
also blow away the staging diretory with 'test-wipe' just to be thorough
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: Ic38d7ec514102759f2eafead59c7f56c5b39f33d
- Use hard links instead of soft links so
that python module import works correctly
for relocated test/vpp_*.py modules.
Type: fix
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: If9c031bf0918344dedd4dee5f9336e869590b0ae
Node variant names have changed to skx and icl.
Type: fix
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I5be832aa59e4e440df53869077d801c011e3defa
This ensures we don't recompute the requirements-{2,3}.txt on each make
test run, and skips patching scapy if it is already patched instead of
failing.
Change-Id: I3da57182ae49f3dd04db139d96734a5d145fedff
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
This fixes an issue where the pinned requirements file can be modified
when running the tests.
Change-Id: Ic89d1844d1fd8d00f62211a9b051a26ac34ee316
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
Support for startup node multi-arch variant selection through startup.conf.
This is to facilitate unit, functional testing and benchmarking of non-default
multi-arch variant node code path. Also added parameters to make test, to
specific using multi-arch variants in unit testing.
Type: improvement
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
Add these two packages to requirements.txt so that their version and
the version of their dependencies are pinned to limit the risk of
unexpected breakage.
Change-Id: If330404f2e840af3d2628f997ce406cd14e7e128
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
