Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.
Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Use 'python3' for all python calls in virtualenv. Using $(PYTHON_INTERP)
(set to $(PYTHON)) is incorrect when $(PYTHON) contains absolute path
(e.g. '/usr/bin/pypy3') as it uses interpreter outside of virtualenv.
Type: fix
Change-Id: I19878360cfc1ab6a591e09a079b0b69c1a1a58cf
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Add API to disable full reassembly of "forus" packets. Mark packets
passing through ip[4|6]-local nodes with a new buffer flag and check for
that flag in reassembly.
Enable IP6 "forus" full reassembly by default to be consistent with
existing IP4 setting.
Type: improvement
Change-Id: I7067792fcd4304182654237968e4c4d9293c6143
Signed-off-by: Klement Sekera <ksekera@cisco.com>
- Also fix docs requirements and venv cleanup
required for docker executor building
Type: test
Change-Id: I839423224d95c45af42f459217887f4201cbb11a
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
This patch refactors the VPP sphinx docs
in order to make it easier to consume
for external readers as well as VPP developers.
It also makes sphinx the single source
of documentation, which simplifies maintenance
and operation.
Most important updates are:
- reformat the existing documentation as rst
- split RELEASE.md and move it into separate rst files
- remove section 'events'
- remove section 'archive'
- remove section 'related projects'
- remove section 'feature by release'
- remove section 'Various links'
- make (Configuration reference, CLI docs,
developer docs) top level items in the list
- move 'Use Cases' as part of 'About VPP'
- move 'Troubleshooting' as part of 'Getting Started'
- move test framework docs into 'Developer Documentation'
- add a 'Contributing' section for gerrit,
docs and other contributer related infos
- deprecate doxygen and test-docs targets
- redirect the "make doxygen" target to "make docs"
Type: refactor
Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f
Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com>
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
- Upgrade python package requirements for test & docs
- Clean up docs generation warnings
- Consolidate python requirements for docs in test
requirements specs.
- Upgrade pip
Type: make
Change-Id: I74a3924b43ed93d15b32ec9f6fc41ed1ba95b69b
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Generate copyright year and version
instead of using hard-coded data
Type: refactor
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I6058f5025323b3aa483f5df4a2c4371e27b5914e
Introduce MAX_CPUS parameters to control maximum number of CPUs used by
VPP(s) during testing, with default value 'auto' corresponding to all
CPUs available.
Calculate test CPU requirements by taking into account the number of
workers, so a test requires 1 (main thread) + # of worker CPUs.
When running tests, keep track of both running test jobs (controlled by
TEST_JOBS parameter) and free CPUs. This then causes two limits in the
system - to not exceed number of jobs in parallel but also to not exceed
number of CPUs available.
Skip tests which require more CPUs than are available in system (or more
than MAX_CPUS) and print a warning message.
Type: improvement
Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Introduce a new option DEBUG=attach to run a test against existing
already running vpp. A new target 'make test-start-gdb' will spawn VPP
in gdb for this purpose. Customization options explained in test-help.
Type: improvement
Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Make test-checkstyle-diff is a new target which checks PEP8 compliance
only for changed files. This makes it faster to execute and also more
readable as most of the time, only changed files will fail.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I71baca76ab3a21a7a3790617cbfb0d48aacbd9ec
- Required to relocate test infra modules
to be co-located in feature test directory.
- Remove/regenerate build-root/build-test/src/
soft links when running make test to allow
re-running 'make test' after moving source files.
- Suppress annoying 'ls' error message when gathering
test source files caused by 'c' test application
directory being named 'test'.
- Use PYTHONPYCACHEPREFIX to generate all
compiled source under build-root/build-test.
This only works for python 3.8 or greater, but
existing 'make test-wipe' target cleans up
in-situ __pycache__ directories for older
python versions.
Type: fix
Change-Id: If3c3ff0c58fb26a17395babc34783e06b1dd84e7
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Even though we don't use shm transport anymore, vpp still populates shm.
Add cleanup before running tests to avoid the nuisance of running out of
space on /dev/shm.
Type: fix
Fixes: eb506becd0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I2ef87e991223ee60892d56af1482025d91e77d46
Not used anymore with socket transport.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
Type: fix
with hard links this sequence doesn't work
echo "GARBAGE" >> test/vpp_ipsec.py
git checkout test/vpp_ipsec.py
also blow away the staging diretory with 'test-wipe' just to be thorough
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: Ic38d7ec514102759f2eafead59c7f56c5b39f33d
- Use hard links instead of soft links so
that python module import works correctly
for relocated test/vpp_*.py modules.
Type: fix
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: If9c031bf0918344dedd4dee5f9336e869590b0ae
Node variant names have changed to skx and icl.
Type: fix
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I5be832aa59e4e440df53869077d801c011e3defa
This ensures we don't recompute the requirements-{2,3}.txt on each make
test run, and skips patching scapy if it is already patched instead of
failing.
Change-Id: I3da57182ae49f3dd04db139d96734a5d145fedff
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
This fixes an issue where the pinned requirements file can be modified
when running the tests.
Change-Id: Ic89d1844d1fd8d00f62211a9b051a26ac34ee316
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
Support for startup node multi-arch variant selection through startup.conf.
This is to facilitate unit, functional testing and benchmarking of non-default
multi-arch variant node code path. Also added parameters to make test, to
specific using multi-arch variants in unit testing.
Type: improvement
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
Add these two packages to requirements.txt so that their version and
the version of their dependencies are pinned to limit the risk of
unexpected breakage.
Change-Id: If330404f2e840af3d2628f997ce406cd14e7e128
Type: fix
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
Make test framework python3 version independence.
Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
This patch addresses the functionality that is
missing for distributed make test source files.
In addition it extends the concept of colocating
test source code with functional source code
(eg. src/vcl/test). It also cleans up deficiencies
in the make test makefiles.
NOTE: Due to the way sphinx document tools work,
all test, all of the make test python code is
gathered using soft links into the directory:
.../build-root/build-test/src
Change summary:
- Remove 'make test' help details from
main makefile help output to reduce clutter
and redundant help output
- Move all generated build output to
.../build-root/build-test
- Move test_vcl.py as first usecase for
distributed core feature make test code
- Add test-wipe-all target and fix wipe targets
to remove all generated files
- Fix 'make test-doc' to generate module
documentation for all source files
- Remove unused targets in test/doc/Makefile
- Fix 'make test-shell'
- Fix test/ext Makefile to not generate
bogus output
Type: fix
Fixes: a43c93f855
Change-Id: Icc2ddb81d474081c3ede4548ecd7a0624651f62d
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Log the random seed used when running tests and provide means to re-use
it in a later run.
Type: feature
Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This change is designed to help the uninformed find the right way
to run extended tests by using the test-all[-debug] targets.
'make test EXTENDED_TESTS=y' fails to build as it has a dependency
on 'vom-install' which is conveniently included in test-all[-debug].
- clarify test-all[-debug] description and
make test-help description
- Also align indentation of make help output
Type: style
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: Ief54cc8a5af68c052aacb0d660237c5eb63451b5
- Relocate plugin tests for 'make test' into
src/plugins/*/test so that plugin test cases
are co-located with the plugin source code.
Type: refactor
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I503e6a43528e14981799b735fa65674155713f67
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Previously, the only option was to wipe/rebuild all the test dependencies.
Type: make
Change-Id: Ia95d6e800f67bef033dcf614dbfa249c3e43159d
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
New pip releases can break pip-tools. This commit pins the version of
pip used in the test virtualenv to prevent uncontrolled breakage.
This fixes the current issue in make test:
TypeError: __init__() got an unexpected keyword argument 'index_urls'
Change-Id: I3b7ecb31e651401ada0d357e2bf093c91c934565
Type: feature
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
This commit ensures that the tests always run with the exact same
version for all the Python dependencies. It uses pip-tools to achieve
that.
Our top-level dependencies are specified in the requirements.txt file.
From this file, pip-tools generates the requirements-{2,3}.txt file,
which pins all the versions of all the recursive dependencies, and is
used to install the packages in the test virtualenv.
To change or add a top-level dependency, update requirements.txt and run
make test as usual with python2 and python3. The requirements-{2,3}.txt
file will be updated and you can verify that nothing breaks. Then add
all requirements* files in your commit.
To refresh the python packages (i.e. get new versions of the recursive
dependencies, or of the dependencies that are not pinned in
requirements.txt), just run:
PYTHON=python2.7 make test-refresh-deps
PYTHON=python3.6 make test-refresh-deps
and this will update the requirements-{2,3}.txt files. Ideally we
should run this after each release.
Type: make
Change-Id: Ic533de3d06ec4019ff38f5231208da6f1025bfc7
Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
The unit and code coverage tests are boring. The rest of the patch
involves test and packet-generator infra cleanups.
Teach the "make test-xxx" family of targets to set the api test plugin
path correctly, to make "binary-api <api-message-name> <args>" debug
CLI commands work correctly in the "make test"
environment. Unfortunately involves both the top-level and test
Makefiles.
Add a minor pg cli feature, a CLI to manually set
s->sw_if_index[VLIB_TX].
Consider the case where one configures an interface with both a
device-input and an output feature. To test the output feature using
the pg, it's necessary to inject packets into the interface output
node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set
correctly. For example:
packet-generator new {
name tx
limit 15
size 128-128
interface local0 # rx: device input feature not configured on local0
tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX]
node loop0-output
data {
hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405
}
}
Fix a longstanding bug in the packet generator stream setup. Remove
kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places]
instead of using the stream value s->sw_if_index[VLIB_TX], and setting
THAT datum correctly.
Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed
Signed-off-by: Dave Barach <dave@barachs.net>
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file
is to be compressed by the test framework unless DEBUG=core is set.
Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Handle the case where buffer overflows.
Then SOCK_SEQPACKET assumption that multiple API messages
are not returned by recv() is broken. Use SOCK_STREAM for
API exchanges instead.
Add support for running tests over sockets.
make test SOCKET=1
Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821
Signed-off-by: Ole Troan <ot@cisco.com>
Since the stated policy regarding CLI commands is:
"Note that the debug CLI is a developer's tool
- no warranty express or implied - and that
we may choose not to fix debug CLI bugs.",
this change emits a deprecation warning whenever a test case calls a CLI command.
Change-Id: I91b30e86ae1b2fca36732837f36cdda762cdf458
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Use the installed version of pip. If a newer version of pip is needed,
it can be specified in requirements.txt. This is to improve idempotence
by providing some control over upstream changes.
Change-Id: I07b7651f8ddba6005bf85d11fef9e3343ee2dcd2
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Pip is installed via virtualenv. No need to reinstall it again.
Change-Id: I45bea1f9db841ca4309b763ab852623180937727
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
