There are a few tests that fail on ARM and thus we can't enable testing
in CI. Skip the failing tests until they're fixed so that we catch new
failures in CI (when we enable testing for the remaining tests).
Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
instead of using build-root, use /test/venv directory for virtualenv
similarly, don't pollute build-root with test-built binaries
Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Add the ability to specify a specific python library version
based on the interpreter/platform/etc.
Change-Id: I027acdf22ad839b5cff63b319f0aa100b0f336c8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
build extended binaries, vom & friends when test*-all targets are run,
remove unneeded test-ext target (which is semantically same as
test-debug-all)
Change-Id: Id710e061fdc93e286a4ed91be6a0cdcf75a2dafd
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Syslog protocol logging transport event messages across network over UDP
protocol based on RFC5426.
Change-Id: Ica74b40bcc2e6d0fbd41e9bf78e76395fbabab3c
Signed-off-by: Matus Fabian <matfabia@cisco.com>
The implementation of parallel test execution in VPP Test Framework.
- VPPTestCase test methods are grouped together
- tests are running in separate processes
- VPP instances spawned by tests are assigned to different cores
- output from these processes is redirected through pipes and printed
out testcase by testcase
- TEST_JOBS env var is used to specify the number of parallel processes
- improved test summary
- a bit of code cleanup
Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
- Ignore warnings W504 (newline after binary operator) which otherwise
occurs a significant number of times.
- Fix two instances of lines >79 chars.
Change-Id: I8cef56f8afc237187995e638e610c8c0554e2bb5
Signed-off-by: Chris Luke <chrisy@flirble.org>
Allow setting of VPP_PYTHON_PREFIX to alternate location
so the python prereqs can be installed into base image
Also added test-dep trigger to isolate dependency install
from actual test run
Change-Id: Ia80f5dbf71bc24eb46cd6586bcadd474ef822704
Signed-off-by: Ed Kern <ejk@cisco.com>
- many of the patches fd.io applies in test/patches/2.3.3 are now upstreamed in 2.4
- 2.4 adds support for IGMPv3 which is my main motivation for the upgrade
Change-Id: If2c0a524e3cba320b4a5d8cd07817c6ea2bf0c5a
Signed-off-by: Neale Ranns <nranns@cisco.com>
One of basic requirements from day one for maek test is that it should
run unpriviledged, so modifying system paremeters is unacceptable.
Disabling script and reassembly tests as they need to be done in
different way.
Change-Id: I063c73eb82f2ebd7499e8adb1574a9fd2475fb5b
Signed-off-by: Damjan Marion <damarion@cisco.com>
PEP8 has been deprecated and python users are being asked to migrate to
pycodestyle.
Change-Id: I52d5f7b2bf72156216a9966e8322ec58763f24d4
Signed-off-by: Marco Varlese <marco.varlese@suse.com>
This makes VAPI/VOM tests part of code coverage report.
Change-Id: I66511b0a59880c2a64daecd37088268bdf5e1269
Signed-off-by: Klement Sekera <ksekera@cisco.com>
New option TEST_DEBUG=1 turns on test framework debugging, which
currently consists of printing difference in allocated objects/memory
and also creates reference graphs for any unfreed VppPapiProvider/VPP
objects - these take a lot of memory and thus should be freed regularly.
Change-Id: I29db0c1341009d4b5c5df9222d14f3095883fd0f
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Verify that /dev/shm size is >= 512M, which should be enough for `make
test' needs. If the verification fails, try to enlarge it automatically.
This helps avoid docker vpp/make test crashes (seen as SIGBUS).
Change-Id: I4e23d59ff8bf8befc320fa229fb6c9bfeb809a8f
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This option allows one to toggle whether captured VPP stdout/stderr
output is printed inline (each line logged immediately after being
collected). Default setting is 0/no (behaviour same as before).
This allows one to see the output in case of vpp crashing during test
run even if test framework gets stuck waiting for shared memory mutex
(which crashed vpp can no longer unlock).
Complete stdout/stderr output is always printed as a block to the
logger after test finishes.
Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Allow skipping sanity when using DEBUG=gdb/gdbserver options to speed
up development process by supplying SANITY=no option to make test.
Change-Id: I6e619229f6866b22bdeb75654b4e750232ae22bd
Signed-off-by: Klement Sekera <ksekera@cisco.com>
- Fix invocation of compress_failed.sh
- Fix compress_failed to copy compressed results
files to $WORKSPACE/archives and return failure
exit code.
Failed test case data will be copied to logs.fd.io
and found in the archives/<make test data dir>-FAILED
directory in the build log link in the
vpp-verify-master-ubuntu1604 jenkins job page.
For example:
https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1604/7353/archives/
Change-Id: Ife9a0737115e69c0a8441e3bb0133af1528d909b
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Compress files in temporary directories of failed tests and symlink
the directories under /tmp/vpp-failed-unittests location - preparation
for jenkins archivation. Automatically cleanup the directory at start
of test run.
The compression is performed only when environment variable
COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1".
This is set in verify target, but left unset by default, so when
invoking make test by hand, files won't be compressed.
Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Run tests in a forked process with a set of pipes to communicate
keep-alives and overall result. This allows us to detect when e.g.
vpp dies mid-API call causing the test to hang waiting for response
(which will never come since vpp died).
Support setting a (per test case) TIMEOUT make test option to set timeout,
with a default timeout of 120 seconds.
Example - fail the test suite if any test-case fails to finish within
300s:
make TIMEOUT=300 test
Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This change introduces a wrapper script which kills all processes in
the same process group as itself (with the exception of the script).
Using this script to run the unit tests should prevent stale processes
left behind in some cases (e.g. when test framework crashes).
Change-Id: If3b9201c06b87fa6be095721436893207d09b5e4
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Introduce faulthandler to print stack trace to stderr on python crash.
Don't disable automatic garbage collection to decrease the chances of
running out of memory.
Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Allow using custom python interpreter via 'PYTHON' env variable.
E.g. env PYTHON=pypy make test.
Get latest pip automatically to properly function in environments,
where old pip is available and a proxy is required. This allows
testing pypy in a centos vagrant-provided vm.
Change-Id: I67a658fc927303468cc67f0ac192317ca2907625
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Damjan Marion <damarion@cisco.com>
Add code which checks if vpp doesn't crash/exit immediately after
startup to aid debugging stuff like mistyped graph node name or so.
Refuse to run tests if the vpp is unable to start, complain loudly
and print vpp's stderr at critical log level if this happens to make
spotting these problems in jenkins easy.
Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
env EXTERN_TESTS="/path/to/extra/tests" make test
causes to run the default test set and tests collected from
test_*.py files under subtree specified in EXTERN_TESTS.
Change-Id: I58c5471dd6010730278a5b47d4318737d920bc28
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Check if vpp_papi is importable before running the tests to avoid
confusing python crashes.
Change-Id: I6adf406e353bf381d590f2ef988a1ea79b95cf37
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This starts a bash with the same environment as the test framework
uses, allowing easy debugging.
Change-Id: I956deda913b73dae5b1e1976417834ae4731f88a
Signed-off-by: Klement Sekera <ksekera@cisco.com>
