68 Commits

Author SHA1 Message Date
Dave Wallace
7b8b465269 tests docs: update python3 venv packages
- Package update performed by
  1. updating pip, pip-tools, setuptools
  2. 'make test-refresh-deps' on ubuntu 22.04
  3. fixing 'make test' and 'make docs' issues
     on ubuntu 22.04
  4. 'make test-refresh-deps' on ubuntu 20.04
- Add dependency for 'make test-refresh-deps'
  to insure python venv is set up.
- Update of python formatter, black,
  caused reformating of 41 python code
  files.

Type: make

Change-Id: I7cafdf4b5189065ac57cb6b254937f6e0897a924
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-08-16 19:53:12 +00:00
Klement Sekera
47f3527108 tests: support for expected failures
- Add support for @unittest.expectedFailure decorator.

Type: improvement

Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I761751cda505e962225dc680b97c1fffa96f5176
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-05-20 02:34:14 +00:00
Klement Sekera
c7d50475ec tests: fix parallel runs skipping some tests
Fix corner case when a test would be skipped if it was not possible to
start it due to insufficient cpus available in the middle of the loop.

Type: fix
Change-Id: Ie4580685ff55688375d649d7009131d9fe1e4f33
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2023-04-25 15:46:42 +00:00
Klement Sekera
08c50e3b7a tests: support multiple filter expressions
Support multiple comma-delimited filter expressions,
e.g. to run both bfd and ip4 tests, it's now possible to do:

make test TEST=bfd,ip4

Same goes for wildcards, e.g.:

make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error

Type: improvement
Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2023-04-25 15:44:13 +00:00
Saima Yunus
c7f93b321d tests: move "venv" to "build-root" directory from "test" directory
Type: refactor

- refactored VPP test code to remove "ignore_path" variable
  from "discover_tests" function and "run_test" code
- configured VPP test makefile, config file, and 'run.sh' shell script
  to move "venv" directory from "test" dir to "build-root" dir

Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com>
Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
2022-08-16 20:27:37 +00:00
Klement Sekera
152a9b6165 tests: fix default failed dir setting
When running tests via run.sh, default setting of None would cause
failed directory symlink to appear in vpp workspace with an ugly name.
This patch places the symlink in temporary directory.

Type: fix
Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: Ic1715eba7ac1f82f71855e2aeb9b659d27bbb3af
2022-05-24 16:13:09 +00:00
Klement Sekera
d9b0c6fbf7 tests: replace pycodestyle with black
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.

Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.

Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.

Also, fixstyle-python is now available for automatic style formatting.

Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location.  This is required to simply the automated generation of
docker executor images in the CI.

Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-05-10 18:52:08 +00:00
Klement Sekera
687cf6faf2 tests: fix handling failed test case
Add missing parameter where required.

Type: fix
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I8cd7c31848836e3233cb79d1dd21884167db4354
2022-04-29 09:58:48 +00:00
Dmitry Valter
3ace4d62e2 tests: fix core file message
Prevent crashing on nonexistent VPP binary path class member when creating
testsuite core message.

Type: fix
Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec
Signed-off-by: Dmitry Valter <d-valter@yandex-team.ru>
Change-Id: Ib9b3dc8c69317e6561e5404bbdcbf672e417cbcd
2022-03-27 09:21:34 +00:00
Klement Sekera
b23ffd7ef2 tests: make tests less make dependent
Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.

Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-02-17 18:02:35 +00:00
Klement Sekera
558ceabc6c tests: cpus awareness
Introduce MAX_CPUS parameters to control maximum number of CPUs used by
VPP(s) during testing, with default value 'auto' corresponding to all
CPUs available.

Calculate test CPU requirements by taking into account the number of
workers, so a test requires 1 (main thread) + # of worker CPUs.

When running tests, keep track of both running test jobs (controlled by
TEST_JOBS parameter) and free CPUs. This then causes two limits in the
system - to not exceed number of jobs in parallel but also to not exceed
number of CPUs available.

Skip tests which require more CPUs than are available in system (or more
than MAX_CPUS) and print a warning message.

Type: improvement
Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-16 09:26:33 +00:00
Klement Sekera
e263685ac8 tests: support attaching to existing vpp
Introduce a new option DEBUG=attach to run a test against existing
already running vpp. A new target 'make test-start-gdb' will spawn VPP
in gdb for this purpose. Customization options explained in test-help.

Type: improvement
Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12 10:11:36 +00:00
Klement Sekera
eb506becd0 tests: remove shm handling
Not used anymore with socket transport.

Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
2021-03-17 11:40:16 +00:00
Klement Sekera
98d82ca04b nat: fix EI hairpinning thread safety
Avoid doing inter-thread reads without locks by doing a handoff before
destination address rewrite. Destination address is read from a session
which is possibly owned by a different thread. By splitting the work in
two parts with a handoff in the middle, we can do both in a thread safe
way.

Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I1c50d188393a610f5564fa230c75771a8065f273
2021-02-10 13:12:33 +00:00
Andrew Yourtchenko
06f328129a tests: add generalized tags for tests, use them for run-solo tests
We have accumulated several scenarios in prod or wishlists
where it would be useful to have a general infra to say yes/no
about a certain test, and potentially make decisions based on that,
for example:

- runs solo (aka 'time-dependent')
- (wishlist) part of quick smoke-test set
- (wishlist) intermittent failure unrelated to timing
- (wishlist) test broken with a multi-worker config in vpp

Refactor the current "run-solo" code to allow for this extension.

Type: test

Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-01-22 15:35:11 +00:00
Andrew Yourtchenko
4d084079cd tests: do not skip solo-run tests if running within a single job and all remaining tests are solo-run
There was a logic error, which results in not running solo tests
if TEST_JOBS=1, and we have finished running the regular test
and the remaining tests are all solo-run. They all get put into
that pile, but the new runners never start.

Solution: check the pile of solo-run jobs unconditionally.

Thanks to Klement Sekera for spotting the issue.

Type: fix
Change-Id: Iaecacbe4de44050cd8d0ff2d524559df99f2bae4
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-01-20 20:28:48 +00:00
Andrew Yourtchenko
a3b7c554c6 tests: "force solo" testcase support
Some of the tests are time-sensitive, and at present require a non-trivial
modification in order to run at high concurrency.

Without these modifications, they intermittently fail, and require
the test retries.

Rather than setting them to the extended tests and forgetting
about them, put them into a "solo" set, which gets run in a
single-threaded mode after the rest of the tests are done.

Mark a few of the tests that showed errors during TEST_JOBS=48
as forced-solo.

Also, give a better diagnostic if the testcase misses a docstring
needed to represent it in the diagnostic outputs.

Type: fix

Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-08-27 08:03:38 +00:00
Paul Vinciguerra
67a7749cac tests: fix TypeError in run_tests.py
Traceback (most recent call last):
  File "run_tests.py", line 886, in <module>
    exit_code, suites = parse_results(results)
  File "run_tests.py", line 732, in parse_results
    results_per_suite.print_results()
  File "run_tests.py", line 683, in print_results
    result.get_testcase_names(failed_test_id)
TypeError: 'NoneType' object is not iterable

Type: test

Change-Id: I2c5ae0b9e11be6bfb4490d9ce057db0b0a84c9bf
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-01-16 15:55:12 +00:00
Paul Vinciguerra
c701e57182 tests: don't prompt to launch gdb for sanity test case
Type: test

Change-Id: I4c54121b76b341381a819cee928c3c2455a83503
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-20 04:09:09 +00:00
Renato Botelho do Couto
ead1e536d6 misc: Fix python scripts shebang line
Type: fix

Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python'
as a valid shebang line.  It requires scripts to explicitly chose
between python2 or python3.

Change all to use python3 as suggested by Paul Vinciguerra.

Depends-On: https://gerrit.fd.io/r/23170

Signed-off-by: Renato Botelho do Couto <renato@netgate.com>
Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05 21:08:59 +00:00
Klement Sekera
45a95dd782 tests: support setting random seed
Log the random seed used when running tests and provide means to re-use
it in a later run.

Type: feature

Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-11-05 13:07:48 +00:00
Andrew Yourtchenko
42693521f6 tests: avoid test runner hanging on child test process join
In parallel test, the single process is spawning
a bunch of child processes running the tests,
and communicates to them. When the child process
signals that it has finished, the parent calls
child.join(). Sometimes this join never returns.
The result is a lot of defunct python processes,
and the test run just hangs.

I have seen this failure intermittently a fair bit
in a busy containerized environment, and by chance,
consistently reproduced it on a Thinkpad X280
with 8G of RAM and Ubuntu 19.04, which allowed
to diagnose it.

Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: If0a3110fc2d23e73d77c310d61c3ea90a2b53610
2019-11-05 09:06:15 +00:00
Klement Sekera
d743dffcd0 tests: fix TEST= filtering for python3
Type: fix

Change-Id: Iceb60e8a507120434cfc68758476ff5351b05f58
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-29 11:26:45 +00:00
Ole Troan
1556b3ad7d classify: tests to support python3
Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I8af6ee6e5239f0836978baa063a18a01c610279f
2019-10-21 18:32:58 +00:00
Paul Vinciguerra
025cd9c867 tests: add cpu/shm info to test runner output
Type: test
Depends-on: https://gerrit.fd.io/r/#/c/20484/

Change-Id: I140132cfcc4347035fe2bb9919f8e7923342940f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-24 17:41:42 +00:00
Paul Vinciguerra
0cbc71d783 tests: treat all truthy env vars the same way
Introduce a new class, that returns the truthiness of a env var.
Since an environment variable is just a string, it would normally
be true if not unset. The new class returns true when the env var is
set to a string that would be considered true.

Type: test
Depends-on: https://gerrit.fd.io/r/20484

Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09 07:16:16 +00:00
juraj.linkes
b5ef26d1ed tests: fix FORCE_FOREGROUND
Type: fix

Change-Id: I145f9984c1d944d5b7de0487e8b41ae4fdb95b6d
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-07-03 11:40:58 +02:00
Klement Sekera
f40ee3a206 make test: introduce COREDUMP_COMPRESS option
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file
is to be compressed by the test framework unless DEBUG=core is set.

Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-08 22:14:44 +00:00
Paul Vinciguerra
c0692a4496 tests: Speedup tests by 30%.
Baseline:
  2598325153 function calls (2598296959 primitive calls) in 2257.698 seconds

Post-changeset:
 1495427 function calls (1467233 primitive calls) in 1572.080 seconds

Change-Id: I191b68b29c9e0f19964bf0b8879ddf357c9cbd5c
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-18 14:08:34 +00:00
Naveen Joy
2cbf2fb05e make-test: Python pep8 compliance for sequences
For sequences, (strings, lists, tuples), use the fact that
empty sequences are false. A tiny change towards getting
full pep8 compliance, which will look good.
Yes: if not seq:
     if seq:
No:  if len(seq):
     if not len(seq):
(https://www.python.org/dev/peps/pep-0008/)

Change-Id: I6b565bef424f81afc66aba3b0041a5029a9f8954
Signed-off-by: Naveen Joy <najoy@cisco.com>
2019-03-07 19:40:04 +00:00
Paul Vinciguerra
38a4ec7342 make test: Add exception handling around subprocess.
This cleans up exception catching to identify oserrors.  By raising the
specific exception closer to the offending call, we get additional stack history
and can add clearer error logging to assist in troubleshooting.

Change-Id: I592e4d46844b822a816485bf54910f8daed92088
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-26 09:39:00 +00:00
juraj.linkes
46e8e91bca Fix interactive tests
When running tests interactively (i.e. not using child process) the
tests are split into small suites each containing tests from a
VPPTestCase, but only the first suite is run. Fix this by creatin a new
suite with all of these small suites in it and running that suite.

Change-Id: I0236f4a980d01765eb993f41d54d98640bb2cd39
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-01-11 18:18:48 +00:00
Paul Vinciguerra
defde0f870 Tests: Cleanup @skip decorator.
The runnning environment is static as of module load time,
so only evalute the conditions once at module load time.

Track-by: VPP-1518

Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20 12:43:38 +00:00
juraj.linkes
2eca70db95 Fix test results
In a rare case a failure can occur in a VppTestCase which hasn't been
part of the original TestSuite, such as in memif testcases. Fix the
reporting after test run in these cases.

Change-Id: Ib2a34fc195179605c22b95bd0e2c7cebca17012f
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-14 19:46:42 +00:00
Ole Troan
7f99183a20 Test framework: StringIO fixes for Python3
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py

Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877
Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-10 08:01:56 +00:00
juraj.linkes
e6b58cf86a Test reader thread hardening
The test reader thread could get stuck in case an unexpected Exception
was raised during test runs in parent thread. Catch the Exception, close
the thread and raise the Exception to avoid this.

Change-Id: I3a9298d593c334d96d04b6207d604b684572a2ac
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-07 10:45:09 +00:00
Klement Sekera
33177d698e make test: print TEST= values for failed tests
This prints the format strings required to rerun the failed test next to
the failures.

Change-Id: I973d8ae025f026129826356bce265b1e70086d2f
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-30 17:01:59 +00:00
Klement Sekera
b8c72a4a8d make test: create virtualenv under /test/
instead of using build-root, use /test/venv directory for virtualenv
similarly, don't pollute build-root with test-built binaries

Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-29 08:22:10 +00:00
juraj.linkes
abec0129b1 Split non-parallel testsuite
Split one big suite into smaller suites when not running tests in
parallel. This results in all tests being executed in one iteration.

Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-26 19:54:25 +00:00
Ole Trøan
5ba9159a15 Revert "Fix instance method call in test/framework.py"
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968.

Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71
Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-22 10:01:09 +00:00
Paul Vinciguerra
c32023b9fd Fix instance method call in test/framework.py
Instance method breaking encapsulation by calling a global fn.

Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-22 06:59:19 +00:00
juraj.linkes
dfb5f2aff3 Fix logging into multiple files
When running tests in one process, only one logger was used and each
testcase added its own file handler, which resulted in logs appearing in
multiple files. Fix this by restoring the creation of new loggers
for each testcase and only reuse the stream handler from parent process.

Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-09 23:33:59 +00:00
juraj.linkes
40dd73bcfa Setup, teardown, DEBUG=core, FAILFAST fixes
- Fixed a bug when an error occuring in tearDownClass would not result in
test being marked as failed
- Improved test results reporting in cases when an error occurs in setUpClass
and tearDownClass
- Fixed DEBUG=core when the core is produced in setUpClass or
tearDownClass
- Reworked DEBUG=core to always be handled after all tests have been
executed
- Fixed FAILFAST=1 for parallel test runs

Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-10 16:05:37 +00:00
Andrew Yourtchenko
d760f79cb8 test framework: add shell-style wildcard matching to filenames
Currently, one can either use "*" to denote all filenames,
or to give the specific file name to run the tests in.
This commit adds the possibility to run all tests
matching the shell wildcard, e.g. TEST="test_acl_plugin*"
will execute all ACL plugin testcases.

Change-Id: I9048a601958947c7b757c3dfd57b19cdd8a1e3c0
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2018-10-03 17:39:21 +00:00
juraj.linkes
cae64f896d VPP-1421: Reworked results gathering
The previous version sent the whole VPPTestResult object through pipe,
which uses pickle to transfer objects. Pickle does not support sending
any arbitrary objects and was causing issues. Now just a tuple of
(test_id, result) is sent.

Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-20 02:42:53 +00:00
Klement Sekera
6c7bda9112 fix TEST_JOBS=auto max jobs detection
Change-Id: I63f747e92d239e878550392df24fb7aac2b5a3e3
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-09-18 19:37:47 +00:00
juraj.linkes
721872e051 Fix test summary and retries
There was an issue when tests crashed and weren't properly retried.

Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-06 00:13:21 +00:00
juraj.linkes
5e2c54d029 Fix hanging test runner when child process dies
When fixing the test summary after a test run doesn't finish properly I
introduced a bug where child process which died would leave the whole
run hanging. This patch fixed the bug while still having the correct
test summary.

Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-30 13:57:39 +00:00
juraj.linkes
0219b8dfbf Fix logging to file and summary for crashed tests
Logging was not being logged to log.txt
Crashed tests were not reported properly when running tests in one
process

Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-25 00:40:54 +00:00
juraj.linkes
184870ac5a CSIT-1139: Implement parallel test execution
The implementation of parallel test execution in VPP Test Framework.
- VPPTestCase test methods are grouped together
- tests are running in separate processes
- VPP instances spawned by tests are assigned to different cores
- output from these processes is redirected through pipes and printed
out testcase by testcase
- TEST_JOBS env var is used to specify the number of parallel processes
- improved test summary
- a bit of code cleanup

Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-23 14:19:35 +00:00