Log the random seed used when running tests and provide means to re-use
it in a later run.
Type: feature
Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
(cherry picked from commit 45a95dd782b91e9ae5665b5f95be4b6d7f99b879)
In parallel test, the single process is spawning
a bunch of child processes running the tests,
and communicates to them. When the child process
signals that it has finished, the parent calls
child.join(). Sometimes this join never returns.
The result is a lot of defunct python processes,
and the test run just hangs.
I have seen this failure intermittently a fair bit
in a busy containerized environment, and by chance,
consistently reproduced it on a Thinkpad X280
with 8G of RAM and Ubuntu 19.04, which allowed
to diagnose it.
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Change-Id: If0a3110fc2d23e73d77c310d61c3ea90a2b53610
(cherry picked from commit 42693521f6046997133c8f63bcfc9d615d96f69d)
Type: test
Depends-on: https://gerrit.fd.io/r/#/c/20484/
Change-Id: I140132cfcc4347035fe2bb9919f8e7923342940f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Introduce a new class, that returns the truthiness of a env var.
Since an environment variable is just a string, it would normally
be true if not unset. The new class returns true when the env var is
set to a string that would be considered true.
Type: test
Depends-on: https://gerrit.fd.io/r/20484
Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file
is to be compressed by the test framework unless DEBUG=core is set.
Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206
Signed-off-by: Klement Sekera <ksekera@cisco.com>
For sequences, (strings, lists, tuples), use the fact that
empty sequences are false. A tiny change towards getting
full pep8 compliance, which will look good.
Yes: if not seq:
if seq:
No: if len(seq):
if not len(seq):
(https://www.python.org/dev/peps/pep-0008/)
Change-Id: I6b565bef424f81afc66aba3b0041a5029a9f8954
Signed-off-by: Naveen Joy <najoy@cisco.com>
This cleans up exception catching to identify oserrors. By raising the
specific exception closer to the offending call, we get additional stack history
and can add clearer error logging to assist in troubleshooting.
Change-Id: I592e4d46844b822a816485bf54910f8daed92088
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
When running tests interactively (i.e. not using child process) the
tests are split into small suites each containing tests from a
VPPTestCase, but only the first suite is run. Fix this by creatin a new
suite with all of these small suites in it and running that suite.
Change-Id: I0236f4a980d01765eb993f41d54d98640bb2cd39
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
The runnning environment is static as of module load time,
so only evalute the conditions once at module load time.
Track-by: VPP-1518
Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
In a rare case a failure can occur in a VppTestCase which hasn't been
part of the original TestSuite, such as in memif testcases. Fix the
reporting after test run in these cases.
Change-Id: Ib2a34fc195179605c22b95bd0e2c7cebca17012f
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py
Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877
Signed-off-by: Ole Troan <ot@cisco.com>
The test reader thread could get stuck in case an unexpected Exception
was raised during test runs in parent thread. Catch the Exception, close
the thread and raise the Exception to avoid this.
Change-Id: I3a9298d593c334d96d04b6207d604b684572a2ac
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
This prints the format strings required to rerun the failed test next to
the failures.
Change-Id: I973d8ae025f026129826356bce265b1e70086d2f
Signed-off-by: Klement Sekera <ksekera@cisco.com>
instead of using build-root, use /test/venv directory for virtualenv
similarly, don't pollute build-root with test-built binaries
Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Split one big suite into smaller suites when not running tests in
parallel. This results in all tests being executed in one iteration.
Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968.
Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71
Signed-off-by: Ole Troan <ot@cisco.com>
Instance method breaking encapsulation by calling a global fn.
Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
When running tests in one process, only one logger was used and each
testcase added its own file handler, which resulted in logs appearing in
multiple files. Fix this by restoring the creation of new loggers
for each testcase and only reuse the stream handler from parent process.
Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
- Fixed a bug when an error occuring in tearDownClass would not result in
test being marked as failed
- Improved test results reporting in cases when an error occurs in setUpClass
and tearDownClass
- Fixed DEBUG=core when the core is produced in setUpClass or
tearDownClass
- Reworked DEBUG=core to always be handled after all tests have been
executed
- Fixed FAILFAST=1 for parallel test runs
Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
Currently, one can either use "*" to denote all filenames,
or to give the specific file name to run the tests in.
This commit adds the possibility to run all tests
matching the shell wildcard, e.g. TEST="test_acl_plugin*"
will execute all ACL plugin testcases.
Change-Id: I9048a601958947c7b757c3dfd57b19cdd8a1e3c0
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
The previous version sent the whole VPPTestResult object through pipe,
which uses pickle to transfer objects. Pickle does not support sending
any arbitrary objects and was causing issues. Now just a tuple of
(test_id, result) is sent.
Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
There was an issue when tests crashed and weren't properly retried.
Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
When fixing the test summary after a test run doesn't finish properly I
introduced a bug where child process which died would leave the whole
run hanging. This patch fixed the bug while still having the correct
test summary.
Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
Logging was not being logged to log.txt
Crashed tests were not reported properly when running tests in one
process
Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
The implementation of parallel test execution in VPP Test Framework.
- VPPTestCase test methods are grouped together
- tests are running in separate processes
- VPP instances spawned by tests are assigned to different cores
- output from these processes is redirected through pipes and printed
out testcase by testcase
- TEST_JOBS env var is used to specify the number of parallel processes
- improved test summary
- a bit of code cleanup
Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
This change adds handling of special case when setUpClass throws. In
this case TestResults receives a mock object called _ErrorHolder.
By parsing its description, we find test class name and use it to lookup
the test class in test suite to be able to add it to the list of failures
for re-running.
Change-Id: I656f21e38aa450fc567cdcbcf6e586967f947a64
Signed-off-by: Klement Sekera <ksekera@cisco.com>
The default config on Ubuntu 16.04.4 desktop results in truncated cores
when running make test which coredumps. Uninstalling the filter program
(apport) makes the corefiles normal size. Print a warning about that fact,
so the others potentially affected didn't have to wonder.
Change-Id: Iba4b0a2765a25100d6e24fd7f4de0e0339efd835
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
This change causes parent process to detect child crash in seconds
instead of waiting for timeout to pass.
Change-Id: Ib45e86b7fc97e687d99c554be4212aaeea781dcc
Signed-off-by: Klement Sekera <ksekera@cisco.com>
- Also change default coredump configuration from
"coredump-size unlimited" to "full-coredump"
Change-Id: Iefedc2636f2d9696b7575b34e91dd7be49f601fa
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Separating test discovery code to it's own script file has the
advantage of easily doing e.g. listing of all existing tests.
Change-Id: I80dc280263cc7e33e7e13cb0d48b39bf08ece24d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Run tests in a forked process with a set of pipes to communicate
keep-alives and overall result. This allows us to detect when e.g.
vpp dies mid-API call causing the test to hang waiting for response
(which will never come since vpp died).
Support setting a (per test case) TIMEOUT make test option to set timeout,
with a default timeout of 120 seconds.
Example - fail the test suite if any test-case fails to finish within
300s:
make TIMEOUT=300 test
Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717
Signed-off-by: Klement Sekera <ksekera@cisco.com>
env EXTERN_TESTS="/path/to/extra/tests" make test
causes to run the default test set and tests collected from
test_*.py files under subtree specified in EXTERN_TESTS.
Change-Id: I58c5471dd6010730278a5b47d4318737d920bc28
Signed-off-by: Klement Sekera <ksekera@cisco.com>
