- Make framework.py classes a subset of asfframework.py classes
- Remove all packet related code from asfframework.py
- Add test class and test case set up debug output to log
- Repatriate packet tests from asf to test directory
- Remove non-packet related code from framework.py and
inherit them from asfframework.py classes
- Clean up unused import variables
- Re-enable BFD tests on Ubuntu 22.04 and fix
intermittent test failures in echo_looped_back
testcases (where # control packets verified but
not guaranteed to be received during test)
- Re-enable Wireguard tests on Ubuntu 22.04 and fix
intermittent test failures in handshake ratelimiting
testcases and event testcase
- Run Wiregard testcase suites solo
- Improve debug output in log.txt
- Increase VCL/LDP post sleep timeout to allow iperf server
to finish cleanly.
- Fix pcap history files to be sorted by suite and testcase
and ensure order/timestamp is correct based on creation
in the testcase.
- Decode pcap files for each suite and testcase for all
errors or if configured via comandline option / env var
- Improve vpp corefile detection to allow complete corefile
generation
- Disable vm vpp interfaces testcases on debian11
- Clean up failed unittest dir when retrying failed testcases
and unify testname directory and failed linknames into
framwork functions
Type: test
Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.
Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.
Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.
Also, fixstyle-python is now available for automatic style formatting.
Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location. This is required to simply the automated generation of
docker executor images in the CI.
Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Prevent crashing on nonexistent VPP binary path class member when creating
testsuite core message.
Type: fix
Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec
Signed-off-by: Dmitry Valter <d-valter@yandex-team.ru>
Change-Id: Ib9b3dc8c69317e6561e5404bbdcbf672e417cbcd
Allow entering a test name on stack trace window with STEP=y option
instead of a number. This allows to run a whole suite and skip all tests
until a particular test is hit.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I23e45f8022b82545365b8921390e0e106e02b39c
- Move Exception into same module as TestCase.
- Move the error reporting logic inside the error.
- Allows testing of the returncode and signal_name for tests to consume.
- Fix the signal reporting code:
VppDiedError: VPP subprocess died unexpectedly with returncode -6 [unknown].
displays as:
VppDiedError: VPP subprocess died unexpectedly with return code: -6 [SIGABRT].
Type: test
Change-Id: I8488ab318a596c9b737308829cedfb7e96e57302
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Without this patch, the hook module raises a ValueError
when tests are run using python3. This patch updates the
exception being caught by ipaddress to ValueError.
Change-Id: I5e11e292a05ddf350fc04ebaf19cfd7dad2bd9d0
Signed-off-by: Naveen Joy <najoy@cisco.com>
File "/vpp/test/hook.py", line 84, in on_crash
self.testcase.vpp_bin, core_path)
AttributeError: 'PollHook' object has no attribute 'testcase'
Change-Id: I84d9d86a5c6a5769a43a91cf23ce8a1141f7cd12
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
This cleans up exception catching to identify oserrors. By raising the
specific exception closer to the offending call, we get additional stack history
and can add clearer error logging to assist in troubleshooting.
Change-Id: I592e4d46844b822a816485bf54910f8daed92088
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
If you pass in a non-unicode 4-byte ipv6 address to ip_address,
ipaddress interprets this as an IPv4Address.
Under python2, ip_address interprets 'a7::' as a packed ipv4:
97.55.58.58
You can test with:
---
import ipaddress
try:
text_type = unicode
except NameError:
text_type = str
addr = ipaddress.ip_address('a7::')
print(addr)
---
Change-Id: I06c561e0ab7315869cc89d0bb08c05e743a90982
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
- Fixed a bug when an error occuring in tearDownClass would not result in
test being marked as failed
- Improved test results reporting in cases when an error occurs in setUpClass
and tearDownClass
- Fixed DEBUG=core when the core is produced in setUpClass or
tearDownClass
- Reworked DEBUG=core to always be handled after all tests have been
executed
- Fixed FAILFAST=1 for parallel test runs
Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
The implementation of parallel test execution in VPP Test Framework.
- VPPTestCase test methods are grouped together
- tests are running in separate processes
- VPP instances spawned by tests are assigned to different cores
- output from these processes is redirected through pipes and printed
out testcase by testcase
- TEST_JOBS env var is used to specify the number of parallel processes
- improved test summary
- a bit of code cleanup
Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c
Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
The default config on Ubuntu 16.04.4 desktop results in truncated cores
when running make test which coredumps. Uninstalling the filter program
(apport) makes the corefiles normal size. Print a warning about that fact,
so the others potentially affected didn't have to wonder.
Change-Id: Iba4b0a2765a25100d6e24fd7f4de0e0339efd835
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Run tests in a forked process with a set of pipes to communicate
keep-alives and overall result. This allows us to detect when e.g.
vpp dies mid-API call causing the test to hang waiting for response
(which will never come since vpp died).
Support setting a (per test case) TIMEOUT make test option to set timeout,
with a default timeout of 120 seconds.
Example - fail the test suite if any test-case fails to finish within
300s:
make TIMEOUT=300 test
Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This is a work-in-progress basic BFD session handling. Only
asynchronous mode is supported at the moment. Setting the session flags
doesn't work.
Change-Id: Idba27f721b5c35be5a66a6d202a63d23ff7ecf6f
Signed-off-by: Klement Sekera <ksekera@cisco.com>
If VPP process dies right at start, do a quick detection instead of being
stuck in the connect timeout (60s).
Change-Id: I41675181635fb81a6a7d93fbf652480a16bf78a0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Opening debug-cli only makes sense if there is time for user to use it,
so either the run must be debugged or stepped. Only open the debug-cli
in these cases.
Change-Id: Ied276071797a549880d730cda43c59230a412efe
Signed-off-by: Klement Sekera <ksekera@cisco.com>
