26 Commits

Author SHA1 Message Date
Dave Wallace
8800f732f8 tests: refactor asf framework code
- Make framework.py classes a subset of asfframework.py classes
- Remove all packet related code from asfframework.py
- Add test class and test case set up debug output to log
- Repatriate packet tests from asf to test directory
- Remove non-packet related code from framework.py and
  inherit them from asfframework.py classes
- Clean up unused import variables
- Re-enable BFD tests on Ubuntu 22.04 and fix
  intermittent test failures in echo_looped_back
  testcases (where # control packets verified but
  not guaranteed to be received during test)
- Re-enable Wireguard tests on Ubuntu 22.04 and fix
  intermittent test failures in handshake ratelimiting
  testcases and event testcase
- Run Wiregard testcase suites solo
- Improve debug output in log.txt
- Increase VCL/LDP post sleep timeout to allow iperf server
  to finish cleanly.
- Fix pcap history files to be sorted by suite and testcase
  and ensure order/timestamp is correct based on creation
  in the testcase.
- Decode pcap files for each suite and testcase for all
  errors or if configured via comandline option / env var
- Improve vpp corefile detection to allow complete corefile
  generation
- Disable vm vpp interfaces testcases on debian11
- Clean up failed unittest dir when retrying failed testcases
  and unify testname directory and failed linknames into
  framwork functions

Type: test

Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-11-03 05:06:43 +00:00
Pratikshya Prasai
657bdf781a tests: initial asf framework refactoring for 'make test'
Type: refactor

Change-Id: I41455b759a5d302ad5c4247c13634c471e7d49a8
Signed-off-by: Pratikshya Prasai <pratikshyaprasai2112@gmail.com>
Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-11-09 21:22:33 -05:00
Klement Sekera
d9b0c6fbf7 tests: replace pycodestyle with black
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.

Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.

Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.

Also, fixstyle-python is now available for automatic style formatting.

Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location.  This is required to simply the automated generation of
docker executor images in the CI.

Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-05-10 18:52:08 +00:00
Ray Kinsella
064b77c742 build: remove unnecessary executable bits
Run 'find src -executable -type f', remove unnecessary executable bits from the
source tree.

Type: refactor

Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I070e22c8fb2ef8712bc3ea620727ee5ab3d9a9fb
2021-11-02 22:26:34 +00:00
Dave Wallace
d170681b24 tests docs: upgrade python packages
- Upgrade python package requirements for test & docs
- Clean up docs generation warnings
- Consolidate python requirements for docs in test
  requirements specs.
- Upgrade pip

Type: make

Change-Id: I74a3924b43ed93d15b32ec9f6fc41ed1ba95b69b
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-08-13 15:57:21 +00:00
Paul Vinciguerra
e061dad55e tests: py2 cleanup - remove subclassing of object
Type: refactor

Change-Id: I9096e3b473110350e1e8e5936e3c4c164f8969a7
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-12-07 08:17:10 +00:00
Paul Vinciguerra
090096bff3 tests: remove py2/py3 six compatability library
Type: test

Change-Id: Idb6b8169845e0239e639429ccfd02a683212b7e6
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-12-03 19:43:23 +00:00
Paul Vinciguerra
27860dd068 tests: remove aenum library
the aenum library was used to provide intflag functionality
for python versions earlier than 3.6

Type: test

Change-Id: I914d390ee5364f337006c1c59abafe4b9a3c1bde
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-12-03 00:46:03 -05:00
Renato Botelho do Couto
ead1e536d6 misc: Fix python scripts shebang line
Type: fix

Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python'
as a valid shebang line.  It requires scripts to explicitly chose
between python2 or python3.

Change all to use python3 as suggested by Paul Vinciguerra.

Depends-On: https://gerrit.fd.io/r/23170

Signed-off-by: Renato Botelho do Couto <renato@netgate.com>
Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05 21:08:59 +00:00
snaramre
d6df3acf5c tests: python3 version check for enum module
Type: fix
Change-Id: Ie210500fba741d3f047e8499957b5b233a78fa36
Signed-off-by: snaramre <snaramre@cisco.com>
2019-10-15 14:43:43 +00:00
Jakub Grajciar
546f955b3d memif: API cleanup
Use consistent API types.

memif_create now enables zero-copy by default.
Add no_zero_copy param to memif_create which
if set, disables zero copy.

Type: refactor

Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
Change-Id: I11df8b9212c40de179ee71dc9da14039b982ede5
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2019-09-09 14:29:48 +00:00
Jakub Grajciar
053204ab03 api: Cleanup APIs interface.api
Use of consistent API types for interface.api

Type: fix

Change-Id: I88206d7d0907cffd564031f73c9a996df2e5e21a
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2019-08-20 09:57:10 +00:00
Jakub Grajciar
7db35de508 tests: fix memif tests
remote_test.py - Remove 'ret' arg from _remote_exec(), so that the function
                 always reads the reply from the pipe. (fix unmatched request/reply)

memif_test.py - Don't register VppIpRoute to VppObjectRegistry.

Type: fix

Change-Id: I8a51e7ffd68df5f379534f5ddd5ec9367a89be32
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2019-06-26 10:10:05 +00:00
Neale Ranns
097fa66b98 fib: fib api updates
Enhance the route add/del APIs to take a set of paths rather than just one.
Most unicast routing protocols calcualte all the available paths in one
run of the algorithm so updating all the paths at once is beneficial for the client.
two knobs control the behaviour:
  is_multipath - if set the the set of paths passed will be added to those
                 that already exist, otherwise the set will replace them.
  is_add - add or remove the set

is_add=0, is_multipath=1 and an empty set, results in deleting the route.

It is also considerably faster to add multiple paths at once, than one at a time:

vat# ip_add_del_route 1.1.1.1/32 count 100000 multipath via 10.10.10.11
100000 routes in .572240 secs, 174751.80 routes/sec
vat# ip_add_del_route 1.1.1.1/32 count 100000 multipath via 10.10.10.12
100000 routes in .528383 secs, 189256.54 routes/sec
vat# ip_add_del_route 1.1.1.1/32 count 100000 multipath via 10.10.10.13
100000 routes in .757131 secs, 132077.52 routes/sec
vat# ip_add_del_route 1.1.1.1/32 count 100000 multipath via 10.10.10.14
100000 routes in .878317 secs, 113854.12 routes/sec

vat# ip_route_add_del 1.1.1.1/32 count 100000 multipath via 10.10.10.11 via 10.10.10.12 via 10.10.10.13 via 10.10.10.14
100000 routes in .900212 secs, 111084.93 routes/sec

Change-Id: I416b93f7684745099c1adb0b33edac58c9339c1a
Signed-off-by: Neale Ranns <neale.ranns@cisco.com>
Signed-off-by: Ole Troan <ot@cisco.com>
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-18 13:31:39 +00:00
Paul Vinciguerra
7f9b7f9f49 Tests Cleanup: Fix missing calls to setUpClass/tearDownClass.
Continuation/Part 2 of https://gerrit.fd.io/r/#/c/17092/

Change-Id: Id0122d84eaf2c05d29e5be63a594d5e528ee7c9a
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-04-10 14:37:40 +00:00
Paul Vinciguerra
8feeaff56f Typos. A bunch of typos I've been collecting.
Change-Id: I53ab8d17914e6563110354e4052109ac02bf8f3b
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-28 06:36:04 +00:00
Ole Trøan
3b0d7e42f6 Revert "API: Cleanup APIs interface.api"
This reverts commit e63325e3ca03c847963863446345e6c80a2c0cfd.

Allow time for CSIT to accommodate.

Change-Id: I59435e4ab5e05e36a2796c3bf44889b5d4823cc2
Signed-off-by: ot@cisco.com
2019-03-15 17:34:06 +00:00
Jakub Grajciar
e63325e3ca API: Cleanup APIs interface.api
Use of consistent API types for interface.api

Change-Id: Ieb54cebb4ac96b432a3f0b41596718aa2f34885b
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2019-03-15 10:24:22 +00:00
Paul Vinciguerra
d3a9be2ca0 Tests: remote_test.py. Private member test catching dunder names.
Private member check also catches __iter__, since it starts with '_'.

Fixes:
  Captured traceback:
  ~~~~~~~~~~~~~~~~~~
      b'Traceback (most recent call last):'
      b'  File "/vpp/test/test_memif.py", line 47, in tearDown'
      b'    remove_all_memif_vpp_config(self.remote_test)'
      b'  File "/vpp/test/vpp_memif.py", line 36, in remove_all_memif_vpp_config'
      b'    for d in dump:'
      b"TypeError: 'SerializableClassCopy' object is not iterable"
      b''

Change-Id: I6a3f3e0f2b1b2d0a2b97faa23bf542ff8f92de43
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11 08:09:58 +00:00
Paul Vinciguerra
f70cead5eb Tests: 'Fix' __del__ in test/remote_test.py.
This is not a fix as much as supressing a warning.
__del__ is not a destructor.

test/remote_test.py:385:13: F821 undefined name 'cls'
            cls.vpp.poll()
            ^
test/remote_test.py:386:16: F821 undefined name 'cls'
            if cls.vpp.returncode is None:
               ^
test/remote_test.py:387:17: F821 undefined name 'cls'
                cls.vpp.terminate()
                ^
test/remote_test.py:388:17: F821 undefined name 'cls'
                cls.vpp.communicate()

Change-Id: I6f0ecf3ae5dee7f279a4e25994cc1c49470bca26
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11 01:14:53 +00:00
Jakub Grajciar
0b1f8a7e69 Remote Test: fix serialization (dict, Enum)
Change-Id: Ia3f8616cbf424c7f2d29d16cf7a457be6cd1b33a
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2019-02-25 19:38:38 +00:00
Paul Vinciguerra
526ad04e01 test/remote_test.py: Fix missing import
Change-Id: I192c33d36f43ae2df0dda509e118f2b920d561ac
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-28 07:28:28 +00:00
Paul Vinciguerra
f1f2aa6bf9 VPP-1508 python3 tests: use six.iteritems
This replaces dictionary.iteritems() on Python 2 and dictionary.items() on Python 3.

Change-Id: I58a3ded7d284c59e28d484b0c285aac435bfc229
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-27 09:02:02 +00:00
Paul Vinciguerra
6c746172ef VPP-1508 Fix for bug introduced in tests on use of repr.
This fixes the change from https://gerrit.fd.io/r/#/c/16175/

...

======
Totals
======
Ran: 856 tests in 2123.0000 sec.
 - Passed: 667
 - Skipped: 157
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 32
Sum of execute time for each test: 1641.0832 sec.

==============
Worker Balance
==============
 - Worker 0 (416 tests) => 0:35:06.788054
 - Worker 1 (440 tests) => 0:31:13.814911
Test id                                                                                 Runtime (s)
--------------------------------------------------------------------------------------  -----------
test.test_vcl.VCLCutThruTestCase.test_ldp_cut_thru_bi_dir_nsock                         120.382
test.test_vcl.VCLThruHostStackGroupBTestCase.test_ldp_thru_host_stack_bi_dir_nsock       60.368
test.test_vcl.VCLIpv6ThruHostStackGroupBTestCase.test_ldp_thru_host_stack_bi_dir_nsock   60.356
test.test_igmp.TestIgmp.test_igmp_host                                                   47.411
test.test_reassembly.TestFIFReassembly.test_fif6                                         38.561
test.test_reassembly.TestFIFReassembly.test_fif4                                         37.045
test.test_gbp.TestGBP.test_gbp                                                           30.041
test.test_dhcp.TestDHCP.test_dhcp_proxy                                                  29.339
test.test_vcl.VCLCutThruTestCase.test_ldp_cut_thru_iperf3                                20.489
test.test_neighbor.ARPTestCase.test_arp                                                  19.004

Change-Id: Ic1565f14962f157d5041230de3aeeab0b85f21e1
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-27 09:00:51 +00:00
Paul Vinciguerra
00671cf9cf VPP-1508 python3 tests: python3 repr.
Use six.reprlib.  Uses repr for python 2 and reprlib for python 3.

Change-Id: Ia343a492d533bd511ed57166381e10a37e452d36
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-25 12:47:04 -08:00
Jakub Grajciar
b1be2a0bb8 memif: Memif Test Case
Change-Id: Ic0d5fc6ccbd496afcc870b908ef799af7c804c30
Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2018-09-27 18:31:53 +00:00