Type: test
There are random failures in the wireguard test cases that are not related to concurrency issues.
The root cause is a retry of the handshake initiation after (REKEY_TIMEOUT + JITTER) ms, where JITTER is a random value between 0 and 333 ms.
Solution: Add a filter parameter for the `send_and_expect` method of the vpptestcase.
This filter allows for excluding unexpected handshake initiation packets when the responder sends two packets (with `message_type = 1` and `message_type = 2`),while only a single packet (with `message_type = 2`) is expected.
Change-Id: I62816931fc1b85e2202f3d36eb6c2a23714644d5
Signed-off-by: Ivan Ivanets <iivanets@cisco.com>
- Required for Ubuntu 24.04 LTS jobs
- temporarily disable TestIpsecEsp1 and
TestIpsecAhAll tests until a patch can
be added to fix them
Type: test
Change-Id: I1ae7b170117182c3252629bbbb770775e2c496c9
Signed-off-by: Benoît Ganne <bganne@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Make framework.py classes a subset of asfframework.py classes
- Remove all packet related code from asfframework.py
- Add test class and test case set up debug output to log
- Repatriate packet tests from asf to test directory
- Remove non-packet related code from framework.py and
inherit them from asfframework.py classes
- Clean up unused import variables
- Re-enable BFD tests on Ubuntu 22.04 and fix
intermittent test failures in echo_looped_back
testcases (where # control packets verified but
not guaranteed to be received during test)
- Re-enable Wireguard tests on Ubuntu 22.04 and fix
intermittent test failures in handshake ratelimiting
testcases and event testcase
- Run Wiregard testcase suites solo
- Improve debug output in log.txt
- Increase VCL/LDP post sleep timeout to allow iperf server
to finish cleanly.
- Fix pcap history files to be sorted by suite and testcase
and ensure order/timestamp is correct based on creation
in the testcase.
- Decode pcap files for each suite and testcase for all
errors or if configured via comandline option / env var
- Improve vpp corefile detection to allow complete corefile
generation
- Disable vm vpp interfaces testcases on debian11
- Clean up failed unittest dir when retrying failed testcases
and unify testname directory and failed linknames into
framwork functions
Type: test
Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Package update performed by
1. updating pip, pip-tools, setuptools
2. 'make test-refresh-deps' on ubuntu 22.04
3. fixing 'make test' and 'make docs' issues
on ubuntu 22.04
4. 'make test-refresh-deps' on ubuntu 20.04
- Add dependency for 'make test-refresh-deps'
to insure python venv is set up.
- Update of python formatter, black,
caused reformating of 41 python code
files.
Type: make
Change-Id: I7cafdf4b5189065ac57cb6b254937f6e0897a924
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
- Make error message more human readable.
Type: improvement
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: Iefc276b3a85ff82b927028a72bb91ed87ebd04ba
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Rename extra_vpp_punt_config to a more generic name extra_vpp_config to
better fit its purpose. It's fit for general use and already used that
way by quic and vcl tests anyway.
Type: refactor
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: Ib0a5789b0dbb3a8c3cae654dea4e32ac5e56dd41
Support running tests with `--tmp-dir` on a filesystem different from /tmp.
os.rename withs only within a single FS whereas shutil.move works accross
different filesystems.
Type: improvement
Signed-off-by: Dmitry Valter <d-valter@yandex-team.ru>
Change-Id: I5371f5d75386bd2b82a75b3e6c1f2c850bc62356
Fool-proof assert_checksum_valid so that one does not verify checksum on
wrong layer (because of how scapy internally works).
Make assert_packet_checksums_valid start checksum checking at inner
layers and outwards to make it more obvious where the error is. With old
behaviour, if one received an ICMP packet carrying a truncated TCP
packet, an error would be raised for ICMP checksum, as that one would be
the first to be wrong after recalculating all packet checksums, while
the real issue is TCP header being truncated and thus unsuitable for use
with this function.
Type: improvement
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I39a2b50ec5610f969cfde9796416ee3a50ae0ba3
Usage:
test/run.py -r -t {test_filter}
Instead of starting a new instance of VPP, when the -r argument
is provided, test is run against a running VPP instance. Optionally,
one can also set the VPP socket directory using the -d
argument. The default location for socket files is
/var/run/user/${uid}/vpp and /var/run/vpp if VPP is started
as root.
Type: improvement
Change-Id: I05e57a067fcb90fb49973f8159fc17925b741f1a
Signed-off-by: Naveen Joy <najoy@cisco.com>
Drop pycodestyle for code style checking in favor of black. Black is
much faster, stable PEP8 compliant code style checker offering also
automatic formatting. It aims to be very stable and produce smallest
diffs. It's used by many small and big projects.
Running checkstyle with black takes a few seconds with a terse output.
Thus, test-checkstyle-diff is no longer necessary.
Expand scope of checkstyle to all python files in the repo, replacing
test-checkstyle with checkstyle-python.
Also, fixstyle-python is now available for automatic style formatting.
Note: python virtualenv has been consolidated in test/Makefile,
test/requirements*.txt which will eventually be moved to a central
location. This is required to simply the automated generation of
docker executor images in the CI.
Type: improvement
Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Make make test-start-vpp-in-gdb work again.
Fix incorrect temp directory when using DEBUG=attach.
Type: fix
Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec
Change-Id: Ie98b637acbbe0221606ccdc7b54f63885e5951a4
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Type: improvement
For error conditions, such as TTL expired, dest unreach, etc, Rate limit the sending of ICMP error messages.
The rate limiting is done based on src,dst IP address of the received packet.
the rate limit has been chosen, somewhat arbitrarily, to be 1e-3. This is the same limit as the ARP throttling.
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: I4a0b791cde8c941a9bf37de6aa5da56779d3cef4
Raise a new UnexpectedPacketErrror, when a packet is captured
unexpectedly. This pretty-prints a terse description of said packet.
Type: improvement
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: Ibac19fc5bbd82a150fec3c90940a37af6344fd4f
Add support for inline packet counter verification to send_and_* functions.
Diff dictionary is a dictionary of dictionaries of interesting stats:
diff_dictionary =
{
"err" : { '/error/counter1' : 4, },
sw_if_index1 : { '/stat/segment/counter1' : 5,
'/stat/segment/counter2' : 6,
},
sw_if_index2 : { '/stat/segment/counter1' : 7,
},
}
It describes a per sw-if-index diffset, where each key is stat segment
path and value is the expected change for that counter for sw-if-index.
Special case string "err" is used for error counters.
This then allows more precise packet counter verification by first
defining a "zero" dictionary, e.g. for ED NAT:
cls.no_diff = StatsDiff({
pg.sw_if_index: {
'/nat44-ed/in2out/fastpath/tcp': 0,
'/nat44-ed/in2out/fastpath/udp': 0,
'/nat44-ed/in2out/fastpath/icmp': 0,
'/nat44-ed/in2out/fastpath/drops': 0,
'/nat44-ed/in2out/slowpath/tcp': 0,
'/nat44-ed/in2out/slowpath/udp': 0,
'/nat44-ed/in2out/slowpath/icmp': 0,
'/nat44-ed/in2out/slowpath/drops': 0,
'/nat44-ed/in2out/fastpath/tcp': 0,
'/nat44-ed/in2out/fastpath/udp': 0,
'/nat44-ed/in2out/fastpath/icmp': 0,
'/nat44-ed/in2out/fastpath/drops': 0,
'/nat44-ed/in2out/slowpath/tcp': 0,
'/nat44-ed/in2out/slowpath/udp': 0,
'/nat44-ed/in2out/slowpath/icmp': 0,
'/nat44-ed/in2out/slowpath/drops': 0,
}
for pg in cls.pg_interfaces
})
and then to specify only changed counters directly when calling
one of send_and_* functions:
self.send_and_assert_no_replies(
self.pg0, pkts, msg="i2o pkts",
stats_diff=self.no_diff | {
"err": {
'/err/nat44-ed-in2out-slowpath/out of ports': len(pkts),
},
self.pg0.sw_if_index: {
'/nat44-ed/in2out/slowpath/drops': len(pkts),
},
}
)
operator | is overloaded by StatsDiff class to perform a deep merge operation,
so in above case, dictionaries for "err" and self.pg0.sw_if_index do not
overwrite whole sub-dictionaries, rather the contents are merged,
assuring that all the remaining counters are verified to be zero.
Type: improvement
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I2b87f7bd58a7d4b34ee72344e2f871b2f372e2d9
Some tests might want to keep vpp config in between test functions.
Add a flag to allow that.
Type: improvement
Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
Change-Id: I90243cd667dce922b43b381f3d52f4ac0f6bf3a7
Type: test
there were multiple instances of send_and_expect_load_balancing
and a send_and_expect_one_itf which has the same functionality as send_and_expect_one.
Put one implementation of both in framework.py (where the other send_and_X functions reside).
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: I7f629d440220bee29368067f475059322e1134f7
Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.
Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Mark broken test suites using @tag_fixme_asan.The main issue
is that some tests do not pass with ASan. These will need to
be fixed, but it takes time and in the meantime, new issues are
introduced that trip up ASan. When this tag is set and Asan
is enabled, failing ASan tests are skipped, so incremental
fixes can pass the CI
Type: improvement
Change-Id: I02602eb74234c25a4c701279e14704b81d4c5b71
Signed-off-by: Naveen Joy <najoy@cisco.com>
Implement proper state machine based on above RFCs. ACKs to SYNs/FINs
are no longer required/tracked. This is more friendly to peers and
accounts for lost packets and retransmits.
This change also means that all traffic is translated and forwarded
while in transitory timeout, which helps delivering e.g. retransmitted
FINs, FINACKs and other messages.
Also support reopening a session in transitory timeout after seeing both
FINs by seeing both SYNs again. This helps quick connection
reestablishment if the peers want to.
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Miklos Tirpak <miklos.tirpak@gmail.com>
Change-Id: Ibf521c79463472db97e593bfa02b32b4a06dfd2a
If a fragment arrives with fragment offset = 0 and M = 0, it means that
this is actually a complete packet and per RFC 8200, it should be
treated independently from other fragments. This patch does that.
Fragmentation header is stripped and fragment is forwarded irregardles
of other existing reassemblies in case of full reassembly and treated
the same way as regular packet in shallow virtual reassembly.
Type: improvement
Change-Id: If3322d5e3160cd755b8465a642702a9166d46cc2
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Put static mappings in flow hash, drop existing hash tables used for
static mappings. Drop refcount variables and use hash table as a single
point of truth. Allow creating a static mapping conflicting with dynamic
mapping, which will take precedence after dynamic mapping is freed, so
that the existing flow can finish transferring data.
Type: fix
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: Idfde8efabc09971be38921d4b0ca5ccf4e9fe412
Put static mappings in flow hash, drop existing hash tables used for
static mappings. Drop refcount variables and use hash table as a single
point of truth. Allow creating a static mapping conflicting with dynamic
mapping, which will take precedence after dynamic mapping is freed, so
that the existing flow can finish transferring data.
Type: fix
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: Ieeba691d83a83887d0a0baccd5f3832f66126096
Type: test
Large and nested unit tests to test AVX-512 optimized versions of the classify
hash and match algorithims.
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: Ie423fee5e0fd1cb4bdf3bec8e0230a5f7cfc75fc
Type: feature
this allows VPP to simulate linux tun devices.
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: I3adf38b49a254804370f78edd5d275d192fd00a6
- Improve readability of log output with respect
to worker role
Type: test
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I1a5787421b31483fa1a2558951568417dfc111d9
- Required to get VCLAppWorker stdin / stdout data
into log.txt when a test fails.
Type: test
Change-Id: I4db467986940f701750aec6d842a4d9aee22b4e6
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
A given interface counter (e.g rx) can be accessed via
/interfaces/<interface_name>/<counter_name>.
Same goes with nodes: /nodes/<node_name>/<counter_name>
As interfaces may contain '/' in their names,
these are replaced by '_' in symlinks
Also added 2 tests for the stat segment
Type: feature
Signed-off-by: Arthur de Kerhor <arthurdekerhor@gmail.com>
Change-Id: I27da252e7b3dc177815616ca46b5c200a456bf0f
Signed-off-by: Ole Troan <ot@cisco.com>
While running test:
==============================================================================
NAT44ED Test Case
==============================================================================
==>
==============================================================================
NAT44ED Test Case [main thread only]
==============================================================================
==============================================================================
NAT44ED Test Case [1 worker thread]
==============================================================================
==============================================================================
NAT44ED Test Case [4 worker threads]
==============================================================================
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I61904b411ff1b58700564698079dc4c07cea5c5e
Introduce MAX_CPUS parameters to control maximum number of CPUs used by
VPP(s) during testing, with default value 'auto' corresponding to all
CPUs available.
Calculate test CPU requirements by taking into account the number of
workers, so a test requires 1 (main thread) + # of worker CPUs.
When running tests, keep track of both running test jobs (controlled by
TEST_JOBS parameter) and free CPUs. This then causes two limits in the
system - to not exceed number of jobs in parallel but also to not exceed
number of CPUs available.
Skip tests which require more CPUs than are available in system (or more
than MAX_CPUS) and print a warning message.
Type: improvement
Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Introduce a new option DEBUG=attach to run a test against existing
already running vpp. A new target 'make test-start-gdb' will spawn VPP
in gdb for this purpose. Customization options explained in test-help.
Type: improvement
Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This change allows one to inject multiple streams for different workers
on the same PG interface at the same time.
Type: improvement
Change-Id: I29d80369aabada261eda466e5a5d8d3518bb8bc8
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This module implement Python access to the VPP statistics segment. It
accesses the data structures directly in shared memory.
VPP uses optimistic locking, so data structures may change underneath
us while we are reading. Data is copied out and it's important to
spend as little time as possible "holding the lock".
Counters are stored in VPP as a two dimensional array.
Index by thread and index (typically sw_if_index).
Simple counters count only packets, Combined counters count packets
and octets.
Counters can be accessed in either dimension.
stat['/if/rx'] - returns 2D lists
stat['/if/rx'][0] - returns counters for all interfaces for thread 0
stat['/if/rx'][0][1] - returns counter for interface 1 on thread 0
stat['/if/rx'][0][1]['packets'] - returns the packet counter
for interface 1 on thread 0
stat['/if/rx'][:, 1] - returns the counters for interface 1 on all threads
stat['/if/rx'][:, 1].packets() - returns the packet counters for
interface 1 on all threads
stat['/if/rx'][:, 1].sum_packets() - returns the sum of packet counters for
interface 1 on all threads
stat['/if/rx-miss'][:, 1].sum() - returns the sum of packet counters for
interface 1 on all threads for simple counters
Type: refactor
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I1fe7f7c7d11378d06be8276db5e1900ecdb8f515
Signed-off-by: Ole Troan <ot@cisco.com>
VppTestCase now has vpp_worker_count property set to number of workers.
This can be overriden by child classes. Also overriden by
VPP_WORKER_CONFIG variable for legacy reasons.
Type: improvement
Change-Id: Ic328bacb9003ddf9e92815767653bd362aa7f086
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This fix the classify filter if we attach several different filters.
This also fix some issues with l3 and l4 parsing.
Type: fix
Change-Id: I9dc6c55049a3bbc0110d1097b40d9da27633626b
Signed-off-by: Benoît Ganne <bganne@cisco.com>
