Type: feature
this allows VPP to simulate linux tun devices.
Signed-off-by: Neale Ranns <neale@graphiant.com>
Change-Id: I3adf38b49a254804370f78edd5d275d192fd00a6
- Improve readability of log output with respect
to worker role
Type: test
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
Change-Id: I1a5787421b31483fa1a2558951568417dfc111d9
- Required to get VCLAppWorker stdin / stdout data
into log.txt when a test fails.
Type: test
Change-Id: I4db467986940f701750aec6d842a4d9aee22b4e6
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
A given interface counter (e.g rx) can be accessed via
/interfaces/<interface_name>/<counter_name>.
Same goes with nodes: /nodes/<node_name>/<counter_name>
As interfaces may contain '/' in their names,
these are replaced by '_' in symlinks
Also added 2 tests for the stat segment
Type: feature
Signed-off-by: Arthur de Kerhor <arthurdekerhor@gmail.com>
Change-Id: I27da252e7b3dc177815616ca46b5c200a456bf0f
Signed-off-by: Ole Troan <ot@cisco.com>
While running test:
==============================================================================
NAT44ED Test Case
==============================================================================
==>
==============================================================================
NAT44ED Test Case [main thread only]
==============================================================================
==============================================================================
NAT44ED Test Case [1 worker thread]
==============================================================================
==============================================================================
NAT44ED Test Case [4 worker threads]
==============================================================================
Type: improvement
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Change-Id: I61904b411ff1b58700564698079dc4c07cea5c5e
Introduce MAX_CPUS parameters to control maximum number of CPUs used by
VPP(s) during testing, with default value 'auto' corresponding to all
CPUs available.
Calculate test CPU requirements by taking into account the number of
workers, so a test requires 1 (main thread) + # of worker CPUs.
When running tests, keep track of both running test jobs (controlled by
TEST_JOBS parameter) and free CPUs. This then causes two limits in the
system - to not exceed number of jobs in parallel but also to not exceed
number of CPUs available.
Skip tests which require more CPUs than are available in system (or more
than MAX_CPUS) and print a warning message.
Type: improvement
Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Introduce a new option DEBUG=attach to run a test against existing
already running vpp. A new target 'make test-start-gdb' will spawn VPP
in gdb for this purpose. Customization options explained in test-help.
Type: improvement
Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This change allows one to inject multiple streams for different workers
on the same PG interface at the same time.
Type: improvement
Change-Id: I29d80369aabada261eda466e5a5d8d3518bb8bc8
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This module implement Python access to the VPP statistics segment. It
accesses the data structures directly in shared memory.
VPP uses optimistic locking, so data structures may change underneath
us while we are reading. Data is copied out and it's important to
spend as little time as possible "holding the lock".
Counters are stored in VPP as a two dimensional array.
Index by thread and index (typically sw_if_index).
Simple counters count only packets, Combined counters count packets
and octets.
Counters can be accessed in either dimension.
stat['/if/rx'] - returns 2D lists
stat['/if/rx'][0] - returns counters for all interfaces for thread 0
stat['/if/rx'][0][1] - returns counter for interface 1 on thread 0
stat['/if/rx'][0][1]['packets'] - returns the packet counter
for interface 1 on thread 0
stat['/if/rx'][:, 1] - returns the counters for interface 1 on all threads
stat['/if/rx'][:, 1].packets() - returns the packet counters for
interface 1 on all threads
stat['/if/rx'][:, 1].sum_packets() - returns the sum of packet counters for
interface 1 on all threads
stat['/if/rx-miss'][:, 1].sum() - returns the sum of packet counters for
interface 1 on all threads for simple counters
Type: refactor
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I1fe7f7c7d11378d06be8276db5e1900ecdb8f515
Signed-off-by: Ole Troan <ot@cisco.com>
VppTestCase now has vpp_worker_count property set to number of workers.
This can be overriden by child classes. Also overriden by
VPP_WORKER_CONFIG variable for legacy reasons.
Type: improvement
Change-Id: Ic328bacb9003ddf9e92815767653bd362aa7f086
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This fix the classify filter if we attach several different filters.
This also fix some issues with l3 and l4 parsing.
Type: fix
Change-Id: I9dc6c55049a3bbc0110d1097b40d9da27633626b
Signed-off-by: Benoît Ganne <bganne@cisco.com>
Allow to supply the external VPP worker config for tests which
do not specify the workers config explicitly, and use
the tags infra to flag those that need attention in this configuration.
This commit shows one example use of such a tag, there will be
a separate commit with the rest of the places needing it,
since that change is rather mechanical.
Thus, the assumption is that the test should by default be agnostic
of the VPP configuration, unless it explicitly specifies so.
Type: test
Change-Id: I3c0077e4e22a75cb9561fb98d3b783b93486b2be
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
We have accumulated several scenarios in prod or wishlists
where it would be useful to have a general infra to say yes/no
about a certain test, and potentially make decisions based on that,
for example:
- runs solo (aka 'time-dependent')
- (wishlist) part of quick smoke-test set
- (wishlist) intermittent failure unrelated to timing
- (wishlist) test broken with a multi-worker config in vpp
Refactor the current "run-solo" code to allow for this extension.
Type: test
Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Signed-off-by: Klement Sekera <ksekera@cisco.com>
The Centos-7 compiler had problems with vapi_cpp_test.cpp.
The Centos-8 compiler does not. This change adds back the test.
==============================================================================
VAPI test
==============================================================================
run C VAPI tests 1.55 OK
run C++ VAPI tests 0.44 OK
==============================================================================
TEST RESULTS:
Scheduled tests: 2
Executed tests: 2
Passed tests: 2
==============================================================================
Type: test
Change-Id: Ic6a655ae99eb3ba0e73b7feef4436a0ca5fd92be
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Some of the tests are time-sensitive, and at present require a non-trivial
modification in order to run at high concurrency.
Without these modifications, they intermittently fail, and require
the test retries.
Rather than setting them to the extended tests and forgetting
about them, put them into a "solo" set, which gets run in a
single-threaded mode after the rest of the tests are done.
Mark a few of the tests that showed errors during TEST_JOBS=48
as forced-solo.
Also, give a better diagnostic if the testcase misses a docstring
needed to represent it in the diagnostic outputs.
Type: fix
Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Tests currently expect the logger to be poked from run_tests.py.
The tests should run without any magic values. This change sets a default
null logger and removes the hasattr checks for the logger.
For reference, see: https://docs.python-guide.org/writing/logging/
Type: test
Change-Id: I98f953d73d12d00e74b59c94a0fb8c7a625b9c44
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Add a hook to src/vlibapi/api_shared.c to fuzz (screw up) binary API
messages, e.g. by xoring random data into them before processing. We
specifically exempt client connection messages, and inband debug CLI
messages. We step over msg_id, client index, client context, and
sw_if_index. Otherwise, "make test" vectors fail too rapidly to learn
anything.
The goal is to reduce the number of crashes caused to zero. We're
fairly close with this patch.
Add vl_msg_api_max_length(void *mp), which returns the maximum
plausible length for a binary API message.
Use it to hardern vl_api_from_api_to_new_vec(...) which takes an
additional argument - message pointer - so it can verify that
astr->length is sane. If it's not sane, return a u8 *vector of the
form "insane astr->length nnnn\0".
Verify array lengths in vl_api_dhcp6_send_client_message_t_handler(...)
and vl_api_dhcp6_pd_send_client_message_t_handler(...).
Add a fairly effective binary API fuzz hook to the unittest plugin,
and modify the "make test" framework.py to pass "api-fuzz { on|off }"
to enable API fuzzing: "make API_FUZZ=on TEST=xxx test-debug" or similar
Type: improvement
Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: I0157267652a163c01553d5267620f719cc6c3bde
Add vpp specific properties to ip addresses for use in the api.
.vapi_af -- returns [ADDRESS_IP4, ADDRESS_IP6]
.vapi_af_name -- returns the string ['ip4', 'ip6']
Update tests to demonstrate usage.
Type: feature
Change-Id: I43447a1522769d99f89debdc714c51700068d771
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Support for startup node multi-arch variant selection through startup.conf.
This is to facilitate unit, functional testing and benchmarking of non-default
multi-arch variant node code path. Also added parameters to make test, to
specific using multi-arch variants in unit testing.
Type: improvement
Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
Add a clib_time_t * argument to clib_timebase_init(...), to encourage
client code to share the vlib_main_t's clib_time_t object.
Display the current day / date in GMT via the "show time" debug CLI.
Fix the test framework so it processes the new "show time" output format.
Type: refactor
Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: I5e52d57eb164b7cdb6355362d520df6928491711
Mark a few code coverage tests appropriately:
@unittest.skipUnless(running_gcov_tests, "part of code coverage tests")
def test_my_coverage_unittest(self):
<etc>
Almost exactly like "make EXTENDED_TESTS=yes test".
Type: feature
Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: Ic6d0a097a608ba93442b3d88252f66f8e6805b97
Make test framework python3 version independence.
Type: fix
Signed-off-by: Ole Troan <ot@cisco.com>
Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
from threading.thread __init__:
This constructor should always be called with keyword arguments.
If a subclass overrides the constructor, it must make sure to invoke
the base class constructor (Thread.__init__()) before doing anything
else to the thread.
Type: test
Change-Id: Ifa89202e97053a4baf19e9a0ca0913430d5087a3
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Absolutely nothing good happens when we force the kernel to briefly
map and then unmap 16gb as vpp starts. Effect exacerbated when
TEST_JOBS = 20...40, and so forth.
Type: test
Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: Id8e3ce1763cad3a0891d5d6c8d2c1e3e610682d7
remove wait-loop on stats socket from test framework.
Type: refactor
Change-Id: I5bb95a7c597707a87f9d9a471215c4b4af1a2280
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
This change fixes the error message, which would previously report
mismatch on IPv6 layer instead of TCP layer.
Type: fix
Change-Id: Ie78779685362a420d155f928b3cdb341bc3fb8bc
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Type: test
This reverts commit 087c811b63.
Change-Id: Ica12597f27d5a75c33c12c2d27a227dc4866adcb
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
vapi.connect() only raises subclasses of vpp_papi.VPPIOError.
vapi.disconnect() doesn't raise any exceptions, removed uneeded try block.
Type: test
Change-Id: Ide6e6db298b205f14382065862e358757619a5ba
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
When running exec scripts, there can be a need to wait between statements.
Type: feature
Change-Id: I2a45b390697e09fc222358c9354f28e3368a06ba
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
Type: fix
Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python'
as a valid shebang line. It requires scripts to explicitly chose
between python2 or python3.
Change all to use python3 as suggested by Paul Vinciguerra.
Depends-On: https://gerrit.fd.io/r/23170
Signed-off-by: Renato Botelho do Couto <renato@netgate.com>
Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
Log the random seed used when running tests and provide means to re-use
it in a later run.
Type: feature
Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0
Signed-off-by: Klement Sekera <ksekera@cisco.com>
This reverts commit 4465fe1e30.
That commit makes *every* test to send SIGQUIT to VPP when it is done,
thereby triggering the coredump if there was none.
It is most surely not what we want.
Type: test
Change-Id: I3cc4c96f44c05527d586142f8feccc2f93c3e536
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
the IPv6 RA tests take timing into the account,
but the time inside VPP may go slightly differently compared
to the time inside the driving python thread,
if the machine running the tests is heavily loaded.
Make a sleep function which sleeps "on VPP time" and use it.
Change-Id: I3b34b0164f6e0ec7a619b92ee308089a4a8935e3
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Some testclasses (e.g. IPSec) have methods with the same names,
this causes a race when running in parallel to save the API and
move it into the test's directory - the name of the file has
only the test method name as a discriminator.
Saving two traces from two VPP instances "succeeds", in that
it silently overwrites the file. But only one mv operation
works - the other gives an error "file not found" and fails the test.
Solution: add the PID of VPP under test to the API dump filename,
this way the filenames become guaranteed unique and the race is avoided.
Change-Id: I1bb47ed73369fea0f1bfbf907588a51e2cabfb7f
Type: test
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
