163 Commits

Author SHA1 Message Date
Klement Sekera
e1204e5049 tests: fix centos detection
Type: fix
Fixes: defde0f870

Change-Id: Ib81e3610bd0fc22901a902dc76692f1951ead2b8
Signed-off-by: Klement Sekera <ksekera@cisco.com>
(cherry picked from commit 3a350702ec24d1bde0483da1e946015c2b81e0c4)
2019-09-28 20:23:04 +00:00
Klement Sekera
69bf8326f1 ip: reassembly: send packet out on correct worker
Note which worker received fragment with offset zero and use this worker
to send out the reassembled packet.

Type: fix
Change-Id: I1d3cee16788db3b230682525239c0100d51dc380
Signed-off-by: Klement Sekera <ksekera@cisco.com>
(cherry picked from commit 630ab5846bceddf8d663e9f488a2dc0378949827)
2019-09-25 22:05:36 +00:00
Neale Ranns
00625a64f4 tests: Split IPSec ESP into parameterized tests per engine
Type: feature

Change-Id: Icb1bd3fce768aebf8919c63a104f771ca7fa1d6f
Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-07-31 12:55:46 +00:00
Mohsin Kazmi
22e9cfd760 pg: add GSO support
Type: feature

Change-Id: I72676495a85fbecc946aa266a75234cce70c3a5e
Signed-off-by: Mohsin Kazmi <sykazmi@cisco.com>
2019-07-28 14:20:06 +00:00
Paul Vinciguerra
f7457521b6 tests: fix error in VppDiedError exception
Discovered running test-debug job in CI.

- fix missing paren () around format value.
Type: test

Change-Id: Iebddd3035a435f8ad1cb1d6fa4e8e8c2d4ddaf96
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-14 17:13:20 +00:00
Paul Vinciguerra
bfb9fe3461 tests: remove unused class attribute in VppTestCase
Identified in post-merge review.

Type: test

Change-Id: I46e19285479437561a43975ba9b5cb68f478736c
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09 16:06:17 +00:00
Paul Vinciguerra
0cbc71d783 tests: treat all truthy env vars the same way
Introduce a new class, that returns the truthiness of a env var.
Since an environment variable is just a string, it would normally
be true if not unset. The new class returns true when the env var is
set to a string that would be considered true.

Type: test
Depends-on: https://gerrit.fd.io/r/20484

Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09 07:16:16 +00:00
Dave Barach
32dcd3b2f2 vppinfra: allocate bihash virtual space on demand
Reduces the vpp image virtual size by multiple gigabytes

Add a "show bihash" command which displays configured and current
virtual space in use by bihash tables.

Modify the .py test framework to call "show bihash" on test tear-down

Type: refactor

Change-Id: Ifc1b7e2c43d29bbef645f6802fa29ff8ef09940c
Signed-off-by: Dave Barach <dave@barachs.net>
2019-07-09 02:23:00 +00:00
Paul Vinciguerra
063366eb23 tests: Have worker return immediately on bad executable
No reason to waste CI cycles if we know the test will not run.

See:
17:55:11  ==============================================================================
17:55:11  VPP Object Model Test
17:55:11  ==============================================================================
17:55:11  Exception in thread Thread-4:
17:55:11  Traceback (most recent call last):
17:55:11    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
17:55:11      self.run()
17:55:11    File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/framework.py", line 1475, in run
17:55:11      stdout=subprocess.PIPE, stderr=subprocess.PIPE)
17:55:11    File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 617, in __init__
17:55:11      restore_signals, start_new_session)
17:55:11    File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 1415, in _execute_child
17:55:11      raise child_exception_type(errno_num, err_msg)
17:55:11  OSError: [Errno 2] No such file or directory: '/w/workspace/vpp-beta-verify-master-ubuntu1804/test/build/vom_test/vom_test'
17:55:11
17:55:11  17:55:11,328 Timeout! Worker did not finish in 120s
17:55:11  run C++ VOM tests                                                        ERROR [ temp dir used by test case: /tmp/vpp-unittest-VOMTestCase-vpMcWF ]
17:55:11

Type: fix

Change-Id: I3d8252807e98a09a8abd70de8a22517151f9d786
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-03 02:07:59 +00:00
Paul Vinciguerra
fea8260771 tests: handle TypeErrors in calls to VppDiedError
Type: test

Change-Id: I869768ed25efa3350f025fba329466db69805803
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-28 11:59:04 +00:00
Paul Vinciguerra
5dd6a7b29f tests: if vpp is dead, specify the test name in the output
'Exception: VPP is dead when setting up the test'
is more helpful with the test info.

Type: test

Change-Id: I6a262fdcf2ecb33aae17bef3d19745053a0bade8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-25 11:26:08 +00:00
Paul Vinciguerra
496b0dee7f tests: refactor VppDiedError.
- Move Exception into same module as TestCase.
- Move the error reporting logic inside the error.
- Allows testing of the returncode and signal_name for tests to consume.
- Fix the signal reporting code:
	VppDiedError: VPP subprocess died unexpectedly with returncode -6 [unknown].
    displays as:
	VppDiedError: VPP subprocess died unexpectedly with return code: -6 [SIGABRT].

Type: test

Change-Id: I8488ab318a596c9b737308829cedfb7e96e57302
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-24 16:03:09 +00:00
Paul Vinciguerra
3a9f11e6d9 tests: add sudo to gdb commands
Prepend sudo to the gdb command line.

Type: test

Change-Id: I09013c3ca512e26de0c46c02f643b21064ba499c
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-20 03:37:54 +00:00
Paul Vinciguerra
499ea648e2 tests: framework gracefully handle 'VppTransportShmemIOError'
Catches:
----
Traceback (most recent call last):
  File "/vpp/test/framework.py", line 593, in tearDown
    self.logger.info(self.vapi.ppcli("api trace save %s" % api_trace))
  File "/vpp/test/vpp_papi_provider.py", line 413, in ppcli
    return cli + "\n" + str(self.cli(cli))
  File "/vpp/test/vpp_papi_provider.py", line 402, in cli
    r = self.papi.cli_inband(cmd=cli)
  File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 100, in __call__
    return self._func(**kwargs)
  File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 414, in f
    return self._call_vpp(i, msg, multipart, **kwargs)
  File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 634, in _call_vpp
    msg = self.transport.read()
  File "/vpp/src/vpp-api/python/vpp_papi/vpp_transport_shmem.py", line 120, in read
    raise VppTransportShmemIOError(rv, 'vac_read failed')
VppTransportShmemIOError: [Errno -1] vac_read failed
----

Change-Id: I767e48c4d03081eb5df6a8aa67da7e192d25e4cc
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-07 09:48:05 +00:00
Klement Sekera
6aa58b732b make test: fix packet counter assert
Change-Id: Ic6e6ac0629bbf07f7dad1803a85aabfdccbc2035
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-29 09:15:53 +00:00
Ole Troan
233e468183 stats: support multiple works for error counters
The current code only allowed access to the main thread error counters.
That is not so useful for a multi worker instance.
No return a vector indexed by thread of counter_t values.

Type: fix

Change-Id: Ie322c8889c0c8175e1116e71de04a2cf453b9ed7
Signed-off-by: Ole Troan <ot@cisco.com>
2019-05-22 09:43:49 +00:00
Klement Sekera
3a343d42d7 reassembly: prevent long chain attack
limit max # of fragments to 3 per packet by default
add API option to configure the limit at runtime

Change-Id: Ie4b9507bf5c6095b9a5925972b37fe0032f4f9e8
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-20 12:13:11 +00:00
Dave Barach
7d31ab2a5c add mactime plugin unit / code coverage tests
The unit and code coverage tests are boring. The rest of the patch
involves test and packet-generator infra cleanups.

Teach the "make test-xxx" family of targets to set the api test plugin
path correctly, to make "binary-api <api-message-name> <args>" debug
CLI commands work correctly in the "make test"
environment. Unfortunately involves both the top-level and test
Makefiles.

Add a minor pg cli feature, a CLI to manually set
s->sw_if_index[VLIB_TX].

Consider the case where one configures an interface with both a
device-input and an output feature. To test the output feature using
the pg, it's necessary to inject packets into the interface output
node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set
correctly. For example:

packet-generator new {
 name tx
 limit 15
 size 128-128
 interface local0  # rx: device input feature not configured on local0
 tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX]
 node loop0-output
 data {
   hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405
   }
}

Fix a longstanding bug in the packet generator stream setup.  Remove
kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places]
instead of using the stream value s->sw_if_index[VLIB_TX], and setting
THAT datum correctly.

Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed
Signed-off-by: Dave Barach <dave@barachs.net>
2019-05-09 20:08:58 +00:00
Alexandre Poirrier
cc991497b8 make test: add python3 compatibility in extend_packet
Change-Id: I342ff8e44e0d1ebd9f2c8cc386372b2f5a731080
Signed-off-by: Alexandre Poirrier <apoirrie@cisco.com>
2019-05-07 19:15:12 +00:00
Dave Wallace
3e9b7a219d make test: Fix truncated core file
- Wait for coredump to complete before killing VPP.

Change-Id: Ib7b96b339f796f9ef7e0eadff81d181464b8a03a
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-05-06 20:49:32 +00:00
Dave Barach
ad64687e8b gcov / test framework: sigterm not sigkill
Otherwise, gcov data vanishes without a trace.

Add a __gcov_flush() call to the unix signal handler, under #ifdef
CLIB_GCOV. Add -DCLIB_GCOV to vpp_gcov_TAG_CFLAGS.

Change-Id: I2726e671b26dfbe7fae88f46a8207bb2b5106884
Signed-off-by: Dave Barach <dave@barachs.net>
2019-05-06 20:48:51 +00:00
Ole Troan
4ff09ae348 API: Python and Unix domain socket improvement
Handle the case where buffer overflows.
Then SOCK_SEQPACKET assumption that multiple API messages
are not returned by recv() is broken. Use SOCK_STREAM for
API exchanges instead.

Add support for running tests over sockets.
make test SOCKET=1

Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821
Signed-off-by: Ole Troan <ot@cisco.com>
2019-04-23 18:49:22 +00:00
Paul Vinciguerra
90cf21b5d8 Tests: Refactor tearDown show command logging, add lifecycle markers.
This change adds a consistent interface for adding test-specific show commands to
log.txt.

It also adds log markers for the execution of setUp[Class], tearDown[Class]
in the logs.

Change-Id: I7d42e396e594a59e866a7d55dac0af25548e657a
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-04-11 07:23:11 +00:00
Ole Troan
2e1c8967fa API: Fix shared memory only action handlers.
Some API action handlers called vl_msg_ai_send_shmem()
directly. That breaks Unix domain socket API transport.

A couple (bond / vhost) also tried to send a sw_interface_event
directly, but did not send the message to all that had
registred interest. That scheme never worked correctly.
Refactored and improved the interface event code.

Change-Id: Idb90edfd8703c6ae593b36b4eeb4d3ed7da5c808
Signed-off-by: Ole Troan <ot@cisco.com>
2019-04-10 13:06:45 +00:00
Neale Ranns
d7603d97e0 IPSEC: tunnel fragmentation
Change-Id: I63741a22bc82f5f861e1c0f26a93b5569cc52061
Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-29 11:10:33 +00:00
Neale Ranns
192b13f96d BVI Interface
a new dedicated BVI interface as opposed to [re]using a loopback.

benefits:
 - removes ambiguity over the purpose of a loopback interface
 - TX node dedicated to BVI only functions.

Change-Id: I749d6b38440d450ac5b909a28053c75ec9df946a
Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-21 20:33:30 +00:00
Paul Vinciguerra
22364e49cb Tests: post-merge review comments for https://gerrit.fd.io/r/#/c/18081/
Change-Id: I2195b3264b5db34d869facf669f3e71a04fe3ff9
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-13 11:54:45 +00:00
Paul Vinciguerra
a7427ec6f8 VPP-1508: Use scapy.compat to manage packet level library differences.
Change-Id: Icdf6abc9e53d33b26fd1d531c7dda6be0bb9cb55
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11 08:11:16 +00:00
Paul Vinciguerra
0f6602cb24 Tests: fix time.sleep(0) # yield. Reduce sleep related log messages.
Reduce the incidence of:
20:04:23,606 unexpected time.sleep() result - slept for 2.187967e-03s instead of ~6.837845e-04s!

Change-Id: Ic576fda7f75e571c9019111588a6a936ee2cf5c2
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11 08:10:34 +00:00
Paul Vinciguerra
eaea421e1f Tests: Refactor payload_to_info()
All callers of payload_to_info were required to wrap payload with str().
Refactor to call scapy's payload.load for raw payloads or specify the
specific fieldname.

Change-Id: I1c80599d4df8dc129dbb8274733afaad406d5bcf
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07 22:29:44 +00:00
Paul Vinciguerra
ea2450fa2d VPP-1508: Python3 tests. Explicitly specify string formatting.
.format() doesn't understand what to do with binary values by default.
Specify that we expect coersion to string.

Change-Id: Ic4b697f70852124dc85ab231f76b7934d0d71f4d
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07 22:29:00 +00:00
Paul Vinciguerra
38a4ec7342 make test: Add exception handling around subprocess.
This cleans up exception catching to identify oserrors.  By raising the
specific exception closer to the offending call, we get additional stack history
and can add clearer error logging to assist in troubleshooting.

Change-Id: I592e4d46844b822a816485bf54910f8daed92088
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-26 09:39:00 +00:00
Paul Vinciguerra
eb414435a1 TESTS: refactor framework send_and_expect.
Create a common pg_send method for cases when results are just discarded.

Change-Id: I786960d2d7bbb96dcb407f6e59aa96951b7b19e7
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-20 19:05:46 +00:00
Paul Vinciguerra
8d991d923b VTL Cleanup: Fix missing calls to setUpClass/tearDownClass, fix numerous TypeErrors.
* TypeError: assertIsNotNone() got an unexpected keyword argument 'msg'

* Correct missing calls to setUpClass/tearDownClass.
    If you want the setUpClass and tearDownClass on base classes called then you must call up to them yourself.
    The implementations in TestCase are empty.
    https://docs.python.org/2/library/unittest.html#setupclass-and-teardownclass

Cleans up issues in parallel test mode:
-------------
FAILURES AND ERRORS IN TESTS:
  Testcase name: VCL Thru Host Stack Bidir Nsock
    FAILURE: test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock [test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock]
  Testcase name: Bidirectional Forwarding Detection (BFD) (changing auth)
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed]
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate]
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed]
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate]
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed]
      ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate]
  Testcase name: Punt Socket for IPv4
      ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_dump [test_punt.TestIP4PuntSocket.test_punt_socket_dump]
      ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets]
      ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket]
      ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket]
  Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6)
      ERROR: test_bfd.BFD6TestCase.test_echo [test_bfd.BFD6TestCase.test_echo]
      ERROR: test_bfd.BFD6TestCase.test_echo_looped_back [test_bfd.BFD6TestCase.test_echo_looped_back]
      ERROR: test_bfd.BFD6TestCase.test_intf_deleted [test_bfd.BFD6TestCase.test_intf_deleted]
      ERROR: test_bfd.BFD6TestCase.test_session_up [test_bfd.BFD6TestCase.test_session_up]
      ERROR: test_bfd.BFD6TestCase.test_session_up_by_ip [test_bfd.BFD6TestCase.test_session_up_by_ip]
  Testcase name: Bidirectional Forwarding Detection (BFD) (CLI)
      ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp]
      ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6 [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6]
      ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth]
      ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth]
      ERROR: test_bfd.BFDCLITestCase.test_admin_up_down [test_bfd.BFDCLITestCase.test_admin_up_down]
      ERROR: test_bfd.BFDCLITestCase.test_auth_on_off [test_bfd.BFDCLITestCase.test_auth_on_off]
      ERROR: test_bfd.BFDCLITestCase.test_auth_on_off_delayed [test_bfd.BFDCLITestCase.test_auth_on_off_delayed]
      ERROR: test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key [test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key]
      ERROR: test_bfd.BFDCLITestCase.test_set_del_sha1_key [test_bfd.BFDCLITestCase.test_set_del_sha1_key]
      ERROR: test_bfd.BFDCLITestCase.test_set_del_udp_echo_source [test_bfd.BFDCLITestCase.test_set_del_udp_echo_source]
      ERROR: test_bfd.BFDCLITestCase.test_show [test_bfd.BFDCLITestCase.test_show]
  Testcase name: VAPI test
      ERROR: test_vapi.VAPITestCase.test_vapi_c [test_vapi.VAPITestCase.test_vapi_c]
  Testcase name: Container integration extended testcases
      ERROR: test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test [test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands [test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands]
      ERROR: test_container.ContainerIntegrationTestCase.test_0410_spoof_test [test_container.ContainerIntegrationTestCase.test_0410_spoof_test]
      ERROR: test_container.ContainerIntegrationTestCase.test_0411_spoof_test [test_container.ContainerIntegrationTestCase.test_0411_spoof_test]
  Testcase name: Re-enable IPFIX
      ERROR: test_flowprobe.ReenableIPFIX.test_0011 [test_flowprobe.ReenableIPFIX.test_0011]
  Testcase name: VXLAN over IPv6 Test Case
      ERROR: setUpClass [setUpClass (test_vxlan6.TestVxlan6)]
  Testcase name: JVPP Core Test Case
      ERROR: test_jvpp.TestJVpp.test_vpp_acl_callback_api [test_jvpp.TestJVpp.test_vpp_acl_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_acl_future_api [test_jvpp.TestJVpp.test_vpp_acl_future_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_core_callback_api [test_jvpp.TestJVpp.test_vpp_core_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_core_future_api [test_jvpp.TestJVpp.test_vpp_core_future_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api [test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_future_api [test_jvpp.TestJVpp.test_vpp_ioamexport_future_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_callback_api [test_jvpp.TestJVpp.test_vpp_ioampot_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_future_api [test_jvpp.TestJVpp.test_vpp_ioampot_future_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_snat_callback_api [test_jvpp.TestJVpp.test_vpp_snat_callback_api]
      ERROR: test_jvpp.TestJVpp.test_vpp_snat_future_api [test_jvpp.TestJVpp.test_vpp_snat_future_api]
  Testcase name: LDP Cut Thru Tests
    FAILURE: test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3 [test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3]
  Testcase name: ACL plugin connection-oriented extended testcases
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test]
      ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test]
  Testcase name: LDP Thru Host Stack Echo
    FAILURE: test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo [test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo]
  Testcase name: Bidirectional Forwarding Detection (BFD) - API
      ERROR: test_bfd.BFDAPITestCase.test_activate_auth [test_bfd.BFDAPITestCase.test_activate_auth]
      ERROR: test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key [test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key]
      ERROR: test_bfd.BFDAPITestCase.test_add_bfd [test_bfd.BFDAPITestCase.test_add_bfd]
      ERROR: test_bfd.BFDAPITestCase.test_add_bfd6 [test_bfd.BFDAPITestCase.test_add_bfd6]
      ERROR: test_bfd.BFDAPITestCase.test_add_bfd_sha1 [test_bfd.BFDAPITestCase.test_add_bfd_sha1]
      ERROR: test_bfd.BFDAPITestCase.test_add_sha1_keys [test_bfd.BFDAPITestCase.test_add_sha1_keys]
      ERROR: test_bfd.BFDAPITestCase.test_change_key [test_bfd.BFDAPITestCase.test_change_key]
      ERROR: test_bfd.BFDAPITestCase.test_deactivate_auth [test_bfd.BFDAPITestCase.test_deactivate_auth]
      ERROR: test_bfd.BFDAPITestCase.test_double_add [test_bfd.BFDAPITestCase.test_double_add]
      ERROR: test_bfd.BFDAPITestCase.test_double_add_sha1 [test_bfd.BFDAPITestCase.test_double_add_sha1]
      ERROR: test_bfd.BFDAPITestCase.test_mod_bfd [test_bfd.BFDAPITestCase.test_mod_bfd]
      ERROR: test_bfd.BFDAPITestCase.test_set_del_udp_echo_source [test_bfd.BFDAPITestCase.test_set_del_udp_echo_source]
      ERROR: test_bfd.BFDAPITestCase.test_shared_sha1_key [test_bfd.BFDAPITestCase.test_shared_sha1_key]
  Testcase name: LDP Thru Host Stack Iperf
    FAILURE: test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3 [test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3]
  Testcase name: Bidirectional Forwarding Detection (BFD) (SHA1 auth)
      ERROR: test_bfd.BFDSHA1TestCase.test_session_up [test_bfd.BFDSHA1TestCase.test_session_up]
  Testcase name: Punt Socket for IPv6
      ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_dump [test_punt.TestIP6PuntSocket.test_punt_socket_dump]
      ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets]
      ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket]
      ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket]
  Testcase name: Disable Flowprobe feature
      ERROR: test_flowprobe.DisableFP.test_0001 [test_flowprobe.DisableFP.test_0001]
  Testcase name: Disable IPFIX
      ERROR: test_flowprobe.DisableIPFIX.test_0001 [test_flowprobe.DisableIPFIX.test_0001]
  Testcase name: VPP Object Model Test
      ERROR: test_vom.VOMTestCase.test_vom_cpp [test_vom.VOMTestCase.test_vom_cpp]
  Testcase name: BFD-FIB interactions (IPv6)
      ERROR: test_bfd.BFDFIBTestCase.test_session_with_fib [test_bfd.BFDFIBTestCase.test_session_with_fib]
  Testcase name: Bidirectional Forwarding Detection (BFD)
      ERROR: test_bfd.BFD4TestCase.test_echo [test_bfd.BFD4TestCase.test_echo]
      ERROR: test_bfd.BFD4TestCase.test_echo_looped_back [test_bfd.BFD4TestCase.test_echo_looped_back]
      ERROR: test_bfd.BFD4TestCase.test_intf_deleted [test_bfd.BFD4TestCase.test_intf_deleted]
      ERROR: test_bfd.BFD4TestCase.test_session_down [test_bfd.BFD4TestCase.test_session_down]
      ERROR: test_bfd.BFD4TestCase.test_session_up [test_bfd.BFD4TestCase.test_session_up]
      ERROR: test_bfd.BFD4TestCase.test_session_up_by_ip [test_bfd.BFD4TestCase.test_session_up_by_ip]
  Testcase name: VXLAN-GPE Test Case
      ERROR: test_vxlan_gpe.TestVxlanGpe.test_decap [test_vxlan_gpe.TestVxlanGpe.test_decap]
      ERROR: test_vxlan_gpe.TestVxlanGpe.test_encap [test_vxlan_gpe.TestVxlanGpe.test_encap]
      ERROR: test_vxlan_gpe.TestVxlanGpe.test_ucast_flood [test_vxlan_gpe.TestVxlanGpe.test_ucast_flood]
  Testcase name: Re-enable Flowprobe feature
      ERROR: test_flowprobe.ReenableFP.test_0001 [test_flowprobe.ReenableFP.test_0001]
  Testcase name: LDP IPv6 Cut Thru Tests
    FAILURE: test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo [test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo]
TESTCASES WHERE NO TESTS WERE SUCCESSFULLY EXECUTED:
  VXLAN over IPv6 Test Case
==============================================================================

0 attempt(s) left.

Change-Id: Id202ed6ee7f57670f34ec87380244c568b509416
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-04 15:39:24 +00:00
Klement Sekera
0ede47aa45 make test: fix logging
Change-Id: I3867d50746f0f424c8e94929171363e5b2e4d470
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-31 22:10:01 +00:00
Paul Vinciguerra
1ec06fffca VTL: VppTestResult: Must implement empty constructor.
Subclasses of (unittest.TestResult|unittest.TextTestResult) must implement an empty constructor.

The standard python library unittests depend on the empty constructor.

Change-Id: I6d1bd2e7d6996d85a9da66cb7d40af0ed3c30e2b
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-22 10:12:39 +00:00
Paul Vinciguerra
9889402b79 VTL: Add **kwargs to VppTestRunner constructor.
Subclasses should accept **kwargs to ensure compatibility as the
        interface changes.

https://github.com/python/cpython/blob/master/Lib/unittest/runner.py#L133

Change-Id: Ia64ffd88899d207a5422f066700d2f7f9c77622e
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-21 15:32:51 +00:00
Klement Sekera
d91fa61e8d make test: raise packet tracing limit to 1000
Change-Id: I4309cefe13ee861342e7962c7652919a47748656
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-16 13:34:34 +00:00
Paul Vinciguerra
dd3c5d250f VTL: Allow running simple unittest.TestCases.
It came to my attention that Ole added a simple test in:
https://gerrit.fd.io/r/#/c/16381/ and the framework forced him
to launch an instance of VPP to test the formatting of a mac address.

This change allows the test framework to run standard unittest.TestCases
without the need to spawn a VPP instance.

Change-Id: I56651ab27c4c6bf920081a526f168a743d643201
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-14 13:09:08 +00:00
Paul Vinciguerra
8aeb2207f9 test framework. Remove unused code.
Change-Id: I1bd14d5aff4918b58aef18fa0a9264e033989f6e
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-09 06:10:59 +00:00
Ole Troan
a45dc07c15 Revert "tests: Rework vpp config generation."
This reverts commit 919efad2671993d4c6d5a0dba8eeb99d5c60edf1.

Change-Id: I5ec4fa08a03f508dac684dc1ddbd7e6b85c49b97
Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-21 16:14:15 +00:00
Paul Vinciguerra
087c811b63 Python Tests: Make VppTestCase unit testable.
* This allows for VppTestCase to instantiated with an empty constructor.

Change-Id: Ib71d3bfb4529e113f9b73c39de9b95f12ee9cfb4
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21 07:50:30 +00:00
Paul Vinciguerra
919efad267 tests: Rework vpp config generation.
* Allows test cases to configure the VPP runtime config
  during fixture setup.

* Sample use in a TestCase:
    @classmethod
    def setUpConstants(cls):
        tempdir = cls.tempdir
        cls.config.add('punt', 'socket', '%s/socket_punt' % cls.tempdir)
        super(TestPuntSocket, cls).setUpConstants()
        # enable/disabe a plugin via:
        #cls.config.add_plugin('dpdk_plugin.so', 'disable')

* Supports the following config stanzas:
	'unix',
        'acl-plugin'
        'api-queue'
        'api-trace'
        'api-segment'
        'cj'
        'cpu'
        'dns'
        'dpdk
        # currently don't support dynamic keys
        # 'heapsize'
        'l2learn'
        'l2tp'
        'mactime'
        'mc'
        'nat'
        'oam'
        'plugins'
        'punt'
        'session'
        'socksvr'
        'statseg'
        'tapcli'
        'tcp'
        'tuntap'
        'vhost-user'

Change-Id: I44f276487267d26eaa46f87e730bdd861003b234
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21 07:49:37 +00:00
Paul Vinciguerra
852f5ef9cf VPP-1508 python3 tests: raw_input
'raw_input' does not exist in python3.

Change-Id: I95d916db2834fab83e5e791f35103a4513e610b8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20 12:44:26 +00:00
Paul Vinciguerra
defde0f870 Tests: Cleanup @skip decorator.
The runnning environment is static as of module load time,
so only evalute the conditions once at module load time.

Track-by: VPP-1518

Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20 12:43:38 +00:00
Klement Sekera
b6d92d811c make test: add option for adding extra vpp config
Change-Id: Ief316756635794dadf2f0f4190aa20745df5e53c
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-17 14:31:00 +01:00
Klement Sekera
14d7e90788 VPP-1522: harden reassembly code
Change-Id: Ib5a20bff7d8a340ecf50bcd4a023d6bf36382ba3
Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-13 14:42:50 +00:00
Ole Troan
413f4a5b21 API: Use string type instead of u8.
The new string type is modelled after string in proto3.
It is always variable length.

Change-Id: I64884067e28a80072c8dac31b7c7c82d6e306051
Signed-off-by: Ole Troan <ot@cisco.com>
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-13 12:11:50 +00:00
Paul Vinciguerra
6919b0de47 test/framework.py: change logger to return 'e' format.
* When the framework logs an unexpected sleep time,
  display the units in the same format. Typically,
  it has been returning a float and an 'e'.
  ex. unexpected time.sleep() result - slept for 0.000107049942017s instead of ~4.88758087158e-05s!

* Defer logger interpolation.

Change-Id: I543ad6d41c7f263d61615341437f80973c0017de
Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-10 08:16:23 +00:00
Ole Troan
7f99183a20 Test framework: StringIO fixes for Python3
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py

Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877
Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-10 08:01:56 +00:00