ligang/vpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hs-test: transition to ginkgo test framework

Browse Source 
Type: test

Change-Id: Ia38bf5549d20b22876f6082085b69a52a03d0142
Signed-off-by: Adrian Villin <avillin@cisco.com>
Signed-off-by: Matus Fabian <matfabia@cisco.com>
This commit is contained in:
Adrian Villin
2024-03-14 11:42:55 -04:00
parent 3a05db6264
commit cee15aa940
25 changed files with 762 additions and 230 deletions
Download Patch File Download Diff File Expand all files Collapse all files

173
extras/hs-test/README.rst
View File

@ -9,7 +9,10 @@ End-to-end tests often want multiple VPP instances, network namespaces, differen
and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient.
For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology.
Go's package `testing`_ together with `go test`_ command form the base framework upon which the *hs-test* is built and run.
`Ginkgo`_ forms the base framework upon which the *hs-test* is built and run.
All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require.
This allows us to run those mentioned groups in parallel, but not individual tests in parallel.
Anatomy of a test case
----------------------
@ -24,15 +27,16 @@ Anatomy of a test case
**Action flow when running a test case**:
#. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run),
and TEST=<test-name> to run specific test.
#. ``make list-tests`` (or ``make help``) shows all test names.
#. ``go test`` compiles package ``main`` along with any files with names matching the file pattern ``*_test.go``
and then runs the resulting test binaries
#. The go test framework runs each function matching :ref:`naming convention<test-convention>`. Each of these corresponds to a `test suite`_
#. Testify toolkit's ``suite.Run(t *testing.T, suite TestingSuite)`` function runs the suite and does the following:
TEST=<test-name> to run a specific test and PARALLEL=[n-cpus].
#. ``make list-tests`` (or ``make help``) shows all tests. The current `list of tests`_ is at the bottom of this document.
#. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary
#. The Ginkgo test framework runs each function that was registered manually using ``registerMySuiteTest(s *MySuite)``. Each of these functions correspond to a suite
#. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following:
#. Ginkgo compiles the spec, builds a spec tree
#. ``Describe`` container nodes in suite\_\*_test.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus])
#. Suite is initialized. The topology is loaded and configured in this step
#. Test suite runs all the tests attached to it
#. Registered tests are run in generated ``It`` subject nodes
#. Execute tear-down functions, which currently consists of stopping running containers
and clean-up of test topology
@ -43,23 +47,25 @@ This describes adding a new test case to an existing suite.
For adding a new suite, please see `Modifying the framework`_ below.
#. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists
#. Declare method whose name starts with ``Test`` and specifies its receiver as a pointer to the suite's struct (defined in ``framework_test.go``)
#. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``suite_*_test.go``)
#. Implement test behaviour inside the test method. This typically includes the following:
#. Retrieve a running container in which to run some action. Method ``getContainerByName``
from ``HstSuite`` struct serves this purpose
#. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
#. Run arbitrary commands inside the containers with ``exec`` method
#. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
For example, use ``wget`` with ``startWget`` function
#. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
#. Verify results of your tests using ``assert`` methods provided by the test suite,
implemented by HstSuite struct
#. Retrieve a running container in which to run some action. Method ``getContainerByName``
from ``HstSuite`` struct serves this purpose
#. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
#. Run arbitrary commands inside the containers with ``exec`` method
#. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
For example, use ``wget`` with ``startWget`` function
#. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
#. Verify results of your tests using ``assert`` methods provided by the test suite, implemented by HstSuite struct or use ``Gomega`` assert functions.
#. Create an ``init()`` function and register the test using ``register*SuiteTests(testCaseFunction)``
**Example test case**
Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other.
This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./test -run TestMySuite/TestMyCase``.
This can be put in file ``extras/hs-test/my_test.go`` and run with command ``make test TEST=MyTest`` or ``ginkgo -v --trace --focus MyTest``.
::
@ -69,7 +75,11 @@ This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./t
"fmt"
)
func (s *MySuite) TestMyCase() {
func init(){
registerMySuiteTest(MyTest)
}
func MyTest(s *MySuite) {
clientVpp := s.getContainerByName("client-vpp").vppInstance
serverVethAddress := s.netInterfaces["server-iface"].AddressString()
@ -86,8 +96,7 @@ Modifying the framework
.. _test-convention:
#. Adding a new suite takes place in ``framework_test.go`` and by creating a new file for the suite.
Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced
#. To add a new suite, create a new file. Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced
by the actual name
#. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member.
@ -99,7 +108,17 @@ Modifying the framework
HstSuite
}
#. In suite file, implement ``SetupSuite`` method which testify runs once before starting any of the tests.
#. Create a new slice that will contain test functions with a pointer to the suite's struct: ``var myTests = []func(s *MySuite){}``
#. Then create a new function that will append test functions to that slice:
::
func registerMySuiteTests(tests ...func(s *MySuite)) {
nginxTests = append(myTests, tests...)
}
#. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests.
It's important here to call ``configureNetworkTopology`` method,
pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder.
Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml``
@ -111,6 +130,8 @@ Modifying the framework
::
func (s *MySuite) SetupSuite() {
s.HstSuite.SetupSuite()
// Add custom setup code here
s.configureNetworkTopology("myTopology")
@ -123,19 +144,62 @@ Modifying the framework
::
func (s *MySuite) SetupTest() {
s.HstSuite.setupTest()
s.SetupVolumes()
s.SetupContainers()
}
#. In order for ``go test`` to run this suite, we need to create a normal test function and pass our suite to ``suite.Run``.
These functions are placed at the end of ``framework_test.go``
#. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node.
Place them at the end of the suite file
* Declare a suite struct variable before anything else
* To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered``
* Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped.
To override this behavior, decorate the container node with ``ContinueOnFailure``
::
func TestMySuite(t *testing.T) {
var m MySuite
suite.Run(t, &m)
}
var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() {
var s MySuite
BeforeAll(func() {
s.SetupSuite()
})
BeforeEach(func() {
s.SetupTest()
})
AfterAll(func() {
s.TearDownSuite()
})
AfterEach(func() {
s.TearDownTest()
})
for _, test := range mySuiteTests {
test := test
pc := reflect.ValueOf(test).Pointer()
funcValue := runtime.FuncForPC(pc)
It(strings.Split(funcValue.Name(), ".")[2], func(ctx SpecContext) {
test(&s)
}, SpecTimeout(time.Minute*5))
}
})
#. Notice the loop - it will generate multiple ``It`` nodes, each running a different test.
``test := test`` is necessary, otherwise only the last test in a suite will run.
For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\.
#. ``funcValue.Name()`` returns the full name of a function (e.g. ``fd.io/hs-test.MyTest``), however, we only need the test name (``MyTest``).
#. To run certain tests solo, create a new slice that will only contain tests that have to run solo and a new register function.
Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node:
::
var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() {
...
It(strings.Split(funcValue.Name(), ".")[2], Label("SOLO"), func(ctx SpecContext) {
test(&s)
}, SpecTimeout(time.Minute*5))
})
#. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above
@ -186,14 +250,9 @@ Alternatively copy the executable from host system to the Docker image, similarl
**Skipping tests**
``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as:
``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``.
``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``.
However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving
test run time it is not advisable to use aforementioned skip methods and instead prefix test name with ``Skip``:
::
func (s *MySuite) SkipTest(){
test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test.
**Debugging a test**
@ -201,11 +260,11 @@ It is possible to debug VPP by attaching ``gdb`` before test execution by adding
::
$ make test TEST=TestVeths/TestLDPreloadIperfVpp DEBUG=true
$ make test TEST=LDPreloadIperfVppTest DEBUG=true
...
run following command in different terminal:
docker exec -it server-vpp gdb -ex "attach $(docker exec server-vpp pidof vpp)"
Afterwards press CTRL+C to continue
docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)"
Afterwards press CTRL+\ to continue
If a test consists of more VPP instances then this is done for each of them.
@ -223,8 +282,38 @@ Generally, these will be updated on a per-need basis, for example when a bug is
or a new version incompatibility issue occurs.
.. _testing: https://pkg.go.dev/testing
.. _go test: https://pkg.go.dev/cmd/go#hdr-Test_packages
.. _test suite: https://github.com/stretchr/testify#suite-package
.. _ginkgo: https://onsi.github.io/ginkgo/
.. _volumes: https://docs.docker.com/storage/volumes/
**List of tests**
.. _list of tests:
Please update this list whenever you add a new test by pasting the output below.
* NsSuite/HttpTpsTest
* NsSuite/VppProxyHttpTcpTest
* NsSuite/VppProxyHttpTlsTest
* NsSuite/EnvoyProxyHttpTcpTest
* NginxSuite/MirroringTest
* VethsSuiteSolo TcpWithLossTest [SOLO]
* NoTopoSuiteSolo HttpStaticPromTest [SOLO]
* TapSuite/LinuxIperfTest
* NoTopoSuite/NginxHttp3Test
* NoTopoSuite/NginxAsServerTest
* NoTopoSuite/NginxPerfCpsTest
* NoTopoSuite/NginxPerfRpsTest
* NoTopoSuite/NginxPerfWrkTest
* VethsSuite/EchoBuiltinTest
* VethsSuite/HttpCliTest
* VethsSuite/LDPreloadIperfVppTest
* VethsSuite/VppEchoQuicTest
* VethsSuite/VppEchoTcpTest
* VethsSuite/VppEchoUdpTest
* VethsSuite/XEchoVclClientUdpTest
* VethsSuite/XEchoVclClientTcpTest
* VethsSuite/XEchoVclServerUdpTest
* VethsSuite/XEchoVclServerTcpTest
* VethsSuite/VclEchoTcpTest
* VethsSuite/VclEchoUdpTest
* VethsSuite/VclRetryAttachTest