| Host stack test framework |
| ========================= |
| |
| Overview |
| -------- |
| |
| The goal of the Host stack test framework (**hs-test**) is to ease writing and running end-to-end tests for VPP's Host Stack. |
| End-to-end tests often want multiple VPP instances, network namespaces, different types of interfaces |
| and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient. |
| For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology. |
| |
| `Ginkgo`_ forms the base framework upon which the *hs-test* is built and run. |
| All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require. |
| This allows us to run those mentioned groups in parallel, but not individual tests in parallel. |
| |
| |
| Anatomy of a test case |
| ---------------------- |
| |
| **Prerequisites**: |
| |
| * Install hs-test dependencies with ``make install-deps`` |
| * Tests use *hs-test*'s own docker image, so building it before starting tests is a prerequisite. Run ``make build[-debug]`` to do so |
| * Docker has to be installed and Go has to be in path of both the running user and root |
| * Root privileges are required to run tests as it uses Linux ``ip`` command for configuring topology |
| |
| **Action flow when running a test case**: |
| |
| #. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run), |
| TEST=<test-name> to run a specific test and PARALLEL=[n-cpus]. |
| #. ``make list-tests`` (or ``make help``) shows all tests. The current `list of tests`_ is at the bottom of this document. |
| #. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary |
| #. The Ginkgo test framework runs each function that was registered manually using ``registerMySuiteTest(s *MySuite)``. Each of these functions correspond to a suite |
| #. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following: |
| |
| #. Ginkgo compiles the spec, builds a spec tree |
| #. ``Describe`` container nodes in suite\_\*_test.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus]) |
| #. Suite is initialized. The topology is loaded and configured in this step |
| #. Registered tests are run in generated ``It`` subject nodes |
| #. Execute tear-down functions, which currently consists of stopping running containers |
| and clean-up of test topology |
| |
| Adding a test case |
| ------------------ |
| |
| This describes adding a new test case to an existing suite. |
| For adding a new suite, please see `Modifying the framework`_ below. |
| |
| #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists |
| #. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``suite_*_test.go``) |
| #. Implement test behaviour inside the test method. This typically includes the following: |
| |
| #. Retrieve a running container in which to run some action. Method ``getContainerByName`` |
| from ``HstSuite`` struct serves this purpose |
| #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI |
| #. Run arbitrary commands inside the containers with ``exec`` method |
| #. Run other external tool with one of the preexisting functions in the ``utils.go`` file. |
| For example, use ``wget`` with ``startWget`` function |
| #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else |
| #. Verify results of your tests using ``assert`` methods provided by the test suite, implemented by HstSuite struct or use ``Gomega`` assert functions. |
| |
| #. Create an ``init()`` function and register the test using ``register*SuiteTests(testCaseFunction)`` |
| |
| |
| **Example test case** |
| |
| Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other. |
| This can be put in file ``extras/hs-test/my_test.go`` and run with command ``make test TEST=MyTest`` or ``ginkgo -v --trace --focus MyTest``. |
| |
| :: |
| |
| package main |
| |
| import ( |
| "fmt" |
| ) |
| |
| func init(){ |
| registerMySuiteTest(MyTest) |
| } |
| |
| func MyTest(s *MySuite) { |
| clientVpp := s.getContainerByName("client-vpp").vppInstance |
| |
| serverVethAddress := s.netInterfaces["server-iface"].AddressString() |
| |
| result := clientVpp.vppctl("ping " + serverVethAddress) |
| s.assertNotNil(result) |
| s.log(result) |
| } |
| |
| |
| Filtering test cases |
| -------------------- |
| |
| The framework allows us to filter test cases in a few different ways, using ``make test TEST=``: |
| * Suite name |
| * File name |
| * Test name |
| * All of the above as long as they are ordered properly, e.g. ``make test TEST=VethsSuite.http_test.go.HeaderServerTest`` |
| |
| **Names are case sensitive!** |
| |
| Names don't have to be complete, as long as they are last: |
| This is valid and will run all tests in every ``http`` file (if there is more than one): |
| ``make test TEST=VethsSuite.http`` |
| This is not valid: |
| ``make test TEST=Veths.http`` |
| |
| They can also be left out: |
| ``make test TEST=http_test.go`` will run every test in ``http_test.go`` |
| ``make test TEST=Nginx`` will run everything that has 'Nginx' in its name - suites, files and tests. |
| ``make test TEST=HeaderServerTest`` will only run the header server test |
| |
| |
| Modifying the framework |
| ----------------------- |
| |
| **Adding a test suite** |
| |
| .. _test-convention: |
| |
| #. To add a new suite, create a new file. Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced |
| by the actual name |
| |
| #. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member. |
| HstSuite provides functionality that can be shared for all suites, like starting containers |
| |
| :: |
| |
| type MySuite struct { |
| HstSuite |
| } |
| |
| #. Create a new slice that will contain test functions with a pointer to the suite's struct: ``var myTests = []func(s *MySuite){}`` |
| |
| #. Then create a new function that will append test functions to that slice: |
| |
| :: |
| |
| func registerMySuiteTests(tests ...func(s *MySuite)) { |
| nginxTests = append(myTests, tests...) |
| } |
| |
| #. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests. |
| It's important here to call ``configureNetworkTopology`` method, |
| pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder. |
| Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml`` |
| This will ensure network topology, such as network interfaces and namespaces, will be created. |
| Another important method to call is ``loadContainerTopology()`` which will load |
| containers and shared volumes used by the suite. This time the name passed to method corresponds |
| to file in ``extras/hs-test/topo-containers`` folder |
| |
| :: |
| |
| func (s *MySuite) SetupSuite() { |
| s.HstSuite.SetupSuite() |
| |
| // Add custom setup code here |
| |
| s.configureNetworkTopology("myTopology") |
| s.loadContainerTopology("2peerVeth") |
| } |
| |
| #. In suite file, implement ``SetupTest`` method which gets executed before each test. Starting containers and |
| configuring VPP is usually placed here |
| |
| :: |
| |
| func (s *MySuite) SetupTest() { |
| s.HstSuite.setupTest() |
| s.SetupVolumes() |
| s.SetupContainers() |
| } |
| |
| #. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node. |
| Place them at the end of the suite file |
| |
| * Declare a suite struct variable before anything else |
| * To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered`` |
| * Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped. |
| To override this behavior, decorate the container node with ``ContinueOnFailure`` |
| |
| :: |
| |
| var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() { |
| var s MySuite |
| BeforeAll(func() { |
| s.SetupSuite() |
| }) |
| BeforeEach(func() { |
| s.SetupTest() |
| }) |
| AfterAll(func() { |
| s.TearDownSuite() |
| }) |
| AfterEach(func() { |
| s.TearDownTest() |
| }) |
| for _, test := range mySuiteTests { |
| test := test |
| pc := reflect.ValueOf(test).Pointer() |
| funcValue := runtime.FuncForPC(pc) |
| It(strings.Split(funcValue.Name(), ".")[2], func(ctx SpecContext) { |
| test(&s) |
| }, SpecTimeout(time.Minute*5)) |
| } |
| }) |
| |
| #. Notice the loop - it will generate multiple ``It`` nodes, each running a different test. |
| ``test := test`` is necessary, otherwise only the last test in a suite will run. |
| For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\. |
| |
| #. ``funcValue.Name()`` returns the full name of a function (e.g. ``fd.io/hs-test.MyTest``), however, we only need the test name (``MyTest``). |
| |
| #. To run certain tests solo, create a new slice that will only contain tests that have to run solo and a new register function. |
| Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node: |
| |
| :: |
| |
| var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() { |
| ... |
| It(strings.Split(funcValue.Name(), ".")[2], Label("SOLO"), func(ctx SpecContext) { |
| test(&s) |
| }, SpecTimeout(time.Minute*5)) |
| }) |
| |
| #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above |
| |
| **Adding a topology element** |
| |
| Topology configuration exists as ``yaml`` files in the ``extras/hs-test/topo-network`` and |
| ``extras/hs-test/topo-containers`` folders. Processing of a network topology file for a particular test suite |
| is started by the ``configureNetworkTopology`` method depending on which file's name is passed to it. |
| Specified file is loaded and converted into internal data structures which represent various elements of the topology. |
| After parsing the configuration, framework loops over the elements and configures them one by one on the host system. |
| |
| These are currently supported types of network elements. |
| |
| * ``netns`` - network namespace |
| * ``veth`` - veth network interface, optionally with target network namespace or IPv4 address |
| * ``bridge`` - ethernet bridge to connect created interfaces, optionally with target network namespace |
| * ``tap`` - tap network interface with IP address |
| |
| Similarly, container topology is started by ``loadContainerTopology()``, configuration file is processed |
| so that test suite retains map of defined containers and uses that to start them at the beginning |
| of each test case and stop containers after the test finishes. Container configuration can specify |
| also volumes which allow to share data between containers or between host system and containers. |
| |
| Supporting a new type of topology element requires adding code to recognize the new element type during loading. |
| And adding code to set up the element in the host system with some Linux tool, such as *ip*. |
| This should be implemented in ``netconfig.go`` for network and in ``container.go`` for containers and volumes. |
| |
| **Communicating between containers** |
| |
| When two VPP instances or other applications, each in its own Docker container, |
| want to communicate there are typically two ways this can be done within *hs-test*. |
| |
| * Network interfaces. Containers are being created with ``-d --network host`` options, |
| so they are connected with interfaces created in host system |
| * Shared folders. Containers are being created with ``-v`` option to create shared `volumes`_ between host system and containers |
| or just between containers |
| |
| Host system connects to VPP instances running in containers using a shared folder |
| where binary API socket is accessible by both sides. |
| |
| **Adding an external tool** |
| |
| If an external program should be executed as part of a test case, it might be useful to wrap its execution in its own function. |
| These types of functions are placed in the ``utils.go`` file. If the external program is not available by default in Docker image, |
| add its installation to ``extras/hs-test/Dockerfile.vpp`` in ``apt-get install`` command. |
| Alternatively copy the executable from host system to the Docker image, similarly how the VPP executables and libraries are being copied. |
| |
| **Skipping tests** |
| |
| ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as: |
| ``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``. |
| However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving |
| test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test. |
| |
| **Debugging a test** |
| |
| It is possible to debug VPP by attaching ``gdb`` before test execution by adding ``DEBUG=true`` like follows: |
| |
| :: |
| |
| $ make test TEST=LDPreloadIperfVppTest DEBUG=true |
| ... |
| run following command in different terminal: |
| docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)" |
| Afterwards press CTRL+\ to continue |
| |
| If a test consists of more VPP instances then this is done for each of them. |
| |
| |
| **Eternal dependencies** |
| |
| * Linux tools ``ip``, ``brctl`` |
| * Standalone programs ``wget``, ``iperf3`` - since these are downloaded when Docker image is made, |
| they are reasonably up-to-date automatically |
| * Programs in Docker images - ``envoyproxy/envoy-contrib`` and ``nginx`` |
| * ``http_server`` - homegrown application that listens on specified port and sends a test file in response |
| * Non-standard Go libraries - see ``extras/hs-test/go.mod`` |
| |
| Generally, these will be updated on a per-need basis, for example when a bug is discovered |
| or a new version incompatibility issue occurs. |
| |
| |
| .. _ginkgo: https://onsi.github.io/ginkgo/ |
| .. _volumes: https://docs.docker.com/storage/volumes/ |
| |
| **List of tests** |
| |
| .. _list of tests: |
| |
| Please update this list whenever you add a new test by pasting the output below. |
| |
| * NsSuite/HttpTpsTest |
| * NsSuite/VppProxyHttpTcpTest |
| * NsSuite/VppProxyHttpTlsTest |
| * NsSuite/EnvoyProxyHttpTcpTest |
| * NginxSuite/MirroringTest |
| * VethsSuiteSolo TcpWithLossTest [SOLO] |
| * NoTopoSuiteSolo HttpStaticPromTest [SOLO] |
| * TapSuite/LinuxIperfTest |
| * NoTopoSuite/NginxHttp3Test |
| * NoTopoSuite/NginxAsServerTest |
| * NoTopoSuite/NginxPerfCpsTest |
| * NoTopoSuite/NginxPerfRpsTest |
| * NoTopoSuite/NginxPerfWrkTest |
| * VethsSuite/EchoBuiltinTest |
| * VethsSuite/HttpCliTest |
| * VethsSuite/LDPreloadIperfVppTest |
| * VethsSuite/VppEchoQuicTest |
| * VethsSuite/VppEchoTcpTest |
| * VethsSuite/VppEchoUdpTest |
| * VethsSuite/XEchoVclClientUdpTest |
| * VethsSuite/XEchoVclClientTcpTest |
| * VethsSuite/XEchoVclServerUdpTest |
| * VethsSuite/XEchoVclServerTcpTest |
| * VethsSuite/VclEchoTcpTest |
| * VethsSuite/VclEchoUdpTest |
| * VethsSuite/VclRetryAttachTest |