test/doc/index.rst - fdio/vpp - Gitiles

 .. _unittest: https://docs.python.org/2/library/unittest.html
 .. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase
 .. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError
 .. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest
 .. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/
 .. _scapy: http://www.secdev.org/projects/scapy/
 .. _logging: https://docs.python.org/2/library/logging.html

 .. |vtf| replace:: VPP Test Framework

 |vtf|
 =====

 Overview
 ########

 The goal of the |vtf| is to ease writing, running and debugging
 unit tests for the VPP. For this, python was chosen as a high level language
 allowing rapid development with scapy_ providing the necessary tool for creating
 and dissecting packets.

 Anatomy of a test case
 ######################

 Python's unittest_ is used as the base framework upon which the VPP test
 framework is built. A test suite in the |vtf| consists of multiple classes
 derived from `VppTestCase`, which is itself derived from TestCase_.
 The test class defines one or more test functions, which act as test cases.

 Function flow when running a test case is:

 1. `setUpClass <VppTestCase.setUpClass>`:
    This function is called once for each test class, allowing a one-time test
    setup to be executed. If this functions throws an exception,
    none of the test functions are executed.
 2. `setUp <VppTestCase.setUp>`:
    The setUp function runs before each of the test functions. If this function
    throws an exception other than AssertionError_ or SkipTest_, then this is
    considered an error, not a test failure.
 3. *test_<name>*:
    This is the guts of the test case. It should execute the test scenario
    and use the various assert functions from the unittest framework to check
    necessary. Multiple test_<name> methods can exist in a test case.
 4. `tearDown <VppTestCase.tearDown>`:
    The tearDown function is called after each test function with the purpose
    of doing partial cleanup.
 5. `tearDownClass <VppTestCase.tearDownClass>`:
    Method called once after running all of the test functions to perform
    the final cleanup.

 Logging
 #######

 Each test case has a logger automatically created for it, stored in
 'logger' property, based on logging_. Use the logger's standard methods
 debug(), info(), error(), ... to emit log messages to the logger.

 All the log messages go always into a log file in temporary directory
 (see below).

 To control the messages printed to console, specify the V= parameter.

 .. code-block:: shell

    make test         # minimum verbosity
    make test V=1     # moderate verbosity
    make test V=2     # maximum verbosity

 Test temporary directory and VPP life cycle
 ###########################################

 Test separation is achieved by separating the test files and vpp instances.
 Each test creates a temporary directory and it's name is used to create
 a shared memory prefix which is used to run a VPP instance.
 The temporary directory name contains the testcase class name for easy
 reference, so for testcase named 'TestVxlan' the directory could be named
 e.g. vpp-unittest-TestVxlan-UNUP3j.
 This way, there is no conflict between any other VPP instances running
 on the box and the test VPP. Any temporary files created by the test case
 are stored in this temporary test directory.

 The test temporary directory holds the following interesting files:

 * log.txt - this contains the logger output on max verbosity
 * pg*_in.pcap - last injected packet stream into VPP, named after the interface,
   so for pg0, the file will be named pg0_in.pcap
 * pg*_out.pcap - last capture file created by VPP for interface, similarly,
   named after the interface, so for e.g. pg1, the file will be named
   pg1_out.pcap
 * history files - whenever the capture is restarted or a new stream is added,
   the existing files are rotated and renamed, soo all the pcap files
   are always saved for later debugging if needed
 * core - if vpp dumps a core, it'll be stored in the temporary directory
 * vpp_stdout.txt - file containing output which vpp printed to stdout
 * vpp_stderr.txt - file containing output which vpp printed to stderr

 *NOTE*: existing temporary directories named vpp-unittest-* are automatically
 removed when invoking 'make test*' or 'make retest*' to keep the temporary
 directory clean.

 Virtual environment
 ###################

 Virtualenv_ is a python module which provides a means to create an environment
 containing the dependencies required by the |vtf|, allowing a separation
 from any existing system-wide packages. |vtf|'s Makefile automatically
 creates a virtualenv_ inside build-root and installs the required packages
 in that environment. The environment is entered whenever executing a test
 via one of the make test targets.

 Naming conventions
 ##################

 Most unit tests do some kind of packet manipulation - sending and receiving
 packets between VPP and virtual hosts connected to the VPP. Referring
 to the sides, addresses, etc. is always done as if looking from the VPP side,
 thus:

 * *local_* prefix is used for the VPP side.
   So e.g. `local_ip4 <VppInterface.local_ip4>` address is the IPv4 address
   assigned to the VPP interface.
 * *remote_* prefix is used for the virtual host side.
   So e.g. `remote_mac <VppInterface.remote_mac>` address is the MAC address
   assigned to the virtual host connected to the VPP.

 Automatically generated addresses
 #################################

 To send packets, one needs to typically provide some addresses, otherwise
 the packets will be dropped. The interface objects in |vtf| automatically
 provide addresses based on (typically) their indexes, which ensures
 there are no conflicts and eases debugging by making the addressing scheme
 consistent.

 The developer of a test case typically doesn't need to work with the actual
 numbers, rather using the properties of the objects. The addresses typically
 come in two flavors: '<address>' and '<address>n' - note the 'n' suffix.
 The former address is a Python string, while the latter is translated using
 socket.inet_pton to raw format in network byte order - this format is suitable
 for passing as an argument to VPP APIs.

 e.g. for the IPv4 address assigned to the VPP interface:

 * local_ip4 - Local IPv4 address on VPP interface (string)
 * local_ip4n - Local IPv4 address - raw, suitable as API parameter.

 These addresses need to be configured in VPP to be usable using e.g.
 `config_ip4` API. Please see the documentation to `VppInterface` for more
 details.

 By default, there is one remote address of each kind created for L3:
 remote_ip4 and remote_ip6. If the test needs more addresses, because it's
 simulating more remote hosts, they can be generated using
 `generate_remote_hosts` API and the entries for them inserted into the ARP
 table using `configure_ipv4_neighbors` API.

 Packet flow in the |vtf|
 ########################

 Test framework -> VPP
 ~~~~~~~~~~~~~~~~~~~~~

 |vtf| doesn't send any packets to VPP directly. Traffic is instead injected
 using packet-generator interfaces, represented by the `VppPGInterface` class.
 Packets are written into a temporary .pcap file, which is then read by the VPP
 and the packets are injected into the VPP world.

 To add a list of packets to an interface, call the `add_stream` method on that
 interface. Once everything is prepared, call `pg_start` method to start
 the packet generator on the VPP side.

 VPP -> test framework
 ~~~~~~~~~~~~~~~~~~~~~

 Similarly, VPP doesn't send any packets to |vtf| directly. Instead, packet
 capture feature is used to capture and write traffic to a temporary .pcap file,
 which is then read and analyzed by the |vtf|.

 The following APIs are available to the test case for reading pcap files.

 * `get_capture`: this API is suitable for bulk & batch style of test, where
   a list of packets is prepared & sent, then the received packets are read
   and verified. The API needs the number of packets which are expected to
   be captured (ignoring filtered packets - see below) to know when the pcap
   file is completely written by the VPP. If using packet infos for verifying
   packets, then the counts of the packet infos can be automatically used
   by `get_capture` to get the proper count (in this case the default value
   None can be supplied as expected_count or ommitted altogether).
 * `wait_for_packet`: this API is suitable for interactive style of test,
   e.g. when doing session management, three-way handsakes, etc. This API waits
   for and returns a single packet, keeping the capture file in place
   and remembering context. Repeated invocations return following packets
   (or raise Exception if timeout is reached) from the same capture file
   (= packets arriving on the same interface).

 *NOTE*: it is not recommended to mix these APIs unless you understand how they
 work internally. None of these APIs rotate the pcap capture file, so calling
 e.g. `get_capture` after `wait_for_packet` will return already read packets.
 It is safe to switch from one API to another after calling `enable_capture`
 as that API rotates the capture file.

 Automatic filtering of packets:
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Both APIs (`get_capture` and `wait_for_packet`) by default filter the packet
 capture, removing known uninteresting packets from it - these are IPv6 Router
 Advertisments and IPv6 Router Alerts. These packets are unsolicitated
 and from the point of |vtf| are random. If a test wants to receive these
 packets, it should specify either None or a custom filtering function
 as the value to the 'filter_out_fn' argument.

 Common API flow for sending/receiving packets:
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 We will describe a simple scenario, where packets are sent from pg0 to pg1
 interface, assuming that the interfaces were created using
 `create_pg_interfaces` API.

 1. Create a list of packets for pg0::

      packet_count = 10
      packets = create_packets(src=self.pg0, dst=self.pg1,
                               count=packet_count)

 2. Add that list of packets to the source interface::

      self.pg0.add_stream(packets)

 3. Enable capture on the destination interface::

      self.pg1.enable_capture()

 4. Start the packet generator::

      self.pg_start()

 5. Wait for capture file to appear and read it::

      capture = self.pg1.get_capture(expected_count=packet_count)

 6. Verify packets match sent packets::

      self.verify_capture(send=packets, captured=capture)

 Test framework objects
 ######################

 The following objects provide VPP abstraction and provide a means to do
 common tasks easily in the test cases.

 * `VppInterface`: abstract class representing generic VPP interface
   and contains some common functionality, which is then used by derived classes
 * `VppPGInterface`: class representing VPP packet-generator interface.
   The interface is created/destroyed when the object is created/destroyed.
 * `VppSubInterface`: VPP sub-interface abstract class, containing common
   functionality for e.g. `VppDot1QSubint` and `VppDot1ADSubint` classes

 How VPP APIs/CLIs are called
 ############################

 Vpp provides python bindings in a python module called vpp-papi, which the test
 framework installs in the virtual environment. A shim layer represented by
 the `VppPapiProvider` class is built on top of the vpp-papi, serving these
 purposes:

 1. Automatic return value checks:
    After each API is called, the return value is checked against the expected
    return value (by default 0, but can be overridden) and an exception
    is raised if the check fails.
 2. Automatic call of hooks:

    a. `before_cli <Hook.before_cli>` and `before_api <Hook.before_api>` hooks
       are used for debug logging and stepping through the test
    b. `after_cli <Hook.after_cli>` and `after_api <Hook.after_api>` hooks
       are used for monitoring the vpp process for crashes
 3. Simplification of API calls:
    Many of the VPP APIs take a lot of parameters and by providing sane defaults
    for these, the API is much easier to use in the common case and the code is
    more readable. E.g. ip_add_del_route API takes ~25 parameters, of which
    in the common case, only 3 are needed.

 Utility methods
 ###############

 Some interesting utility methods are:

 * `ppp`: 'Pretty Print Packet' - returns a string containing the same output
   as Scapy's packet.show() would print
 * `ppc`: 'Pretty Print Capture' - returns a string containing printout of
   a capture (with configurable limit on the number of packets printed from it)
   using `ppp`

 *NOTE*: Do not use Scapy's packet.show() in the tests, because it prints
 the output to stdout. All output should go to the logger associated with
 the test case.

 Example: how to add a new test
 ##############################

 In this example, we will describe how to add a new test case which tests
 basic IPv4 forwarding.

 1. Add a new file called test_ip4_fwd.py in the test directory, starting
    with a few imports::

      from framework import VppTestCase
      from scapy.layers.l2 import Ether
      from scapy.packet import Raw
      from scapy.layers.inet import IP, UDP
      from random import randint

 2. Create a class inherited from the VppTestCase::

      class IP4FwdTestCase(VppTestCase):
          """ IPv4 simple forwarding test case """

 2. Add a setUpClass function containing the setup needed for our test to run::

          @classmethod
          def setUpClass(self):
              super(IP4FwdTestCase, self).setUpClass()
              self.create_pg_interfaces(range(2))  #  create pg0 and pg1
              for i in self.pg_interfaces:
                  i.admin_up()  # put the interface up
                  i.config_ip4()  # configure IPv4 address on the interface
                  i.resolve_arp()  # resolve ARP, so that we know VPP MAC

 3. Create a helper method to create the packets to send::

          def create_stream(self, src_if, dst_if, count):
              packets = []
              for i in range(count):
                  # create packet info stored in the test case instance
                  info = self.create_packet_info(src_if, dst_if)
                  # convert the info into packet payload
                  payload = self.info_to_payload(info)
                  # create the packet itself
                  p = (Ether(dst=src_if.local_mac, src=src_if.remote_mac) /
                       IP(src=src_if.remote_ip4, dst=dst_if.remote_ip4) /
                       UDP(sport=randint(1000, 2000), dport=5678) /
                       Raw(payload))
                  # store a copy of the packet in the packet info
                  info.data = p.copy()
                  # append the packet to the list
                  packets.append(p)

              # return the created packet list
              return packets

 4. Create a helper method to verify the capture::

          def verify_capture(self, src_if, dst_if, capture):
              packet_info = None
              for packet in capture:
                  try:
                      ip = packet[IP]
                      udp = packet[UDP]
                      # convert the payload to packet info object
                      payload_info = self.payload_to_info(str(packet[Raw]))
                      # make sure the indexes match
                      self.assert_equal(payload_info.src, src_if.sw_if_index,
                                        "source sw_if_index")
                      self.assert_equal(payload_info.dst, dst_if.sw_if_index,
                                        "destination sw_if_index")
                      packet_info = self.get_next_packet_info_for_interface2(
                                        src_if.sw_if_index,
                                        dst_if.sw_if_index,
                                        packet_info)
                      # make sure we didn't run out of saved packets
                      self.assertIsNotNone(packet_info)
                      self.assert_equal(payload_info.index, packet_info.index,
                                        "packet info index")
                      saved_packet = packet_info.data  # fetch the saved packet
                      # assert the values match
                      self.assert_equal(ip.src, saved_packet[IP].src,
                                        "IP source address")
                      # ... more assertions here
                      self.assert_equal(udp.sport, saved_packet[UDP].sport,
                                        "UDP source port")
                  except:
                      self.logger.error(ppp("Unexpected or invalid packet:",
                                        packet))
                      raise
              remaining_packet = self.get_next_packet_info_for_interface2(
                         src_if.sw_if_index,
                         dst_if.sw_if_index,
                         packet_info)
              self.assertIsNone(remaining_packet,
                                "Interface %s: Packet expected from interface "
                                "%s didn't arrive" % (dst_if.name, src_if.name))

 5. Add the test code to test_basic function::

          def test_basic(self):
              count = 10
              # create the packet stream
              packets = self.create_stream(self.pg0, self.pg1, count)
              # add the stream to the source interface
              self.pg0.add_stream(packets)
              # enable capture on both interfaces
              self.pg0.enable_capture()
              self.pg1.enable_capture()
              # start the packet generator
              self.pg_start()
              # get capture - the proper count of packets was saved by
              # create_packet_info() based on dst_if parameter
              capture = self.pg1.get_capture()
              # assert nothing captured on pg0 (always do this last, so that
              # some time has already passed since pg_start())
              self.pg0.assert_nothing_captured()
              # verify capture
              self.verify_capture(self.pg0, self.pg1, capture)

 6. Run the test by issuing 'make test'.


 |vtf| module documentation
 ##########################

 .. toctree::
    :maxdepth: 2
    :glob:

    modules.rst

 Indices and tables
 ==================

 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
	.. _unittest: https://docs.python.org/2/library/unittest.html
	.. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase
	.. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError
	.. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest
	.. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/
	.. _scapy: http://www.secdev.org/projects/scapy/
	.. _logging: https://docs.python.org/2/library/logging.html

	.. \|vtf\| replace:: VPP Test Framework

	\|vtf\|
	=====

	Overview
	########

	The goal of the \|vtf\| is to ease writing, running and debugging
	unit tests for the VPP. For this, python was chosen as a high level language
	allowing rapid development with scapy_ providing the necessary tool for creating
	and dissecting packets.

	Anatomy of a test case
	######################

	Python's unittest_ is used as the base framework upon which the VPP test
	framework is built. A test suite in the \|vtf\| consists of multiple classes
	derived from `VppTestCase`, which is itself derived from TestCase_.
	The test class defines one or more test functions, which act as test cases.

	Function flow when running a test case is:

	1. `setUpClass <VppTestCase.setUpClass>`:
	This function is called once for each test class, allowing a one-time test
	setup to be executed. If this functions throws an exception,
	none of the test functions are executed.
	2. `setUp <VppTestCase.setUp>`:
	The setUp function runs before each of the test functions. If this function
	throws an exception other than AssertionError_ or SkipTest_, then this is
	considered an error, not a test failure.
	3. test_<name>:
	This is the guts of the test case. It should execute the test scenario
	and use the various assert functions from the unittest framework to check
	necessary. Multiple test_<name> methods can exist in a test case.
	4. `tearDown <VppTestCase.tearDown>`:
	The tearDown function is called after each test function with the purpose
	of doing partial cleanup.
	5. `tearDownClass <VppTestCase.tearDownClass>`:
	Method called once after running all of the test functions to perform
	the final cleanup.

	Logging
	#######

	Each test case has a logger automatically created for it, stored in
	'logger' property, based on logging_. Use the logger's standard methods
	debug(), info(), error(), ... to emit log messages to the logger.

	All the log messages go always into a log file in temporary directory
	(see below).

	To control the messages printed to console, specify the V= parameter.

	.. code-block:: shell

	make test # minimum verbosity
	make test V=1 # moderate verbosity
	make test V=2 # maximum verbosity

	Test temporary directory and VPP life cycle
	###########################################

	Test separation is achieved by separating the test files and vpp instances.
	Each test creates a temporary directory and it's name is used to create
	a shared memory prefix which is used to run a VPP instance.
	The temporary directory name contains the testcase class name for easy
	reference, so for testcase named 'TestVxlan' the directory could be named
	e.g. vpp-unittest-TestVxlan-UNUP3j.
	This way, there is no conflict between any other VPP instances running
	on the box and the test VPP. Any temporary files created by the test case
	are stored in this temporary test directory.

	The test temporary directory holds the following interesting files:

	* log.txt - this contains the logger output on max verbosity
	* pg*_in.pcap - last injected packet stream into VPP, named after the interface,
	so for pg0, the file will be named pg0_in.pcap
	* pg*_out.pcap - last capture file created by VPP for interface, similarly,
	named after the interface, so for e.g. pg1, the file will be named
	pg1_out.pcap
	* history files - whenever the capture is restarted or a new stream is added,
	the existing files are rotated and renamed, soo all the pcap files
	are always saved for later debugging if needed
	* core - if vpp dumps a core, it'll be stored in the temporary directory
	* vpp_stdout.txt - file containing output which vpp printed to stdout
	* vpp_stderr.txt - file containing output which vpp printed to stderr

	NOTE: existing temporary directories named vpp-unittest-* are automatically
	removed when invoking 'make test' or 'make retest' to keep the temporary
	directory clean.

	Virtual environment
	###################

	Virtualenv_ is a python module which provides a means to create an environment
	containing the dependencies required by the \|vtf\|, allowing a separation
	from any existing system-wide packages. \|vtf\|'s Makefile automatically
	creates a virtualenv_ inside build-root and installs the required packages
	in that environment. The environment is entered whenever executing a test
	via one of the make test targets.

	Naming conventions
	##################

	Most unit tests do some kind of packet manipulation - sending and receiving
	packets between VPP and virtual hosts connected to the VPP. Referring
	to the sides, addresses, etc. is always done as if looking from the VPP side,
	thus:

	* local_ prefix is used for the VPP side.
	So e.g. `local_ip4 <VppInterface.local_ip4>` address is the IPv4 address
	assigned to the VPP interface.
	* remote_ prefix is used for the virtual host side.
	So e.g. `remote_mac <VppInterface.remote_mac>` address is the MAC address
	assigned to the virtual host connected to the VPP.

	Automatically generated addresses
	#################################

	To send packets, one needs to typically provide some addresses, otherwise
	the packets will be dropped. The interface objects in \|vtf\| automatically
	provide addresses based on (typically) their indexes, which ensures
	there are no conflicts and eases debugging by making the addressing scheme
	consistent.

	The developer of a test case typically doesn't need to work with the actual
	numbers, rather using the properties of the objects. The addresses typically
	come in two flavors: '<address>' and '<address>n' - note the 'n' suffix.
	The former address is a Python string, while the latter is translated using
	socket.inet_pton to raw format in network byte order - this format is suitable
	for passing as an argument to VPP APIs.

	e.g. for the IPv4 address assigned to the VPP interface:

	* local_ip4 - Local IPv4 address on VPP interface (string)
	* local_ip4n - Local IPv4 address - raw, suitable as API parameter.

	These addresses need to be configured in VPP to be usable using e.g.
	`config_ip4` API. Please see the documentation to `VppInterface` for more
	details.

	By default, there is one remote address of each kind created for L3:
	remote_ip4 and remote_ip6. If the test needs more addresses, because it's
	simulating more remote hosts, they can be generated using
	`generate_remote_hosts` API and the entries for them inserted into the ARP
	table using `configure_ipv4_neighbors` API.

	Packet flow in the \|vtf\|
	########################

	Test framework -> VPP
	~~~~~~~~~~~~~~~~~~~~~

	\|vtf\| doesn't send any packets to VPP directly. Traffic is instead injected
	using packet-generator interfaces, represented by the `VppPGInterface` class.
	Packets are written into a temporary .pcap file, which is then read by the VPP
	and the packets are injected into the VPP world.

	To add a list of packets to an interface, call the `add_stream` method on that
	interface. Once everything is prepared, call `pg_start` method to start
	the packet generator on the VPP side.

	VPP -> test framework
	~~~~~~~~~~~~~~~~~~~~~

	Similarly, VPP doesn't send any packets to \|vtf\| directly. Instead, packet
	capture feature is used to capture and write traffic to a temporary .pcap file,
	which is then read and analyzed by the \|vtf\|.

	The following APIs are available to the test case for reading pcap files.

	* `get_capture`: this API is suitable for bulk & batch style of test, where
	a list of packets is prepared & sent, then the received packets are read
	and verified. The API needs the number of packets which are expected to
	be captured (ignoring filtered packets - see below) to know when the pcap
	file is completely written by the VPP. If using packet infos for verifying
	packets, then the counts of the packet infos can be automatically used
	by `get_capture` to get the proper count (in this case the default value
	None can be supplied as expected_count or ommitted altogether).
	* `wait_for_packet`: this API is suitable for interactive style of test,
	e.g. when doing session management, three-way handsakes, etc. This API waits
	for and returns a single packet, keeping the capture file in place
	and remembering context. Repeated invocations return following packets
	(or raise Exception if timeout is reached) from the same capture file
	(= packets arriving on the same interface).

	NOTE: it is not recommended to mix these APIs unless you understand how they
	work internally. None of these APIs rotate the pcap capture file, so calling
	e.g. `get_capture` after `wait_for_packet` will return already read packets.
	It is safe to switch from one API to another after calling `enable_capture`
	as that API rotates the capture file.

	Automatic filtering of packets:
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Both APIs (`get_capture` and `wait_for_packet`) by default filter the packet
	capture, removing known uninteresting packets from it - these are IPv6 Router
	Advertisments and IPv6 Router Alerts. These packets are unsolicitated
	and from the point of \|vtf\| are random. If a test wants to receive these
	packets, it should specify either None or a custom filtering function
	as the value to the 'filter_out_fn' argument.

	Common API flow for sending/receiving packets:
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	We will describe a simple scenario, where packets are sent from pg0 to pg1
	interface, assuming that the interfaces were created using
	`create_pg_interfaces` API.

	1. Create a list of packets for pg0::

	packet_count = 10
	packets = create_packets(src=self.pg0, dst=self.pg1,
	count=packet_count)

	2. Add that list of packets to the source interface::

	self.pg0.add_stream(packets)

	3. Enable capture on the destination interface::

	self.pg1.enable_capture()

	4. Start the packet generator::

	self.pg_start()

	5. Wait for capture file to appear and read it::

	capture = self.pg1.get_capture(expected_count=packet_count)

	6. Verify packets match sent packets::

	self.verify_capture(send=packets, captured=capture)

	Test framework objects
	######################

	The following objects provide VPP abstraction and provide a means to do
	common tasks easily in the test cases.

	* `VppInterface`: abstract class representing generic VPP interface
	and contains some common functionality, which is then used by derived classes
	* `VppPGInterface`: class representing VPP packet-generator interface.
	The interface is created/destroyed when the object is created/destroyed.
	* `VppSubInterface`: VPP sub-interface abstract class, containing common
	functionality for e.g. `VppDot1QSubint` and `VppDot1ADSubint` classes

	How VPP APIs/CLIs are called
	############################

	Vpp provides python bindings in a python module called vpp-papi, which the test
	framework installs in the virtual environment. A shim layer represented by
	the `VppPapiProvider` class is built on top of the vpp-papi, serving these
	purposes:

	1. Automatic return value checks:
	After each API is called, the return value is checked against the expected
	return value (by default 0, but can be overridden) and an exception
	is raised if the check fails.
	2. Automatic call of hooks:

	a. `before_cli <Hook.before_cli>` and `before_api <Hook.before_api>` hooks
	are used for debug logging and stepping through the test
	b. `after_cli <Hook.after_cli>` and `after_api <Hook.after_api>` hooks
	are used for monitoring the vpp process for crashes
	3. Simplification of API calls:
	Many of the VPP APIs take a lot of parameters and by providing sane defaults
	for these, the API is much easier to use in the common case and the code is
	more readable. E.g. ip_add_del_route API takes ~25 parameters, of which
	in the common case, only 3 are needed.

	Utility methods
	###############

	Some interesting utility methods are:

	* `ppp`: 'Pretty Print Packet' - returns a string containing the same output
	as Scapy's packet.show() would print
	* `ppc`: 'Pretty Print Capture' - returns a string containing printout of
	a capture (with configurable limit on the number of packets printed from it)
	using `ppp`

	NOTE: Do not use Scapy's packet.show() in the tests, because it prints
	the output to stdout. All output should go to the logger associated with
	the test case.

	Example: how to add a new test
	##############################

	In this example, we will describe how to add a new test case which tests
	basic IPv4 forwarding.

	1. Add a new file called test_ip4_fwd.py in the test directory, starting
	with a few imports::

	from framework import VppTestCase
	from scapy.layers.l2 import Ether
	from scapy.packet import Raw
	from scapy.layers.inet import IP, UDP
	from random import randint

	2. Create a class inherited from the VppTestCase::

	class IP4FwdTestCase(VppTestCase):
	""" IPv4 simple forwarding test case """

	2. Add a setUpClass function containing the setup needed for our test to run::

	@classmethod
	def setUpClass(self):
	super(IP4FwdTestCase, self).setUpClass()
	self.create_pg_interfaces(range(2)) # create pg0 and pg1
	for i in self.pg_interfaces:
	i.admin_up() # put the interface up
	i.config_ip4() # configure IPv4 address on the interface
	i.resolve_arp() # resolve ARP, so that we know VPP MAC

	3. Create a helper method to create the packets to send::

	def create_stream(self, src_if, dst_if, count):
	packets = []
	for i in range(count):
	# create packet info stored in the test case instance
	info = self.create_packet_info(src_if, dst_if)
	# convert the info into packet payload
	payload = self.info_to_payload(info)
	# create the packet itself
	p = (Ether(dst=src_if.local_mac, src=src_if.remote_mac) /
	IP(src=src_if.remote_ip4, dst=dst_if.remote_ip4) /
	UDP(sport=randint(1000, 2000), dport=5678) /
	Raw(payload))
	# store a copy of the packet in the packet info
	info.data = p.copy()
	# append the packet to the list
	packets.append(p)

	# return the created packet list
	return packets

	4. Create a helper method to verify the capture::

	def verify_capture(self, src_if, dst_if, capture):
	packet_info = None
	for packet in capture:
	try:
	ip = packet[IP]
	udp = packet[UDP]
	# convert the payload to packet info object
	payload_info = self.payload_to_info(str(packet[Raw]))
	# make sure the indexes match
	self.assert_equal(payload_info.src, src_if.sw_if_index,
	"source sw_if_index")
	self.assert_equal(payload_info.dst, dst_if.sw_if_index,
	"destination sw_if_index")
	packet_info = self.get_next_packet_info_for_interface2(
	src_if.sw_if_index,
	dst_if.sw_if_index,
	packet_info)
	# make sure we didn't run out of saved packets
	self.assertIsNotNone(packet_info)
	self.assert_equal(payload_info.index, packet_info.index,
	"packet info index")
	saved_packet = packet_info.data # fetch the saved packet
	# assert the values match
	self.assert_equal(ip.src, saved_packet[IP].src,
	"IP source address")
	# ... more assertions here
	self.assert_equal(udp.sport, saved_packet[UDP].sport,
	"UDP source port")
	except:
	self.logger.error(ppp("Unexpected or invalid packet:",
	packet))
	raise
	remaining_packet = self.get_next_packet_info_for_interface2(
	src_if.sw_if_index,
	dst_if.sw_if_index,
	packet_info)
	self.assertIsNone(remaining_packet,
	"Interface %s: Packet expected from interface "
	"%s didn't arrive" % (dst_if.name, src_if.name))

	5. Add the test code to test_basic function::

	def test_basic(self):
	count = 10
	# create the packet stream
	packets = self.create_stream(self.pg0, self.pg1, count)
	# add the stream to the source interface
	self.pg0.add_stream(packets)
	# enable capture on both interfaces
	self.pg0.enable_capture()
	self.pg1.enable_capture()
	# start the packet generator
	self.pg_start()
	# get capture - the proper count of packets was saved by
	# create_packet_info() based on dst_if parameter
	capture = self.pg1.get_capture()
	# assert nothing captured on pg0 (always do this last, so that
	# some time has already passed since pg_start())
	self.pg0.assert_nothing_captured()
	# verify capture
	self.verify_capture(self.pg0, self.pg1, capture)

	6. Run the test by issuing 'make test'.


	\|vtf\| module documentation
	##########################

	.. toctree::
	:maxdepth: 2
	:glob:

	modules.rst

	Indices and tables
	==================

	* :ref:`genindex`
	* :ref:`modindex`
	* :ref:`search`